I don't think an operator needs a specific appender. An appender can be dynamically assigned to an application designer, application master and container.
Thanks, Sergey On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath <r...@datatorrent.com> wrote: > I don't have one, I thought that was what the intent of the proposal was, > but looks like > I misunderstood. After re-reading some of the earlier responses, I > understand the > proposal better. > > Ram > > > > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov <v.ro...@datatorrent.com> > wrote: > > > I don't see a use case where an individual operators need to define a > > specific appender, can you provide one? > > > > Thank you, > > > > Vlad > > > > On 4/10/17 16:53, Munagala Ramanath wrote: > > > >> Yes, totally agree, it would be helpful to have a detailed use case > and/or > >> a detailed spec > >> of the desired capabilities -- not necessarily a complete spec but with > >> enough detail to > >> understand why existing capabilities are inadequate. > >> > >> Ram > >> > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <v.ro...@datatorrent.com> > >> wrote: > >> > >> It will be good to understand a use case where an operator needs a > >>> specific appender. > >>> > >>> IMO, an operator designer defines *what* should be logged and dev-ops > >>> team > >>> defines *where* to log. > >>> > >>> Thank you, > >>> > >>> Vlad > >>> On 4/10/17 16:27, Munagala Ramanath wrote: > >>> > >>> Yes, I understand, I was just wondering if individual operators could > >>>> define the appenders > >>>> they potentially need at compile time and then the operator callbacks > >>>> could > >>>> simply > >>>> check the desired runtime condition and add the appropriate appender. > >>>> > >>>> Or are we saying there are scenarios where we absolutely cannot create > >>>> the > >>>> appender beforehand ? > >>>> > >>>> So broadly speaking, my question is whether the combination of > providing > >>>> predefined appenders > >>>> and the PropertyConfigurator capabilities meets the need. > >>>> > >>>> Ram > >>>> > >>>> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko < > ser...@datatorrent.com > >>>> > > >>>> wrote: > >>>> > >>>> Ram, > >>>> > >>>>> Really the new appender class must extend the abstract class > >>>>> AppenderSkeleton. And in order to add a new appender programmatically > >>>>> in > >>>>> Java, some code in Apex should call the following log4j method: > >>>>> > >>>>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender > >>>>> newAppender) > >>>>> > >>>>> The general idea of my proposal is "*based on some runtime > parameter(s) > >>>>> to > >>>>> provide ability to create an appender instance via reflection and add > >>>>> it > >>>>> to > >>>>> the list of active log4j appenders*". > >>>>> > >>>>> Thanks, > >>>>> Sergey > >>>>> > >>>>> > >>>>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <v.ro...@datatorrent.com > > > >>>>> wrote: > >>>>> > >>>>> It will require application recompilation and repackaging. The > proposed > >>>>> > >>>>>> functionality is for dev-ops to be able to route application logging > >>>>>> to > >>>>>> a > >>>>>> preferred destination without recompiling applications. It is > run-time > >>>>>> configuration vs compile time hardcoded appender. > >>>>>> > >>>>>> Thank you, > >>>>>> > >>>>>> Vlad > >>>>>> > >>>>>> On 4/10/17 11:23, Munagala Ramanath wrote: > >>>>>> > >>>>>> You can do it in a trivial derived class without changing the base > >>>>>> class. > >>>>>> Ram > >>>>>> > >>>>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov < > >>>>>>> v.ro...@datatorrent.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>> Does not the proposal to use Logger.addAppender() requires > >>>>>>> modifications > >>>>>>> > >>>>>>> to used operators code? > >>>>>>>> > >>>>>>>> Thank you, > >>>>>>>> > >>>>>>>> Vlad > >>>>>>>> > >>>>>>>> On 4/10/17 10:58, Munagala Ramanath wrote: > >>>>>>>> > >>>>>>>> People can currently do this by simply implementing the Appender > >>>>>>>> > >>>>>>>> interface > >>>>>>>>> and adding it > >>>>>>>>> with Logger.addAppender() in the setup method. Why do we need > >>>>>>>>> > >>>>>>>>> something > >>>>>>>> > >>>>>>> more elaborate ? > >>>>>> > >>>>>>> Ram > >>>>>>>>> > >>>>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko < > >>>>>>>>> ser...@datatorrent.com> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> The configuration of a log4j appender via log4j configuration > file > >>>>>>>>> is > >>>>>>>>> > >>>>>>>>> a > >>>>>>>> > >>>>>>> static configuration that cannot be disabled/enabled and managed > >>>>>> > >>>>>>> dynamically by an application designer. The programmatic approach > >>>>>>>>>> > >>>>>>>>>> will > >>>>>>>>> > >>>>>>>> allow an application designer to specify which of the available > >>>>>> > >>>>>>> log4j > >>>>>>>>> > >>>>>>>> appenders should be used for the specific application. > >>>>>> > >>>>>>> It is not necessary Apex should use the predefined log4j appenders > >>>>>>>>>> only. > >>>>>>>>>> The log4j events contain useful but the very limited number of > >>>>>>>>>> properties > >>>>>>>>>> which values can be printed into output log4j sources. But based > >>>>>>>>>> on > >>>>>>>>>> > >>>>>>>>>> the > >>>>>>>>> > >>>>>>>> knowledge of the software product workflow, the custom defined > log4j > >>>>>> > >>>>>>> appender can extend a list of predefined output log events > >>>>>>>>>> properties > >>>>>>>>>> and, > >>>>>>>>>> for instance for Apex, return: node, user name, application > name, > >>>>>>>>>> application id, container id, operator name, etc. > >>>>>>>>>> > >>>>>>>>>> Also the output log events that are generated by a custom > defined > >>>>>>>>>> > >>>>>>>>>> log4j > >>>>>>>>> > >>>>>>>> appender can be stored and indexed by any type of a full text > search > >>>>>> > >>>>>>> database. It will allow the customers and developers to simplify > >>>>>>>>>> collection > >>>>>>>>>> of log events statistics and searching/filtering of specific > >>>>>>>>>> events > >>>>>>>>>> > >>>>>>>>>> for > >>>>>>>>> > >>>>>>>> debugging and investigation. > >>>>>> > >>>>>>> Thanks, > >>>>>>>>>> Sergey > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov < > >>>>>>>>>> v.ro...@datatorrent.com > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> +1 Apex engine does not own log4j config file - it is provided > >>>>>>>>>> either > >>>>>>>>>> by > >>>>>>>>>> > >>>>>>>>>> Hadoop or an application. Hadoop log4j config does not > necessarily > >>>>>>>>>> > >>>>>>>>>>> meet > >>>>>>>>>>> application logging requirements, but if log4j is provided by > an > >>>>>>>>>>> application designer, who can only specify what to log, it may > >>>>>>>>>>> not > >>>>>>>>>>> meet > >>>>>>>>>>> operations requirements. Dev-ops should have an ability to > >>>>>>>>>>> specify > >>>>>>>>>>> where > >>>>>>>>>>> > >>>>>>>>>>> to > >>>>>>>>>>> > >>>>>>>>>>> log depending on the available infrastructure at run-time. > >>>>>>>>>> > >>>>>>>>>> It will be good to have an ability not only specify extra log4j > >>>>>>>>>>> appenders > >>>>>>>>>>> at lunch time, but also at run-time, the same way how log4j > >>>>>>>>>>> logger > >>>>>>>>>>> levels > >>>>>>>>>>> may be changed. > >>>>>>>>>>> > >>>>>>>>>>> Thank you, > >>>>>>>>>>> > >>>>>>>>>>> Vlad > >>>>>>>>>>> > >>>>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote: > >>>>>>>>>>> > >>>>>>>>>>> We can always write a custom appender and add it by changing > root > >>>>>>>>>>> appender > >>>>>>>>>>> in log4j config file. Can you explain how adding appender > >>>>>>>>>>> grammatically > >>>>>>>>>>> > >>>>>>>>>>> would help? > >>>>>>>>>>> > >>>>>>>>>>>> -Priyanka > >>>>>>>>>>>> > >>>>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare < > >>>>>>>>>>>> san...@datatorrent.com > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Please give some examples and/or use cases of this > programmatic > >>>>>>>>>>>> > >>>>>>>>>>>> log4j > >>>>>>>>>>> > >>>>>>>>>> appender. > >>>>>> > >>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko < > >>>>>>>>>>>>> ser...@datatorrent.com > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi All, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I'd like to add supporting of a custom defined log4j appender > >>>>>>>>>>>>> that > >>>>>>>>>>>>> > >>>>>>>>>>>>> can > >>>>>>>>>>>>>> be > >>>>>>>>>>>>>> added to Apex Application Master and Containers and be > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> configurable > >>>>>>>>>>>>> > >>>>>>>>>>>> programmatically. > >>>>>> > >>>>>>> Sometimes it is not trivial to control log4j configuration via > >>>>>>>>>>>>>> log4j > >>>>>>>>>>>>>> properties. And I think the having of the approach to add a > >>>>>>>>>>>>>> log4j > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> appender > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> programmatically will allow the customers and developers to > >>>>>>>>>>>>>> > >>>>>>>>>>>>> plugin > >>>>>>>>>>>>> their > >>>>>>>>>>>>> > >>>>>>>>>>>>> own custom defined log4j appenders and be much flexible for > >>>>>>>>>>>>> > >>>>>>>>>>>> streaming > >>>>>>>>>>> > >>>>>>>>>> and > >>>>>> > >>>>>>> collection of Apex log events. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I assume to provide generic approach for definition of the > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> programmatic > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> log4j appender and to pass all configuration parameters > >>>>>>>>>>>>> > >>>>>>>>>>>> including a > >>>>>>>>>>>> name > >>>>>>>>>>>> of > >>>>>>>>>>>> the Java class with implementation of the log4j appender via > >>>>>>>>>>>> system > >>>>>>>>>>>> > >>>>>>>>>>>> and/or > >>>>>>>>>>>>> > >>>>>>>>>>>>>> command line properties. > >>>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Sergey > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>> > >> > > > > > -- > > _______________________________________________________ > > Munagala V. Ramanath > > Software Engineer > > E: r...@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam > > www.datatorrent.com | apex.apache.org >
