+1 Also the proposed feature would need to be implemented in a way that avoids a hard dependency on log4j. The interface for logging is slf4j and it should be possible to use other logger backends.
On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko <ser...@datatorrent.com> wrote: > I don't think an operator needs a specific appender. An appender can be > dynamically assigned to an application designer, application master and > container. > > Thanks, > Sergey > > > On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath <r...@datatorrent.com> > wrote: > > > I don't have one, I thought that was what the intent of the proposal was, > > but looks like > > I misunderstood. After re-reading some of the earlier responses, I > > understand the > > proposal better. > > > > Ram > > > > > > > > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov <v.ro...@datatorrent.com> > > wrote: > > > > > I don't see a use case where an individual operators need to define a > > > specific appender, can you provide one? > > > > > > Thank you, > > > > > > Vlad > > > > > > On 4/10/17 16:53, Munagala Ramanath wrote: > > > > > >> Yes, totally agree, it would be helpful to have a detailed use case > > and/or > > >> a detailed spec > > >> of the desired capabilities -- not necessarily a complete spec but > with > > >> enough detail to > > >> understand why existing capabilities are inadequate. > > >> > > >> Ram > > >> > > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <v.ro...@datatorrent.com> > > >> wrote: > > >> > > >> It will be good to understand a use case where an operator needs a > > >>> specific appender. > > >>> > > >>> IMO, an operator designer defines *what* should be logged and dev-ops > > >>> team > > >>> defines *where* to log. > > >>> > > >>> Thank you, > > >>> > > >>> Vlad > > >>> On 4/10/17 16:27, Munagala Ramanath wrote: > > >>> > > >>> Yes, I understand, I was just wondering if individual operators could > > >>>> define the appenders > > >>>> they potentially need at compile time and then the operator > callbacks > > >>>> could > > >>>> simply > > >>>> check the desired runtime condition and add the appropriate > appender. > > >>>> > > >>>> Or are we saying there are scenarios where we absolutely cannot > create > > >>>> the > > >>>> appender beforehand ? > > >>>> > > >>>> So broadly speaking, my question is whether the combination of > > providing > > >>>> predefined appenders > > >>>> and the PropertyConfigurator capabilities meets the need. > > >>>> > > >>>> Ram > > >>>> > > >>>> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko < > > ser...@datatorrent.com > > >>>> > > > >>>> wrote: > > >>>> > > >>>> Ram, > > >>>> > > >>>>> Really the new appender class must extend the abstract class > > >>>>> AppenderSkeleton. And in order to add a new appender > programmatically > > >>>>> in > > >>>>> Java, some code in Apex should call the following log4j method: > > >>>>> > > >>>>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender > > >>>>> newAppender) > > >>>>> > > >>>>> The general idea of my proposal is "*based on some runtime > > parameter(s) > > >>>>> to > > >>>>> provide ability to create an appender instance via reflection and > add > > >>>>> it > > >>>>> to > > >>>>> the list of active log4j appenders*". > > >>>>> > > >>>>> Thanks, > > >>>>> Sergey > > >>>>> > > >>>>> > > >>>>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov < > v.ro...@datatorrent.com > > > > > >>>>> wrote: > > >>>>> > > >>>>> It will require application recompilation and repackaging. The > > proposed > > >>>>> > > >>>>>> functionality is for dev-ops to be able to route application > logging > > >>>>>> to > > >>>>>> a > > >>>>>> preferred destination without recompiling applications. It is > > run-time > > >>>>>> configuration vs compile time hardcoded appender. > > >>>>>> > > >>>>>> Thank you, > > >>>>>> > > >>>>>> Vlad > > >>>>>> > > >>>>>> On 4/10/17 11:23, Munagala Ramanath wrote: > > >>>>>> > > >>>>>> You can do it in a trivial derived class without changing the base > > >>>>>> class. > > >>>>>> Ram > > >>>>>> > > >>>>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov < > > >>>>>>> v.ro...@datatorrent.com> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>> Does not the proposal to use Logger.addAppender() requires > > >>>>>>> modifications > > >>>>>>> > > >>>>>>> to used operators code? > > >>>>>>>> > > >>>>>>>> Thank you, > > >>>>>>>> > > >>>>>>>> Vlad > > >>>>>>>> > > >>>>>>>> On 4/10/17 10:58, Munagala Ramanath wrote: > > >>>>>>>> > > >>>>>>>> People can currently do this by simply implementing the Appender > > >>>>>>>> > > >>>>>>>> interface > > >>>>>>>>> and adding it > > >>>>>>>>> with Logger.addAppender() in the setup method. Why do we need > > >>>>>>>>> > > >>>>>>>>> something > > >>>>>>>> > > >>>>>>> more elaborate ? > > >>>>>> > > >>>>>>> Ram > > >>>>>>>>> > > >>>>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko < > > >>>>>>>>> ser...@datatorrent.com> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>> The configuration of a log4j appender via log4j configuration > > file > > >>>>>>>>> is > > >>>>>>>>> > > >>>>>>>>> a > > >>>>>>>> > > >>>>>>> static configuration that cannot be disabled/enabled and managed > > >>>>>> > > >>>>>>> dynamically by an application designer. The programmatic approach > > >>>>>>>>>> > > >>>>>>>>>> will > > >>>>>>>>> > > >>>>>>>> allow an application designer to specify which of the available > > >>>>>> > > >>>>>>> log4j > > >>>>>>>>> > > >>>>>>>> appenders should be used for the specific application. > > >>>>>> > > >>>>>>> It is not necessary Apex should use the predefined log4j > appenders > > >>>>>>>>>> only. > > >>>>>>>>>> The log4j events contain useful but the very limited number of > > >>>>>>>>>> properties > > >>>>>>>>>> which values can be printed into output log4j sources. But > based > > >>>>>>>>>> on > > >>>>>>>>>> > > >>>>>>>>>> the > > >>>>>>>>> > > >>>>>>>> knowledge of the software product workflow, the custom defined > > log4j > > >>>>>> > > >>>>>>> appender can extend a list of predefined output log events > > >>>>>>>>>> properties > > >>>>>>>>>> and, > > >>>>>>>>>> for instance for Apex, return: node, user name, application > > name, > > >>>>>>>>>> application id, container id, operator name, etc. > > >>>>>>>>>> > > >>>>>>>>>> Also the output log events that are generated by a custom > > defined > > >>>>>>>>>> > > >>>>>>>>>> log4j > > >>>>>>>>> > > >>>>>>>> appender can be stored and indexed by any type of a full text > > search > > >>>>>> > > >>>>>>> database. It will allow the customers and developers to simplify > > >>>>>>>>>> collection > > >>>>>>>>>> of log events statistics and searching/filtering of specific > > >>>>>>>>>> events > > >>>>>>>>>> > > >>>>>>>>>> for > > >>>>>>>>> > > >>>>>>>> debugging and investigation. > > >>>>>> > > >>>>>>> Thanks, > > >>>>>>>>>> Sergey > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov < > > >>>>>>>>>> v.ro...@datatorrent.com > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>> +1 Apex engine does not own log4j config file - it is provided > > >>>>>>>>>> either > > >>>>>>>>>> by > > >>>>>>>>>> > > >>>>>>>>>> Hadoop or an application. Hadoop log4j config does not > > necessarily > > >>>>>>>>>> > > >>>>>>>>>>> meet > > >>>>>>>>>>> application logging requirements, but if log4j is provided by > > an > > >>>>>>>>>>> application designer, who can only specify what to log, it > may > > >>>>>>>>>>> not > > >>>>>>>>>>> meet > > >>>>>>>>>>> operations requirements. Dev-ops should have an ability to > > >>>>>>>>>>> specify > > >>>>>>>>>>> where > > >>>>>>>>>>> > > >>>>>>>>>>> to > > >>>>>>>>>>> > > >>>>>>>>>>> log depending on the available infrastructure at run-time. > > >>>>>>>>>> > > >>>>>>>>>> It will be good to have an ability not only specify extra > log4j > > >>>>>>>>>>> appenders > > >>>>>>>>>>> at lunch time, but also at run-time, the same way how log4j > > >>>>>>>>>>> logger > > >>>>>>>>>>> levels > > >>>>>>>>>>> may be changed. > > >>>>>>>>>>> > > >>>>>>>>>>> Thank you, > > >>>>>>>>>>> > > >>>>>>>>>>> Vlad > > >>>>>>>>>>> > > >>>>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote: > > >>>>>>>>>>> > > >>>>>>>>>>> We can always write a custom appender and add it by changing > > root > > >>>>>>>>>>> appender > > >>>>>>>>>>> in log4j config file. Can you explain how adding appender > > >>>>>>>>>>> grammatically > > >>>>>>>>>>> > > >>>>>>>>>>> would help? > > >>>>>>>>>>> > > >>>>>>>>>>>> -Priyanka > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare < > > >>>>>>>>>>>> san...@datatorrent.com > > >>>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>> Please give some examples and/or use cases of this > > programmatic > > >>>>>>>>>>>> > > >>>>>>>>>>>> log4j > > >>>>>>>>>>> > > >>>>>>>>>> appender. > > >>>>>> > > >>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko < > > >>>>>>>>>>>>> ser...@datatorrent.com > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Hi All, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> I'd like to add supporting of a custom defined log4j > appender > > >>>>>>>>>>>>> that > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> can > > >>>>>>>>>>>>>> be > > >>>>>>>>>>>>>> added to Apex Application Master and Containers and be > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> configurable > > >>>>>>>>>>>>> > > >>>>>>>>>>>> programmatically. > > >>>>>> > > >>>>>>> Sometimes it is not trivial to control log4j configuration via > > >>>>>>>>>>>>>> log4j > > >>>>>>>>>>>>>> properties. And I think the having of the approach to add > a > > >>>>>>>>>>>>>> log4j > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> appender > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> programmatically will allow the customers and developers > to > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> plugin > > >>>>>>>>>>>>> their > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> own custom defined log4j appenders and be much flexible for > > >>>>>>>>>>>>> > > >>>>>>>>>>>> streaming > > >>>>>>>>>>> > > >>>>>>>>>> and > > >>>>>> > > >>>>>>> collection of Apex log events. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> I assume to provide generic approach for definition of the > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> programmatic > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> log4j appender and to pass all configuration parameters > > >>>>>>>>>>>>> > > >>>>>>>>>>>> including a > > >>>>>>>>>>>> name > > >>>>>>>>>>>> of > > >>>>>>>>>>>> the Java class with implementation of the log4j appender via > > >>>>>>>>>>>> system > > >>>>>>>>>>>> > > >>>>>>>>>>>> and/or > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> command line properties. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Sergey > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>> > > >> > > > > > > > > > -- > > > > _______________________________________________________ > > > > Munagala V. Ramanath > > > > Software Engineer > > > > E: r...@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam > > > > www.datatorrent.com | apex.apache.org > > >
