Hi Thomas, I assume to have maximum generic implementation. The signatures of all methods in the implementation will not contain anything log4j specific. And if we decide to add an abstraction layer to the logger calls in Apex or to use another logger implementation, it can be easily changed to support any new appender interfaces.
Thanks, Sergey On Sat, Apr 15, 2017 at 4:00 PM, Thomas Weise <t...@apache.org> wrote: > Hi Sergey, > > What I'm asking is that the feature is implemented in a way that will allow > Apex to run with different logger backend. That means that log4j needs to > be optional. > > Thanks, > Thomas > > > On Sat, Apr 15, 2017 at 2:01 PM, Sergey Golovko <ser...@datatorrent.com> > wrote: > > > I agree it would be very nice to use only slf4j interfaces for the > > implementation. But unfortunately the interface Appender belongs to > > org.apache.log4j package. > > > > "SLF4J is only a facade, meaning that it does not provide a complete > > logging solution. Operations such as configuring appenders or setting > > logging levels cannot be performed with SLF4J. Thus, at some point in > time, > > any non-trivial application will need to directly invoke the underlying > > logging system. In other words, complete independence from the API > > underlying logging system is not possible for a stand-alone application. > > Nevertheless, SLF4J reduces the impact of this dependence to > near-painless > > levels." > > > > https://www.slf4j.org/faq.html#when > > > > Thanks, > > Sergey > > > > > > On Thu, Apr 13, 2017 at 7:56 AM, Thomas Weise <t...@apache.org> wrote: > > > > > +1 > > > > > > Also the proposed feature would need to be implemented in a way that > > avoids > > > a hard dependency on log4j. The interface for logging is slf4j and it > > > should be possible to use other logger backends. > > > > > > > > > On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko < > ser...@datatorrent.com> > > > wrote: > > > > > > > I don't think an operator needs a specific appender. An appender can > be > > > > dynamically assigned to an application designer, application master > and > > > > container. > > > > > > > > Thanks, > > > > Sergey > > > > > > > > > > > > On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath < > > r...@datatorrent.com> > > > > wrote: > > > > > > > > > I don't have one, I thought that was what the intent of the > proposal > > > was, > > > > > but looks like > > > > > I misunderstood. After re-reading some of the earlier responses, I > > > > > understand the > > > > > proposal better. > > > > > > > > > > Ram > > > > > > > > > > > > > > > > > > > > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov < > v.ro...@datatorrent.com > > > > > > > > wrote: > > > > > > > > > > > I don't see a use case where an individual operators need to > > define a > > > > > > specific appender, can you provide one? > > > > > > > > > > > > Thank you, > > > > > > > > > > > > Vlad > > > > > > > > > > > > On 4/10/17 16:53, Munagala Ramanath wrote: > > > > > > > > > > > >> Yes, totally agree, it would be helpful to have a detailed use > > case > > > > > and/or > > > > > >> a detailed spec > > > > > >> of the desired capabilities -- not necessarily a complete spec > but > > > > with > > > > > >> enough detail to > > > > > >> understand why existing capabilities are inadequate. > > > > > >> > > > > > >> Ram > > > > > >> > > > > > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov < > > > v.ro...@datatorrent.com> > > > > > >> wrote: > > > > > >> > > > > > >> It will be good to understand a use case where an operator > needs a > > > > > >>> specific appender. > > > > > >>> > > > > > >>> IMO, an operator designer defines *what* should be logged and > > > dev-ops > > > > > >>> team > > > > > >>> defines *where* to log. > > > > > >>> > > > > > >>> Thank you, > > > > > >>> > > > > > >>> Vlad > > > > > >>> On 4/10/17 16:27, Munagala Ramanath wrote: > > > > > >>> > > > > > >>> Yes, I understand, I was just wondering if individual operators > > > could > > > > > >>>> define the appenders > > > > > >>>> they potentially need at compile time and then the operator > > > > callbacks > > > > > >>>> could > > > > > >>>> simply > > > > > >>>> check the desired runtime condition and add the appropriate > > > > appender. > > > > > >>>> > > > > > >>>> Or are we saying there are scenarios where we absolutely > cannot > > > > create > > > > > >>>> the > > > > > >>>> appender beforehand ? > > > > > >>>> > > > > > >>>> So broadly speaking, my question is whether the combination of > > > > > providing > > > > > >>>> predefined appenders > > > > > >>>> and the PropertyConfigurator capabilities meets the need. > > > > > >>>> > > > > > >>>> Ram > > > > > >>>> > > > > > >>>> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko < > > > > > ser...@datatorrent.com > > > > > >>>> > > > > > > >>>> wrote: > > > > > >>>> > > > > > >>>> Ram, > > > > > >>>> > > > > > >>>>> Really the new appender class must extend the abstract class > > > > > >>>>> AppenderSkeleton. And in order to add a new appender > > > > programmatically > > > > > >>>>> in > > > > > >>>>> Java, some code in Apex should call the following log4j > method: > > > > > >>>>> > > > > > >>>>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender > > > > > >>>>> newAppender) > > > > > >>>>> > > > > > >>>>> The general idea of my proposal is "*based on some runtime > > > > > parameter(s) > > > > > >>>>> to > > > > > >>>>> provide ability to create an appender instance via reflection > > and > > > > add > > > > > >>>>> it > > > > > >>>>> to > > > > > >>>>> the list of active log4j appenders*". > > > > > >>>>> > > > > > >>>>> Thanks, > > > > > >>>>> Sergey > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov < > > > > v.ro...@datatorrent.com > > > > > > > > > > > >>>>> wrote: > > > > > >>>>> > > > > > >>>>> It will require application recompilation and repackaging. > The > > > > > proposed > > > > > >>>>> > > > > > >>>>>> functionality is for dev-ops to be able to route application > > > > logging > > > > > >>>>>> to > > > > > >>>>>> a > > > > > >>>>>> preferred destination without recompiling applications. It > is > > > > > run-time > > > > > >>>>>> configuration vs compile time hardcoded appender. > > > > > >>>>>> > > > > > >>>>>> Thank you, > > > > > >>>>>> > > > > > >>>>>> Vlad > > > > > >>>>>> > > > > > >>>>>> On 4/10/17 11:23, Munagala Ramanath wrote: > > > > > >>>>>> > > > > > >>>>>> You can do it in a trivial derived class without changing > the > > > base > > > > > >>>>>> class. > > > > > >>>>>> Ram > > > > > >>>>>> > > > > > >>>>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov < > > > > > >>>>>>> v.ro...@datatorrent.com> > > > > > >>>>>>> wrote: > > > > > >>>>>>> > > > > > >>>>>>> Does not the proposal to use Logger.addAppender() requires > > > > > >>>>>>> modifications > > > > > >>>>>>> > > > > > >>>>>>> to used operators code? > > > > > >>>>>>>> > > > > > >>>>>>>> Thank you, > > > > > >>>>>>>> > > > > > >>>>>>>> Vlad > > > > > >>>>>>>> > > > > > >>>>>>>> On 4/10/17 10:58, Munagala Ramanath wrote: > > > > > >>>>>>>> > > > > > >>>>>>>> People can currently do this by simply implementing the > > > Appender > > > > > >>>>>>>> > > > > > >>>>>>>> interface > > > > > >>>>>>>>> and adding it > > > > > >>>>>>>>> with Logger.addAppender() in the setup method. Why do we > > need > > > > > >>>>>>>>> > > > > > >>>>>>>>> something > > > > > >>>>>>>> > > > > > >>>>>>> more elaborate ? > > > > > >>>>>> > > > > > >>>>>>> Ram > > > > > >>>>>>>>> > > > > > >>>>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko < > > > > > >>>>>>>>> ser...@datatorrent.com> > > > > > >>>>>>>>> wrote: > > > > > >>>>>>>>> > > > > > >>>>>>>>> The configuration of a log4j appender via log4j > > configuration > > > > > file > > > > > >>>>>>>>> is > > > > > >>>>>>>>> > > > > > >>>>>>>>> a > > > > > >>>>>>>> > > > > > >>>>>>> static configuration that cannot be disabled/enabled and > > > managed > > > > > >>>>>> > > > > > >>>>>>> dynamically by an application designer. The programmatic > > > approach > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> will > > > > > >>>>>>>>> > > > > > >>>>>>>> allow an application designer to specify which of the > > > available > > > > > >>>>>> > > > > > >>>>>>> log4j > > > > > >>>>>>>>> > > > > > >>>>>>>> appenders should be used for the specific application. > > > > > >>>>>> > > > > > >>>>>>> It is not necessary Apex should use the predefined log4j > > > > appenders > > > > > >>>>>>>>>> only. > > > > > >>>>>>>>>> The log4j events contain useful but the very limited > > number > > > of > > > > > >>>>>>>>>> properties > > > > > >>>>>>>>>> which values can be printed into output log4j sources. > But > > > > based > > > > > >>>>>>>>>> on > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> the > > > > > >>>>>>>>> > > > > > >>>>>>>> knowledge of the software product workflow, the custom > > defined > > > > > log4j > > > > > >>>>>> > > > > > >>>>>>> appender can extend a list of predefined output log events > > > > > >>>>>>>>>> properties > > > > > >>>>>>>>>> and, > > > > > >>>>>>>>>> for instance for Apex, return: node, user name, > > application > > > > > name, > > > > > >>>>>>>>>> application id, container id, operator name, etc. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Also the output log events that are generated by a > custom > > > > > defined > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> log4j > > > > > >>>>>>>>> > > > > > >>>>>>>> appender can be stored and indexed by any type of a full > > text > > > > > search > > > > > >>>>>> > > > > > >>>>>>> database. It will allow the customers and developers to > > > simplify > > > > > >>>>>>>>>> collection > > > > > >>>>>>>>>> of log events statistics and searching/filtering of > > specific > > > > > >>>>>>>>>> events > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> for > > > > > >>>>>>>>> > > > > > >>>>>>>> debugging and investigation. > > > > > >>>>>> > > > > > >>>>>>> Thanks, > > > > > >>>>>>>>>> Sergey > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov < > > > > > >>>>>>>>>> v.ro...@datatorrent.com > > > > > >>>>>>>>>> wrote: > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> +1 Apex engine does not own log4j config file - it is > > > provided > > > > > >>>>>>>>>> either > > > > > >>>>>>>>>> by > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> Hadoop or an application. Hadoop log4j config does not > > > > > necessarily > > > > > >>>>>>>>>> > > > > > >>>>>>>>>>> meet > > > > > >>>>>>>>>>> application logging requirements, but if log4j is > > provided > > > by > > > > > an > > > > > >>>>>>>>>>> application designer, who can only specify what to log, > > it > > > > may > > > > > >>>>>>>>>>> not > > > > > >>>>>>>>>>> meet > > > > > >>>>>>>>>>> operations requirements. Dev-ops should have an ability > > to > > > > > >>>>>>>>>>> specify > > > > > >>>>>>>>>>> where > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> to > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> log depending on the available infrastructure at > > run-time. > > > > > >>>>>>>>>> > > > > > >>>>>>>>>> It will be good to have an ability not only specify > extra > > > > log4j > > > > > >>>>>>>>>>> appenders > > > > > >>>>>>>>>>> at lunch time, but also at run-time, the same way how > > log4j > > > > > >>>>>>>>>>> logger > > > > > >>>>>>>>>>> levels > > > > > >>>>>>>>>>> may be changed. > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Thank you, > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> Vlad > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote: > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> We can always write a custom appender and add it by > > > changing > > > > > root > > > > > >>>>>>>>>>> appender > > > > > >>>>>>>>>>> in log4j config file. Can you explain how adding > appender > > > > > >>>>>>>>>>> grammatically > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>> would help? > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>>>> -Priyanka > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare < > > > > > >>>>>>>>>>>> san...@datatorrent.com > > > > > >>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> Please give some examples and/or use cases of this > > > > > programmatic > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> log4j > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>> appender. > > > > > >>>>>> > > > > > >>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko < > > > > > >>>>>>>>>>>>> ser...@datatorrent.com > > > > > >>>>>>>>>>>>> wrote: > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Hi All, > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> I'd like to add supporting of a custom defined log4j > > > > appender > > > > > >>>>>>>>>>>>> that > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> can > > > > > >>>>>>>>>>>>>> be > > > > > >>>>>>>>>>>>>> added to Apex Application Master and Containers and > be > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> configurable > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> programmatically. > > > > > >>>>>> > > > > > >>>>>>> Sometimes it is not trivial to control log4j configuration > > via > > > > > >>>>>>>>>>>>>> log4j > > > > > >>>>>>>>>>>>>> properties. And I think the having of the approach > to > > > add > > > > a > > > > > >>>>>>>>>>>>>> log4j > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> appender > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> programmatically will allow the customers and > > developers > > > > to > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> plugin > > > > > >>>>>>>>>>>>> their > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> own custom defined log4j appenders and be much > flexible > > > for > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> streaming > > > > > >>>>>>>>>>> > > > > > >>>>>>>>>> and > > > > > >>>>>> > > > > > >>>>>>> collection of Apex log events. > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> I assume to provide generic approach for definition > of > > > the > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> programmatic > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> log4j appender and to pass all configuration > > parameters > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>> including a > > > > > >>>>>>>>>>>> name > > > > > >>>>>>>>>>>> of > > > > > >>>>>>>>>>>> the Java class with implementation of the log4j > appender > > > via > > > > > >>>>>>>>>>>> system > > > > > >>>>>>>>>>>> > > > > > >>>>>>>>>>>> and/or > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> command line properties. > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>> Thanks, > > > > > >>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> Sergey > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>>>>>>>>>>>> > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > _______________________________________________________ > > > > > > > > > > Munagala V. Ramanath > > > > > > > > > > Software Engineer > > > > > > > > > > E: r...@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam > > > > > > > > > > www.datatorrent.com | apex.apache.org > > > > > > > > > > > > > > >
