Yes, totally agree, it would be helpful to have a detailed use case and/or a detailed spec of the desired capabilities -- not necessarily a complete spec but with enough detail to understand why existing capabilities are inadequate.
Ram On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <[email protected]> wrote: > It will be good to understand a use case where an operator needs a > specific appender. > > IMO, an operator designer defines *what* should be logged and dev-ops team > defines *where* to log. > > Thank you, > > Vlad > On 4/10/17 16:27, Munagala Ramanath wrote: > >> Yes, I understand, I was just wondering if individual operators could >> define the appenders >> they potentially need at compile time and then the operator callbacks >> could >> simply >> check the desired runtime condition and add the appropriate appender. >> >> Or are we saying there are scenarios where we absolutely cannot create the >> appender beforehand ? >> >> So broadly speaking, my question is whether the combination of providing >> predefined appenders >> and the PropertyConfigurator capabilities meets the need. >> >> Ram >> >> On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <[email protected]> >> wrote: >> >> Ram, >>> >>> Really the new appender class must extend the abstract class >>> AppenderSkeleton. And in order to add a new appender programmatically in >>> Java, some code in Apex should call the following log4j method: >>> >>> org.apache.log4j.Logger.getRootLogger().addAppender(Appender >>> newAppender) >>> >>> The general idea of my proposal is "*based on some runtime parameter(s) >>> to >>> provide ability to create an appender instance via reflection and add it >>> to >>> the list of active log4j appenders*". >>> >>> Thanks, >>> Sergey >>> >>> >>> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <[email protected]> >>> wrote: >>> >>> It will require application recompilation and repackaging. The proposed >>>> functionality is for dev-ops to be able to route application logging to >>>> a >>>> preferred destination without recompiling applications. It is run-time >>>> configuration vs compile time hardcoded appender. >>>> >>>> Thank you, >>>> >>>> Vlad >>>> >>>> On 4/10/17 11:23, Munagala Ramanath wrote: >>>> >>>> You can do it in a trivial derived class without changing the base >>>>> >>>> class. >>> >>>> Ram >>>>> >>>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <[email protected]> >>>>> wrote: >>>>> >>>>> Does not the proposal to use Logger.addAppender() requires >>>>> modifications >>>>> >>>>>> to used operators code? >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Vlad >>>>>> >>>>>> On 4/10/17 10:58, Munagala Ramanath wrote: >>>>>> >>>>>> People can currently do this by simply implementing the Appender >>>>>> >>>>>>> interface >>>>>>> and adding it >>>>>>> with Logger.addAppender() in the setup method. Why do we need >>>>>>> >>>>>> something >>> >>>> more elaborate ? >>>>>>> >>>>>>> Ram >>>>>>> >>>>>>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko < >>>>>>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> The configuration of a log4j appender via log4j configuration file is >>>>>>> >>>>>> a >>> >>>> static configuration that cannot be disabled/enabled and managed >>>>>>>> dynamically by an application designer. The programmatic approach >>>>>>>> >>>>>>> will >>> >>>> allow an application designer to specify which of the available >>>>>>>> >>>>>>> log4j >>> >>>> appenders should be used for the specific application. >>>>>>>> >>>>>>>> It is not necessary Apex should use the predefined log4j appenders >>>>>>>> only. >>>>>>>> The log4j events contain useful but the very limited number of >>>>>>>> properties >>>>>>>> which values can be printed into output log4j sources. But based on >>>>>>>> >>>>>>> the >>> >>>> knowledge of the software product workflow, the custom defined log4j >>>>>>>> appender can extend a list of predefined output log events >>>>>>>> properties >>>>>>>> and, >>>>>>>> for instance for Apex, return: node, user name, application name, >>>>>>>> application id, container id, operator name, etc. >>>>>>>> >>>>>>>> Also the output log events that are generated by a custom defined >>>>>>>> >>>>>>> log4j >>> >>>> appender can be stored and indexed by any type of a full text search >>>>>>>> database. It will allow the customers and developers to simplify >>>>>>>> collection >>>>>>>> of log events statistics and searching/filtering of specific events >>>>>>>> >>>>>>> for >>> >>>> debugging and investigation. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sergey >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov < >>>>>>>> [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>> +1 Apex engine does not own log4j config file - it is provided >>>>>>>> either >>>>>>>> by >>>>>>>> >>>>>>>> Hadoop or an application. Hadoop log4j config does not necessarily >>>>>>>>> meet >>>>>>>>> application logging requirements, but if log4j is provided by an >>>>>>>>> application designer, who can only specify what to log, it may not >>>>>>>>> meet >>>>>>>>> operations requirements. Dev-ops should have an ability to specify >>>>>>>>> where >>>>>>>>> >>>>>>>>> to >>>>>>>>> >>>>>>>> log depending on the available infrastructure at run-time. >>>>>>>> >>>>>>>>> It will be good to have an ability not only specify extra log4j >>>>>>>>> appenders >>>>>>>>> at lunch time, but also at run-time, the same way how log4j logger >>>>>>>>> levels >>>>>>>>> may be changed. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> >>>>>>>>> Vlad >>>>>>>>> >>>>>>>>> On 4/9/17 23:14, Priyanka Gugale wrote: >>>>>>>>> >>>>>>>>> We can always write a custom appender and add it by changing root >>>>>>>>> appender >>>>>>>>> in log4j config file. Can you explain how adding appender >>>>>>>>> grammatically >>>>>>>>> >>>>>>>>> would help? >>>>>>>>>> >>>>>>>>>> -Priyanka >>>>>>>>>> >>>>>>>>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare < >>>>>>>>>> [email protected] >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Please give some examples and/or use cases of this programmatic >>>>>>>>>> >>>>>>>>> log4j >>> >>>> appender. >>>>>>>>>> >>>>>>>>>>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko < >>>>>>>>>>> [email protected] >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> I'd like to add supporting of a custom defined log4j appender >>>>>>>>>>> that >>>>>>>>>>> >>>>>>>>>>>> can >>>>>>>>>>>> be >>>>>>>>>>>> added to Apex Application Master and Containers and be >>>>>>>>>>>> >>>>>>>>>>> configurable >>> >>>> programmatically. >>>>>>>>>>>> >>>>>>>>>>>> Sometimes it is not trivial to control log4j configuration via >>>>>>>>>>>> log4j >>>>>>>>>>>> properties. And I think the having of the approach to add a >>>>>>>>>>>> log4j >>>>>>>>>>>> >>>>>>>>>>>> appender >>>>>>>>>>>> >>>>>>>>>>>> programmatically will allow the customers and developers to >>>>>>>>>>> plugin >>>>>>>>>>> their >>>>>>>>>>> >>>>>>>>>>> own custom defined log4j appenders and be much flexible for >>>>>>>>>> >>>>>>>>> streaming >>> >>>> and >>>>>>>>>> >>>>>>>>>>> collection of Apex log events. >>>>>>>>>>>> >>>>>>>>>>>> I assume to provide generic approach for definition of the >>>>>>>>>>>> >>>>>>>>>>>> programmatic >>>>>>>>>>>> >>>>>>>>>>> log4j appender and to pass all configuration parameters >>>>>>>>>> including a >>>>>>>>>> name >>>>>>>>>> of >>>>>>>>>> the Java class with implementation of the log4j appender via >>>>>>>>>> system >>>>>>>>>> >>>>>>>>>>> and/or >>>>>>>>>>>> >>>>>>>>>>>> command line properties. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>>> Sergey >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >> >> > -- _______________________________________________________ Munagala V. Ramanath Software Engineer E: [email protected] | M: (408) 331-5034 | Twitter: @UnknownRam www.datatorrent.com | apex.apache.org
