Re: Programmatic log4j appender in Apex

[Sergey Golovko]

Hi Thomas,

I assume to have maximum generic implementation. The signatures of all
methods in the implementation will not contain anything log4j specific. And
if we decide to add an abstraction layer to the logger calls in Apex or to
use another logger implementation, it can be easily changed to support any
new appender interfaces.

Thanks,
Sergey


On Sat, Apr 15, 2017 at 4:00 PM, Thomas Weise  wrote:

> Hi Sergey,
>
> What I'm asking is that the feature is implemented in a way that will allow
> Apex to run with different logger backend. That means that log4j needs to
> be optional.
>
> Thanks,
> Thomas
>
>
> On Sat, Apr 15, 2017 at 2:01 PM, Sergey Golovko 
> wrote:
>
> > I agree it would be very nice to use only slf4j interfaces for the
> > implementation. But unfortunately the interface Appender belongs to
> > org.apache.log4j package.
> >
> > "SLF4J is only a facade, meaning that it does not provide a complete
> > logging solution. Operations such as configuring appenders or setting
> > logging levels cannot be performed with SLF4J. Thus, at some point in
> time,
> > any non-trivial application will need to directly invoke the underlying
> > logging system. In other words, complete independence from the API
> > underlying logging system is not possible for a stand-alone application.
> > Nevertheless, SLF4J reduces the impact of this dependence to
> near-painless
> > levels."
> >
> > https://www.slf4j.org/faq.html#when
> >
> > Thanks,
> > Sergey
> >
> >
> > On Thu, Apr 13, 2017 at 7:56 AM, Thomas Weise  wrote:
> >
> > > +1
> > >
> > > Also the proposed feature would need to be implemented in a way that
> > avoids
> > > a hard dependency on log4j. The interface for logging is slf4j and it
> > > should be possible to use other logger backends.
> > >
> > >
> > > On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko <
> ser...@datatorrent.com>
> > > wrote:
> > >
> > > > I don't think an operator needs a specific appender. An appender can
> be
> > > > dynamically assigned to an application designer, application master
> and
> > > > container.
> > > >
> > > > Thanks,
> > > > Sergey
> > > >
> > > >
> > > > On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath <
> > r...@datatorrent.com>
> > > > wrote:
> > > >
> > > > > I don't have one, I thought that was what the intent of the
> proposal
> > > was,
> > > > > but looks like
> > > > > I misunderstood. After re-reading some of the earlier responses, I
> > > > > understand the
> > > > > proposal better.
> > > > >
> > > > > Ram
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov <
> v.ro...@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I don't see a use case where an individual operators need to
> > define a
> > > > > > specific appender, can you provide one?
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Vlad
> > > > > >
> > > > > > On 4/10/17 16:53, Munagala Ramanath wrote:
> > > > > >
> > > > > >> Yes, totally agree, it would be helpful to have a detailed use
> > case
> > > > > and/or
> > > > > >> a detailed spec
> > > > > >> of the desired capabilities -- not necessarily a complete spec
> but
> > > > with
> > > > > >> enough detail to
> > > > > >> understand why existing capabilities are inadequate.
> > > > > >>
> > > > > >> Ram
> > > > > >>
> > > > > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <
> > > v.ro...@datatorrent.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> It will be good to understand a use case where an operator
> needs a
> > > > > >>> specific appender.
> > > > > >>>
> > > > > >>> IMO, an operator designer defines *what* should be logged and
> > > dev-ops
> > > > > >>> team
> > > > > >>> defines *where* to log.
> > > > > >>>
> > > > > >>> Thank you,
> > > > > >>>
> > > > > >>> Vlad
> > > > > >>> On 4/10/17 16:27, Munagala Ramanath wrote:
> > > > > >>>
> > > > > >>> Yes, I understand, I was just wondering if individual operators
> > > could
> > > > >  define the appenders
> > > > >  they potentially need at compile time and then the operator
> > > > callbacks
> > > > >  could
> > > > >  simply
> > > > >  check the desired runtime condition and add the appropriate
> > > > appender.
> > > > > 
> > > > >  Or are we saying there are scenarios where we absolutely
> cannot
> > > > create
> > > > >  the
> > > > >  appender beforehand ?
> > > > > 
> > > > >  So broadly speaking, my question is whether the combination of
> > > > > providing
> > > > >  predefined appenders
> > > > >  and the PropertyConfigurator capabilities meets the need.
> > > > > 
> > > > >  Ram
> > > > > 
> > > > >  On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <
> > > > > ser...@datatorrent.com
> > > > >  >
> > > > >  wrote:
> > > > > 
> > > > >  Ram,
> > > > > 
> > > > > > Really the new appender class must extend the abstract class
> > > > > > AppenderSkeleton. And in 

Re: Programmatic log4j appender in Apex

[Thomas Weise]

Hi Sergey,

What I'm asking is that the feature is implemented in a way that will allow
Apex to run with different logger backend. That means that log4j needs to
be optional.

Thanks,
Thomas


On Sat, Apr 15, 2017 at 2:01 PM, Sergey Golovko 
wrote:

> I agree it would be very nice to use only slf4j interfaces for the
> implementation. But unfortunately the interface Appender belongs to
> org.apache.log4j package.
>
> "SLF4J is only a facade, meaning that it does not provide a complete
> logging solution. Operations such as configuring appenders or setting
> logging levels cannot be performed with SLF4J. Thus, at some point in time,
> any non-trivial application will need to directly invoke the underlying
> logging system. In other words, complete independence from the API
> underlying logging system is not possible for a stand-alone application.
> Nevertheless, SLF4J reduces the impact of this dependence to near-painless
> levels."
>
> https://www.slf4j.org/faq.html#when
>
> Thanks,
> Sergey
>
>
> On Thu, Apr 13, 2017 at 7:56 AM, Thomas Weise  wrote:
>
> > +1
> >
> > Also the proposed feature would need to be implemented in a way that
> avoids
> > a hard dependency on log4j. The interface for logging is slf4j and it
> > should be possible to use other logger backends.
> >
> >
> > On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko 
> > wrote:
> >
> > > I don't think an operator needs a specific appender. An appender can be
> > > dynamically assigned to an application designer, application master and
> > > container.
> > >
> > > Thanks,
> > > Sergey
> > >
> > >
> > > On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath <
> r...@datatorrent.com>
> > > wrote:
> > >
> > > > I don't have one, I thought that was what the intent of the proposal
> > was,
> > > > but looks like
> > > > I misunderstood. After re-reading some of the earlier responses, I
> > > > understand the
> > > > proposal better.
> > > >
> > > > Ram
> > > >
> > > >
> > > >
> > > > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov  >
> > > > wrote:
> > > >
> > > > > I don't see a use case where an individual operators need to
> define a
> > > > > specific appender, can you provide one?
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Vlad
> > > > >
> > > > > On 4/10/17 16:53, Munagala Ramanath wrote:
> > > > >
> > > > >> Yes, totally agree, it would be helpful to have a detailed use
> case
> > > > and/or
> > > > >> a detailed spec
> > > > >> of the desired capabilities -- not necessarily a complete spec but
> > > with
> > > > >> enough detail to
> > > > >> understand why existing capabilities are inadequate.
> > > > >>
> > > > >> Ram
> > > > >>
> > > > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <
> > v.ro...@datatorrent.com>
> > > > >> wrote:
> > > > >>
> > > > >> It will be good to understand a use case where an operator needs a
> > > > >>> specific appender.
> > > > >>>
> > > > >>> IMO, an operator designer defines *what* should be logged and
> > dev-ops
> > > > >>> team
> > > > >>> defines *where* to log.
> > > > >>>
> > > > >>> Thank you,
> > > > >>>
> > > > >>> Vlad
> > > > >>> On 4/10/17 16:27, Munagala Ramanath wrote:
> > > > >>>
> > > > >>> Yes, I understand, I was just wondering if individual operators
> > could
> > > >  define the appenders
> > > >  they potentially need at compile time and then the operator
> > > callbacks
> > > >  could
> > > >  simply
> > > >  check the desired runtime condition and add the appropriate
> > > appender.
> > > > 
> > > >  Or are we saying there are scenarios where we absolutely cannot
> > > create
> > > >  the
> > > >  appender beforehand ?
> > > > 
> > > >  So broadly speaking, my question is whether the combination of
> > > > providing
> > > >  predefined appenders
> > > >  and the PropertyConfigurator capabilities meets the need.
> > > > 
> > > >  Ram
> > > > 
> > > >  On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <
> > > > ser...@datatorrent.com
> > > >  >
> > > >  wrote:
> > > > 
> > > >  Ram,
> > > > 
> > > > > Really the new appender class must extend the abstract class
> > > > > AppenderSkeleton. And in order to add a new appender
> > > programmatically
> > > > > in
> > > > > Java, some code in Apex should call the following log4j method:
> > > > >
> > > > > org.apache.log4j.Logger.getRootLogger().addAppender(Appender
> > > > > newAppender)
> > > > >
> > > > > The general idea of my proposal is "*based on some runtime
> > > > parameter(s)
> > > > > to
> > > > > provide ability to create an appender instance via reflection
> and
> > > add
> > > > > it
> > > > > to
> > > > > the list of active log4j appenders*".
> > > > >
> > > > > Thanks,
> > > > > Sergey
> > > > >
> > > > >
> > > > > On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <
> > > 

Re: Programmatic log4j appender in Apex

[Sergey Golovko]

I agree it would be very nice to use only slf4j interfaces for the
implementation. But unfortunately the interface Appender belongs to
org.apache.log4j package.

"SLF4J is only a facade, meaning that it does not provide a complete
logging solution. Operations such as configuring appenders or setting
logging levels cannot be performed with SLF4J. Thus, at some point in time,
any non-trivial application will need to directly invoke the underlying
logging system. In other words, complete independence from the API
underlying logging system is not possible for a stand-alone application.
Nevertheless, SLF4J reduces the impact of this dependence to near-painless
levels."

https://www.slf4j.org/faq.html#when

Thanks,
Sergey


On Thu, Apr 13, 2017 at 7:56 AM, Thomas Weise  wrote:

> +1
>
> Also the proposed feature would need to be implemented in a way that avoids
> a hard dependency on log4j. The interface for logging is slf4j and it
> should be possible to use other logger backends.
>
>
> On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko 
> wrote:
>
> > I don't think an operator needs a specific appender. An appender can be
> > dynamically assigned to an application designer, application master and
> > container.
> >
> > Thanks,
> > Sergey
> >
> >
> > On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath 
> > wrote:
> >
> > > I don't have one, I thought that was what the intent of the proposal
> was,
> > > but looks like
> > > I misunderstood. After re-reading some of the earlier responses, I
> > > understand the
> > > proposal better.
> > >
> > > Ram
> > >
> > >
> > >
> > > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov 
> > > wrote:
> > >
> > > > I don't see a use case where an individual operators need to define a
> > > > specific appender, can you provide one?
> > > >
> > > > Thank you,
> > > >
> > > > Vlad
> > > >
> > > > On 4/10/17 16:53, Munagala Ramanath wrote:
> > > >
> > > >> Yes, totally agree, it would be helpful to have a detailed use case
> > > and/or
> > > >> a detailed spec
> > > >> of the desired capabilities -- not necessarily a complete spec but
> > with
> > > >> enough detail to
> > > >> understand why existing capabilities are inadequate.
> > > >>
> > > >> Ram
> > > >>
> > > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov <
> v.ro...@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> It will be good to understand a use case where an operator needs a
> > > >>> specific appender.
> > > >>>
> > > >>> IMO, an operator designer defines *what* should be logged and
> dev-ops
> > > >>> team
> > > >>> defines *where* to log.
> > > >>>
> > > >>> Thank you,
> > > >>>
> > > >>> Vlad
> > > >>> On 4/10/17 16:27, Munagala Ramanath wrote:
> > > >>>
> > > >>> Yes, I understand, I was just wondering if individual operators
> could
> > >  define the appenders
> > >  they potentially need at compile time and then the operator
> > callbacks
> > >  could
> > >  simply
> > >  check the desired runtime condition and add the appropriate
> > appender.
> > > 
> > >  Or are we saying there are scenarios where we absolutely cannot
> > create
> > >  the
> > >  appender beforehand ?
> > > 
> > >  So broadly speaking, my question is whether the combination of
> > > providing
> > >  predefined appenders
> > >  and the PropertyConfigurator capabilities meets the need.
> > > 
> > >  Ram
> > > 
> > >  On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <
> > > ser...@datatorrent.com
> > >  >
> > >  wrote:
> > > 
> > >  Ram,
> > > 
> > > > Really the new appender class must extend the abstract class
> > > > AppenderSkeleton. And in order to add a new appender
> > programmatically
> > > > in
> > > > Java, some code in Apex should call the following log4j method:
> > > >
> > > > org.apache.log4j.Logger.getRootLogger().addAppender(Appender
> > > > newAppender)
> > > >
> > > > The general idea of my proposal is "*based on some runtime
> > > parameter(s)
> > > > to
> > > > provide ability to create an appender instance via reflection and
> > add
> > > > it
> > > > to
> > > > the list of active log4j appenders*".
> > > >
> > > > Thanks,
> > > > Sergey
> > > >
> > > >
> > > > On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <
> > v.ro...@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > It will require application recompilation and repackaging. The
> > > proposed
> > > >
> > > >> functionality is for dev-ops to be able to route application
> > logging
> > > >> to
> > > >> a
> > > >> preferred destination without recompiling applications. It is
> > > run-time
> > > >> configuration vs compile time hardcoded appender.
> > > >>
> > > >> Thank you,
> > > >>
> > > >> Vlad
> > > >>
> > > >> On 4/10/17 11:23, Munagala Ramanath wrote:
> > > 

Re: Programmatic log4j appender in Apex

[Thomas Weise]

+1

Also the proposed feature would need to be implemented in a way that avoids
a hard dependency on log4j. The interface for logging is slf4j and it
should be possible to use other logger backends.


On Mon, Apr 10, 2017 at 9:21 PM, Sergey Golovko 
wrote:

> I don't think an operator needs a specific appender. An appender can be
> dynamically assigned to an application designer, application master and
> container.
>
> Thanks,
> Sergey
>
>
> On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath 
> wrote:
>
> > I don't have one, I thought that was what the intent of the proposal was,
> > but looks like
> > I misunderstood. After re-reading some of the earlier responses, I
> > understand the
> > proposal better.
> >
> > Ram
> >
> >
> >
> > On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov 
> > wrote:
> >
> > > I don't see a use case where an individual operators need to define a
> > > specific appender, can you provide one?
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > > On 4/10/17 16:53, Munagala Ramanath wrote:
> > >
> > >> Yes, totally agree, it would be helpful to have a detailed use case
> > and/or
> > >> a detailed spec
> > >> of the desired capabilities -- not necessarily a complete spec but
> with
> > >> enough detail to
> > >> understand why existing capabilities are inadequate.
> > >>
> > >> Ram
> > >>
> > >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov 
> > >> wrote:
> > >>
> > >> It will be good to understand a use case where an operator needs a
> > >>> specific appender.
> > >>>
> > >>> IMO, an operator designer defines *what* should be logged and dev-ops
> > >>> team
> > >>> defines *where* to log.
> > >>>
> > >>> Thank you,
> > >>>
> > >>> Vlad
> > >>> On 4/10/17 16:27, Munagala Ramanath wrote:
> > >>>
> > >>> Yes, I understand, I was just wondering if individual operators could
> >  define the appenders
> >  they potentially need at compile time and then the operator
> callbacks
> >  could
> >  simply
> >  check the desired runtime condition and add the appropriate
> appender.
> > 
> >  Or are we saying there are scenarios where we absolutely cannot
> create
> >  the
> >  appender beforehand ?
> > 
> >  So broadly speaking, my question is whether the combination of
> > providing
> >  predefined appenders
> >  and the PropertyConfigurator capabilities meets the need.
> > 
> >  Ram
> > 
> >  On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <
> > ser...@datatorrent.com
> >  >
> >  wrote:
> > 
> >  Ram,
> > 
> > > Really the new appender class must extend the abstract class
> > > AppenderSkeleton. And in order to add a new appender
> programmatically
> > > in
> > > Java, some code in Apex should call the following log4j method:
> > >
> > > org.apache.log4j.Logger.getRootLogger().addAppender(Appender
> > > newAppender)
> > >
> > > The general idea of my proposal is "*based on some runtime
> > parameter(s)
> > > to
> > > provide ability to create an appender instance via reflection and
> add
> > > it
> > > to
> > > the list of active log4j appenders*".
> > >
> > > Thanks,
> > > Sergey
> > >
> > >
> > > On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov <
> v.ro...@datatorrent.com
> > >
> > > wrote:
> > >
> > > It will require application recompilation and repackaging. The
> > proposed
> > >
> > >> functionality is for dev-ops to be able to route application
> logging
> > >> to
> > >> a
> > >> preferred destination without recompiling applications. It is
> > run-time
> > >> configuration vs compile time hardcoded appender.
> > >>
> > >> Thank you,
> > >>
> > >> Vlad
> > >>
> > >> On 4/10/17 11:23, Munagala Ramanath wrote:
> > >>
> > >> You can do it in a trivial derived class without changing the base
> > >> class.
> > >> Ram
> > >>
> > >>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <
> > >>> v.ro...@datatorrent.com>
> > >>> wrote:
> > >>>
> > >>> Does not the proposal to use Logger.addAppender() requires
> > >>> modifications
> > >>>
> > >>> to used operators code?
> > 
> >  Thank you,
> > 
> >  Vlad
> > 
> >  On 4/10/17 10:58, Munagala Ramanath wrote:
> > 
> >  People can currently do this by simply implementing the Appender
> > 
> >  interface
> > > and adding it
> > > with Logger.addAppender() in the setup method. Why do we need
> > >
> > > something
> > 
> > >>> more elaborate ?
> > >>
> > >>> Ram
> > >
> > > On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
> > > ser...@datatorrent.com>
> > > wrote:
> > >
> > > The configuration of a log4j appender via log4j 

Re: Programmatic log4j appender in Apex

[Sergey Golovko]

I don't think an operator needs a specific appender. An appender can be
dynamically assigned to an application designer, application master and
container.

Thanks,
Sergey


On Mon, Apr 10, 2017 at 6:26 PM, Munagala Ramanath 
wrote:

> I don't have one, I thought that was what the intent of the proposal was,
> but looks like
> I misunderstood. After re-reading some of the earlier responses, I
> understand the
> proposal better.
>
> Ram
>
>
>
> On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov 
> wrote:
>
> > I don't see a use case where an individual operators need to define a
> > specific appender, can you provide one?
> >
> > Thank you,
> >
> > Vlad
> >
> > On 4/10/17 16:53, Munagala Ramanath wrote:
> >
> >> Yes, totally agree, it would be helpful to have a detailed use case
> and/or
> >> a detailed spec
> >> of the desired capabilities -- not necessarily a complete spec but with
> >> enough detail to
> >> understand why existing capabilities are inadequate.
> >>
> >> Ram
> >>
> >> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov 
> >> wrote:
> >>
> >> It will be good to understand a use case where an operator needs a
> >>> specific appender.
> >>>
> >>> IMO, an operator designer defines *what* should be logged and dev-ops
> >>> team
> >>> defines *where* to log.
> >>>
> >>> Thank you,
> >>>
> >>> Vlad
> >>> On 4/10/17 16:27, Munagala Ramanath wrote:
> >>>
> >>> Yes, I understand, I was just wondering if individual operators could
>  define the appenders
>  they potentially need at compile time and then the operator callbacks
>  could
>  simply
>  check the desired runtime condition and add the appropriate appender.
> 
>  Or are we saying there are scenarios where we absolutely cannot create
>  the
>  appender beforehand ?
> 
>  So broadly speaking, my question is whether the combination of
> providing
>  predefined appenders
>  and the PropertyConfigurator capabilities meets the need.
> 
>  Ram
> 
>  On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko <
> ser...@datatorrent.com
>  >
>  wrote:
> 
>  Ram,
> 
> > Really the new appender class must extend the abstract class
> > AppenderSkeleton. And in order to add a new appender programmatically
> > in
> > Java, some code in Apex should call the following log4j method:
> >
> > org.apache.log4j.Logger.getRootLogger().addAppender(Appender
> > newAppender)
> >
> > The general idea of my proposal is "*based on some runtime
> parameter(s)
> > to
> > provide ability to create an appender instance via reflection and add
> > it
> > to
> > the list of active log4j appenders*".
> >
> > Thanks,
> > Sergey
> >
> >
> > On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov  >
> > wrote:
> >
> > It will require application recompilation and repackaging. The
> proposed
> >
> >> functionality is for dev-ops to be able to route application logging
> >> to
> >> a
> >> preferred destination without recompiling applications. It is
> run-time
> >> configuration vs compile time hardcoded appender.
> >>
> >> Thank you,
> >>
> >> Vlad
> >>
> >> On 4/10/17 11:23, Munagala Ramanath wrote:
> >>
> >> You can do it in a trivial derived class without changing the base
> >> class.
> >> Ram
> >>
> >>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <
> >>> v.ro...@datatorrent.com>
> >>> wrote:
> >>>
> >>> Does not the proposal to use Logger.addAppender() requires
> >>> modifications
> >>>
> >>> to used operators code?
> 
>  Thank you,
> 
>  Vlad
> 
>  On 4/10/17 10:58, Munagala Ramanath wrote:
> 
>  People can currently do this by simply implementing the Appender
> 
>  interface
> > and adding it
> > with Logger.addAppender() in the setup method. Why do we need
> >
> > something
> 
> >>> more elaborate ?
> >>
> >>> Ram
> >
> > On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
> > ser...@datatorrent.com>
> > wrote:
> >
> > The configuration of a log4j appender via log4j configuration
> file
> > is
> >
> > a
> 
> >>> static configuration that cannot be disabled/enabled and managed
> >>
> >>> dynamically by an application designer. The programmatic approach
> >>
> >> will
> >
>  allow  an application designer to specify which of the available
> >>
> >>> log4j
> >
>  appenders should be used for the specific application.
> >>
> >>> It is not necessary Apex should use the predefined log4j appenders
> >> only.
> >> The log4j events contain useful but 

Re: Programmatic log4j appender in Apex

[Munagala Ramanath]

I don't have one, I thought that was what the intent of the proposal was,
but looks like
I misunderstood. After re-reading some of the earlier responses, I
understand the
proposal better.

Ram



On Mon, Apr 10, 2017 at 5:39 PM, Vlad Rozov  wrote:

> I don't see a use case where an individual operators need to define a
> specific appender, can you provide one?
>
> Thank you,
>
> Vlad
>
> On 4/10/17 16:53, Munagala Ramanath wrote:
>
>> Yes, totally agree, it would be helpful to have a detailed use case and/or
>> a detailed spec
>> of the desired capabilities -- not necessarily a complete spec but with
>> enough detail to
>> understand why existing capabilities are inadequate.
>>
>> Ram
>>
>> On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov 
>> wrote:
>>
>> It will be good to understand a use case where an operator needs a
>>> specific appender.
>>>
>>> IMO, an operator designer defines *what* should be logged and dev-ops
>>> team
>>> defines *where* to log.
>>>
>>> Thank you,
>>>
>>> Vlad
>>> On 4/10/17 16:27, Munagala Ramanath wrote:
>>>
>>> Yes, I understand, I was just wondering if individual operators could
 define the appenders
 they potentially need at compile time and then the operator callbacks
 could
 simply
 check the desired runtime condition and add the appropriate appender.

 Or are we saying there are scenarios where we absolutely cannot create
 the
 appender beforehand ?

 So broadly speaking, my question is whether the combination of providing
 predefined appenders
 and the PropertyConfigurator capabilities meets the need.

 Ram

 On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko 
 wrote:

 Ram,

> Really the new appender class must extend the abstract class
> AppenderSkeleton. And in order to add a new appender programmatically
> in
> Java, some code in Apex should call the following log4j method:
>
> org.apache.log4j.Logger.getRootLogger().addAppender(Appender
> newAppender)
>
> The general idea of my proposal is "*based on some runtime parameter(s)
> to
> provide ability to create an appender instance via reflection and add
> it
> to
> the list of active log4j appenders*".
>
> Thanks,
> Sergey
>
>
> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov 
> wrote:
>
> It will require application recompilation and repackaging. The proposed
>
>> functionality is for dev-ops to be able to route application logging
>> to
>> a
>> preferred destination without recompiling applications. It is run-time
>> configuration vs compile time hardcoded appender.
>>
>> Thank you,
>>
>> Vlad
>>
>> On 4/10/17 11:23, Munagala Ramanath wrote:
>>
>> You can do it in a trivial derived class without changing the base
>> class.
>> Ram
>>
>>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov <
>>> v.ro...@datatorrent.com>
>>> wrote:
>>>
>>> Does not the proposal to use Logger.addAppender() requires
>>> modifications
>>>
>>> to used operators code?

 Thank you,

 Vlad

 On 4/10/17 10:58, Munagala Ramanath wrote:

 People can currently do this by simply implementing the Appender

 interface
> and adding it
> with Logger.addAppender() in the setup method. Why do we need
>
> something

>>> more elaborate ?
>>
>>> Ram
>
> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
> ser...@datatorrent.com>
> wrote:
>
> The configuration of a log4j appender via log4j configuration file
> is
>
> a

>>> static configuration that cannot be disabled/enabled and managed
>>
>>> dynamically by an application designer. The programmatic approach
>>
>> will
>
 allow  an application designer to specify which of the available
>>
>>> log4j
>
 appenders should be used for the specific application.
>>
>>> It is not necessary Apex should use the predefined log4j appenders
>> only.
>> The log4j events contain useful but the very limited number of
>> properties
>> which values can be printed into output log4j sources. But based
>> on
>>
>> the
>
 knowledge of the software product workflow, the custom defined log4j
>>
>>> appender can extend a list of predefined output log events
>> properties
>> and,
>> for instance for Apex, return: node, user name, application name,
>> application id, container id, operator name, etc.
>>
>> Also the output log events that are generated by a custom 

Re: Programmatic log4j appender in Apex

[Vlad Rozov]

I don't see a use case where an individual operators need to define a 
specific appender, can you provide one?


Thank you,

Vlad

On 4/10/17 16:53, Munagala Ramanath wrote:

Yes, totally agree, it would be helpful to have a detailed use case and/or
a detailed spec
of the desired capabilities -- not necessarily a complete spec but with
enough detail to
understand why existing capabilities are inadequate.

Ram

On Mon, Apr 10, 2017 at 4:43 PM, Vlad Rozov  wrote:


It will be good to understand a use case where an operator needs a
specific appender.

IMO, an operator designer defines *what* should be logged and dev-ops team
defines *where* to log.

Thank you,

Vlad
On 4/10/17 16:27, Munagala Ramanath wrote:


Yes, I understand, I was just wondering if individual operators could
define the appenders
they potentially need at compile time and then the operator callbacks
could
simply
check the desired runtime condition and add the appropriate appender.

Or are we saying there are scenarios where we absolutely cannot create the
appender beforehand ?

So broadly speaking, my question is whether the combination of providing
predefined appenders
and the PropertyConfigurator capabilities meets the need.

Ram

On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko 
wrote:

Ram,

Really the new appender class must extend the abstract class
AppenderSkeleton. And in order to add a new appender programmatically in
Java, some code in Apex should call the following log4j method:

org.apache.log4j.Logger.getRootLogger().addAppender(Appender
newAppender)

The general idea of my proposal is "*based on some runtime parameter(s)
to
provide ability to create an appender instance via reflection and add it
to
the list of active log4j appenders*".

Thanks,
Sergey


On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov 
wrote:

It will require application recompilation and repackaging. The proposed

functionality is for dev-ops to be able to route application logging to
a
preferred destination without recompiling applications. It is run-time
configuration vs compile time hardcoded appender.

Thank you,

Vlad

On 4/10/17 11:23, Munagala Ramanath wrote:

You can do it in a trivial derived class without changing the base
class.
Ram

On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov 
wrote:

Does not the proposal to use Logger.addAppender() requires
modifications


to used operators code?

Thank you,

Vlad

On 4/10/17 10:58, Munagala Ramanath wrote:

People can currently do this by simply implementing the Appender


interface
and adding it
with Logger.addAppender() in the setup method. Why do we need


something

more elaborate ?

Ram

On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
ser...@datatorrent.com>
wrote:

The configuration of a log4j appender via log4j configuration file is


a

static configuration that cannot be disabled/enabled and managed

dynamically by an application designer. The programmatic approach


will

allow  an application designer to specify which of the available

log4j

appenders should be used for the specific application.

It is not necessary Apex should use the predefined log4j appenders
only.
The log4j events contain useful but the very limited number of
properties
which values can be printed into output log4j sources. But based on


the

knowledge of the software product workflow, the custom defined log4j

appender can extend a list of predefined output log events
properties
and,
for instance for Apex, return: node, user name, application name,
application id, container id, operator name, etc.

Also the output log events that are generated by a custom defined


log4j

appender can be stored and indexed by any type of a full text search

database. It will allow the customers and developers to simplify
collection
of log events statistics and searching/filtering of specific events


for

debugging and investigation.

Thanks,
Sergey


On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov <
v.ro...@datatorrent.com
wrote:

+1 Apex engine does not own log4j config file - it is provided
either
by

Hadoop or an application. Hadoop log4j config does not necessarily

meet
application logging requirements, but if log4j is provided by an
application designer, who can only specify what to log, it may not
meet
operations requirements. Dev-ops should have an ability to specify
where

to


log depending on the available infrastructure at run-time.


It will be good to have an ability not only specify extra log4j
appenders
at lunch time, but also at run-time, the same way how log4j logger
levels
may be changed.

Thank you,

Vlad

On 4/9/17 23:14, Priyanka Gugale wrote:

We can always write a custom appender and add it by changing root
appender
in log4j config file. Can you explain how adding appender
grammatically

would help?

-Priyanka

On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
san...@datatorrent.com
wrote:

Please give some examples and/or use 

Re: Programmatic log4j appender in Apex

[Vlad Rozov]

It will be good to understand a use case where an operator needs a 
specific appender.


IMO, an operator designer defines *what* should be logged and dev-ops 
team defines *where* to log.


Thank you,

Vlad
On 4/10/17 16:27, Munagala Ramanath wrote:

Yes, I understand, I was just wondering if individual operators could
define the appenders
they potentially need at compile time and then the operator callbacks could
simply
check the desired runtime condition and add the appropriate appender.

Or are we saying there are scenarios where we absolutely cannot create the
appender beforehand ?

So broadly speaking, my question is whether the combination of providing
predefined appenders
and the PropertyConfigurator capabilities meets the need.

Ram

On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko 
wrote:


Ram,

Really the new appender class must extend the abstract class
AppenderSkeleton. And in order to add a new appender programmatically in
Java, some code in Apex should call the following log4j method:

org.apache.log4j.Logger.getRootLogger().addAppender(Appender newAppender)

The general idea of my proposal is "*based on some runtime parameter(s) to
provide ability to create an appender instance via reflection and add it to
the list of active log4j appenders*".

Thanks,
Sergey


On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov 
wrote:


It will require application recompilation and repackaging. The proposed
functionality is for dev-ops to be able to route application logging to a
preferred destination without recompiling applications. It is run-time
configuration vs compile time hardcoded appender.

Thank you,

Vlad

On 4/10/17 11:23, Munagala Ramanath wrote:


You can do it in a trivial derived class without changing the base

class.

Ram

On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov 
wrote:

Does not the proposal to use Logger.addAppender() requires modifications

to used operators code?

Thank you,

Vlad

On 4/10/17 10:58, Munagala Ramanath wrote:

People can currently do this by simply implementing the Appender

interface
and adding it
with Logger.addAppender() in the setup method. Why do we need

something

more elaborate ?

Ram

On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
ser...@datatorrent.com>
wrote:

The configuration of a log4j appender via log4j configuration file is

a

static configuration that cannot be disabled/enabled and managed
dynamically by an application designer. The programmatic approach

will

allow  an application designer to specify which of the available

log4j

appenders should be used for the specific application.

It is not necessary Apex should use the predefined log4j appenders
only.
The log4j events contain useful but the very limited number of
properties
which values can be printed into output log4j sources. But based on

the

knowledge of the software product workflow, the custom defined log4j
appender can extend a list of predefined output log events properties
and,
for instance for Apex, return: node, user name, application name,
application id, container id, operator name, etc.

Also the output log events that are generated by a custom defined

log4j

appender can be stored and indexed by any type of a full text search
database. It will allow the customers and developers to simplify
collection
of log events statistics and searching/filtering of specific events

for

debugging and investigation.

Thanks,
Sergey


On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov 

Re: Programmatic log4j appender in Apex

[Vlad Rozov]

How this functionality of log4j can be used in the Yarn environment? It 
sounds like a more complicated solutions compared to providing an 
ability to programmatically add an appender at launch time and/or run-time.


Thank you,

Vlad

On 4/10/17 16:18, Munagala Ramanath wrote:

http://logging.apache.org/log4j/1.2/faq.html#3.6

Log4j has the ability to dynamically reload changes to the properties file
via the
*configureAndWatch()* method of *PropertyConfigurator*. If this is already
baked in, could
devops simply change the properties file if we provide an enhanced set of
appenders
for the desired use cases ?

Ram

On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov  wrote:


It will require application recompilation and repackaging. The proposed
functionality is for dev-ops to be able to route application logging to a
preferred destination without recompiling applications. It is run-time
configuration vs compile time hardcoded appender.

Thank you,

Vlad
On 4/10/17 11:23, Munagala Ramanath wrote:


You can do it in a trivial derived class without changing the base class.

Ram

On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov 
wrote:

Does not the proposal to use Logger.addAppender() requires modifications

to used operators code?

Thank you,

Vlad

On 4/10/17 10:58, Munagala Ramanath wrote:

People can currently do this by simply implementing the Appender

interface
and adding it
with Logger.addAppender() in the setup method. Why do we need something
more elaborate ?

Ram

On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
ser...@datatorrent.com>
wrote:

The configuration of a log4j appender via log4j configuration file is a


static configuration that cannot be disabled/enabled and managed
dynamically by an application designer. The programmatic approach will
allow  an application designer to specify which of the available log4j
appenders should be used for the specific application.

It is not necessary Apex should use the predefined log4j appenders
only.
The log4j events contain useful but the very limited number of
properties
which values can be printed into output log4j sources. But based on the
knowledge of the software product workflow, the custom defined log4j
appender can extend a list of predefined output log events properties
and,
for instance for Apex, return: node, user name, application name,
application id, container id, operator name, etc.

Also the output log events that are generated by a custom defined log4j
appender can be stored and indexed by any type of a full text search
database. It will allow the customers and developers to simplify
collection
of log events statistics and searching/filtering of specific events for
debugging and investigation.

Thanks,
Sergey


On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov 
wrote:

+1 Apex engine does not own log4j config file - it is provided either
by


Hadoop or an application. Hadoop log4j config does not necessarily
meet
application logging requirements, but if log4j is provided by an
application designer, who can only specify what to log, it may not
meet
operations requirements. Dev-ops should have an ability to specify
where

to

log depending on the available infrastructure at run-time.

It will be good to have an ability not only specify extra log4j
appenders
at lunch time, but also at run-time, the same way how log4j logger
levels
may be changed.

Thank you,

Vlad

On 4/9/17 23:14, Priyanka Gugale wrote:

We can always write a custom appender and add it by changing root
appender
in log4j config file. Can you explain how adding appender
grammatically


would help?

-Priyanka

On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
san...@datatorrent.com
wrote:

Please give some examples and/or use cases of this programmatic log4j

appender.

On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
ser...@datatorrent.com
wrote:

Hi All,

I'd like to add supporting of a custom defined log4j appender that

can
be
added to Apex Application Master and Containers and be configurable
programmatically.

Sometimes it is not trivial to control log4j configuration via
log4j
properties. And I think the having of the approach to add a log4j

appender


programmatically will allow the customers and developers to plugin
their


own custom defined log4j appenders and be much flexible for streaming
and

collection of Apex log events.

I assume to provide generic approach for definition of the

programmatic

log4j appender and to pass all configuration parameters including a
name
of
the Java class with implementation of the log4j appender via system

and/or


command line properties.


Thanks,
Sergey

Re: Programmatic log4j appender in Apex

[Munagala Ramanath]

http://logging.apache.org/log4j/1.2/faq.html#3.6

Log4j has the ability to dynamically reload changes to the properties file
via the
*configureAndWatch()* method of *PropertyConfigurator*. If this is already
baked in, could
devops simply change the properties file if we provide an enhanced set of
appenders
for the desired use cases ?

Ram

On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov  wrote:

> It will require application recompilation and repackaging. The proposed
> functionality is for dev-ops to be able to route application logging to a
> preferred destination without recompiling applications. It is run-time
> configuration vs compile time hardcoded appender.
>
> Thank you,
>
> Vlad
> On 4/10/17 11:23, Munagala Ramanath wrote:
>
>> You can do it in a trivial derived class without changing the base class.
>>
>> Ram
>>
>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov 
>> wrote:
>>
>> Does not the proposal to use Logger.addAppender() requires modifications
>>> to used operators code?
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 4/10/17 10:58, Munagala Ramanath wrote:
>>>
>>> People can currently do this by simply implementing the Appender
 interface
 and adding it
 with Logger.addAppender() in the setup method. Why do we need something
 more elaborate ?

 Ram

 On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
 ser...@datatorrent.com>
 wrote:

 The configuration of a log4j appender via log4j configuration file is a

> static configuration that cannot be disabled/enabled and managed
> dynamically by an application designer. The programmatic approach will
> allow  an application designer to specify which of the available log4j
> appenders should be used for the specific application.
>
> It is not necessary Apex should use the predefined log4j appenders
> only.
> The log4j events contain useful but the very limited number of
> properties
> which values can be printed into output log4j sources. But based on the
> knowledge of the software product workflow, the custom defined log4j
> appender can extend a list of predefined output log events properties
> and,
> for instance for Apex, return: node, user name, application name,
> application id, container id, operator name, etc.
>
> Also the output log events that are generated by a custom defined log4j
> appender can be stored and indexed by any type of a full text search
> database. It will allow the customers and developers to simplify
> collection
> of log events statistics and searching/filtering of specific events for
> debugging and investigation.
>
> Thanks,
> Sergey
>
>
> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov 
> wrote:
>
> +1 Apex engine does not own log4j config file - it is provided either
> by
>
>> Hadoop or an application. Hadoop log4j config does not necessarily
>> meet
>> application logging requirements, but if log4j is provided by an
>> application designer, who can only specify what to log, it may not
>> meet
>> operations requirements. Dev-ops should have an ability to specify
>> where
>>
>> to
>
> log depending on the available infrastructure at run-time.
>>
>> It will be good to have an ability not only specify extra log4j
>> appenders
>> at lunch time, but also at run-time, the same way how log4j logger
>> levels
>> may be changed.
>>
>> Thank you,
>>
>> Vlad
>>
>> On 4/9/17 23:14, Priyanka Gugale wrote:
>>
>> We can always write a custom appender and add it by changing root
>> appender
>> in log4j config file. Can you explain how adding appender
>> grammatically
>>
>>> would help?
>>>
>>> -Priyanka
>>>
>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
>>> san...@datatorrent.com
>>> wrote:
>>>
>>> Please give some examples and/or use cases of this programmatic log4j
>>>
>>> appender.

 On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
 ser...@datatorrent.com
 wrote:

 Hi All,

 I'd like to add supporting of a custom defined log4j appender that
> can
> be
> added to Apex Application Master and Containers and be configurable
> programmatically.
>
> Sometimes it is not trivial to control log4j configuration via
> log4j
> properties. And I think the having of the approach to add a log4j
>
> appender
>
 programmatically will allow the customers and developers to plugin
 their

>>> own custom defined log4j appenders and be much flexible for streaming
>>
>>> and
> collection of Apex log events.
>
> I assume to provide 

Re: Programmatic log4j appender in Apex

[Pramod Immaneni]

As we are already doing dynamic log level changes and are talking about
dynamically adding appenders, does it make sense and technically feasible
to apply a delta log4j configuration dynamically (which can include log
levels + appeneders + ?) on top of the static configuration provided by the
system.

On Mon, Apr 10, 2017 at 2:18 PM, Sergey Golovko 
wrote:

> Ram,
>
> Really the new appender class must extend the abstract class
> AppenderSkeleton. And in order to add a new appender programmatically in
> Java, some code in Apex should call the following log4j method:
>
> org.apache.log4j.Logger.getRootLogger().addAppender(Appender newAppender)
>
> The general idea of my proposal is "*based on some runtime parameter(s) to
> provide ability to create an appender instance via reflection and add it to
> the list of active log4j appenders*".
>
> Thanks,
> Sergey
>
>
> On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov 
> wrote:
>
> > It will require application recompilation and repackaging. The proposed
> > functionality is for dev-ops to be able to route application logging to a
> > preferred destination without recompiling applications. It is run-time
> > configuration vs compile time hardcoded appender.
> >
> > Thank you,
> >
> > Vlad
> >
> > On 4/10/17 11:23, Munagala Ramanath wrote:
> >
> >> You can do it in a trivial derived class without changing the base
> class.
> >>
> >> Ram
> >>
> >> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov 
> >> wrote:
> >>
> >> Does not the proposal to use Logger.addAppender() requires modifications
> >>> to used operators code?
> >>>
> >>> Thank you,
> >>>
> >>> Vlad
> >>>
> >>> On 4/10/17 10:58, Munagala Ramanath wrote:
> >>>
> >>> People can currently do this by simply implementing the Appender
>  interface
>  and adding it
>  with Logger.addAppender() in the setup method. Why do we need
> something
>  more elaborate ?
> 
>  Ram
> 
>  On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
>  ser...@datatorrent.com>
>  wrote:
> 
>  The configuration of a log4j appender via log4j configuration file is
> a
> 
> > static configuration that cannot be disabled/enabled and managed
> > dynamically by an application designer. The programmatic approach
> will
> > allow  an application designer to specify which of the available
> log4j
> > appenders should be used for the specific application.
> >
> > It is not necessary Apex should use the predefined log4j appenders
> > only.
> > The log4j events contain useful but the very limited number of
> > properties
> > which values can be printed into output log4j sources. But based on
> the
> > knowledge of the software product workflow, the custom defined log4j
> > appender can extend a list of predefined output log events properties
> > and,
> > for instance for Apex, return: node, user name, application name,
> > application id, container id, operator name, etc.
> >
> > Also the output log events that are generated by a custom defined
> log4j
> > appender can be stored and indexed by any type of a full text search
> > database. It will allow the customers and developers to simplify
> > collection
> > of log events statistics and searching/filtering of specific events
> for
> > debugging and investigation.
> >
> > Thanks,
> > Sergey
> >
> >
> > On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov  >
> > wrote:
> >
> > +1 Apex engine does not own log4j config file - it is provided either
> > by
> >
> >> Hadoop or an application. Hadoop log4j config does not necessarily
> >> meet
> >> application logging requirements, but if log4j is provided by an
> >> application designer, who can only specify what to log, it may not
> >> meet
> >> operations requirements. Dev-ops should have an ability to specify
> >> where
> >>
> >> to
> >
> > log depending on the available infrastructure at run-time.
> >>
> >> It will be good to have an ability not only specify extra log4j
> >> appenders
> >> at lunch time, but also at run-time, the same way how log4j logger
> >> levels
> >> may be changed.
> >>
> >> Thank you,
> >>
> >> Vlad
> >>
> >> On 4/9/17 23:14, Priyanka Gugale wrote:
> >>
> >> We can always write a custom appender and add it by changing root
> >> appender
> >> in log4j config file. Can you explain how adding appender
> >> grammatically
> >>
> >>> would help?
> >>>
> >>> -Priyanka
> >>>
> >>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
> >>> san...@datatorrent.com
> >>> wrote:
> >>>
> >>> Please give some examples and/or use cases of this programmatic
> log4j
> >>>
> >>> appender.
> 
>  On Fri, Apr 7, 2017 at 

Re: Programmatic log4j appender in Apex

[Sergey Golovko]

Ram,

Really the new appender class must extend the abstract class
AppenderSkeleton. And in order to add a new appender programmatically in
Java, some code in Apex should call the following log4j method:

org.apache.log4j.Logger.getRootLogger().addAppender(Appender newAppender)

The general idea of my proposal is "*based on some runtime parameter(s) to
provide ability to create an appender instance via reflection and add it to
the list of active log4j appenders*".

Thanks,
Sergey


On Mon, Apr 10, 2017 at 2:04 PM, Vlad Rozov  wrote:

> It will require application recompilation and repackaging. The proposed
> functionality is for dev-ops to be able to route application logging to a
> preferred destination without recompiling applications. It is run-time
> configuration vs compile time hardcoded appender.
>
> Thank you,
>
> Vlad
>
> On 4/10/17 11:23, Munagala Ramanath wrote:
>
>> You can do it in a trivial derived class without changing the base class.
>>
>> Ram
>>
>> On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov 
>> wrote:
>>
>> Does not the proposal to use Logger.addAppender() requires modifications
>>> to used operators code?
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 4/10/17 10:58, Munagala Ramanath wrote:
>>>
>>> People can currently do this by simply implementing the Appender
 interface
 and adding it
 with Logger.addAppender() in the setup method. Why do we need something
 more elaborate ?

 Ram

 On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko <
 ser...@datatorrent.com>
 wrote:

 The configuration of a log4j appender via log4j configuration file is a

> static configuration that cannot be disabled/enabled and managed
> dynamically by an application designer. The programmatic approach will
> allow  an application designer to specify which of the available log4j
> appenders should be used for the specific application.
>
> It is not necessary Apex should use the predefined log4j appenders
> only.
> The log4j events contain useful but the very limited number of
> properties
> which values can be printed into output log4j sources. But based on the
> knowledge of the software product workflow, the custom defined log4j
> appender can extend a list of predefined output log events properties
> and,
> for instance for Apex, return: node, user name, application name,
> application id, container id, operator name, etc.
>
> Also the output log events that are generated by a custom defined log4j
> appender can be stored and indexed by any type of a full text search
> database. It will allow the customers and developers to simplify
> collection
> of log events statistics and searching/filtering of specific events for
> debugging and investigation.
>
> Thanks,
> Sergey
>
>
> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov 
> wrote:
>
> +1 Apex engine does not own log4j config file - it is provided either
> by
>
>> Hadoop or an application. Hadoop log4j config does not necessarily
>> meet
>> application logging requirements, but if log4j is provided by an
>> application designer, who can only specify what to log, it may not
>> meet
>> operations requirements. Dev-ops should have an ability to specify
>> where
>>
>> to
>
> log depending on the available infrastructure at run-time.
>>
>> It will be good to have an ability not only specify extra log4j
>> appenders
>> at lunch time, but also at run-time, the same way how log4j logger
>> levels
>> may be changed.
>>
>> Thank you,
>>
>> Vlad
>>
>> On 4/9/17 23:14, Priyanka Gugale wrote:
>>
>> We can always write a custom appender and add it by changing root
>> appender
>> in log4j config file. Can you explain how adding appender
>> grammatically
>>
>>> would help?
>>>
>>> -Priyanka
>>>
>>> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare <
>>> san...@datatorrent.com
>>> wrote:
>>>
>>> Please give some examples and/or use cases of this programmatic log4j
>>>
>>> appender.

 On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
 ser...@datatorrent.com
 wrote:

 Hi All,

 I'd like to add supporting of a custom defined log4j appender that
> can
> be
> added to Apex Application Master and Containers and be configurable
> programmatically.
>
> Sometimes it is not trivial to control log4j configuration via
> log4j
> properties. And I think the having of the approach to add a log4j
>
> appender
>
 programmatically will allow the customers and developers to plugin
 their

>>> own custom defined log4j 

Re: Programmatic log4j appender in Apex

[Vlad Rozov]

It will require application recompilation and repackaging. The proposed 
functionality is for dev-ops to be able to route application logging to 
a preferred destination without recompiling applications. It is run-time 
configuration vs compile time hardcoded appender.


Thank you,

Vlad
On 4/10/17 11:23, Munagala Ramanath wrote:

You can do it in a trivial derived class without changing the base class.

Ram

On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov 
wrote:


Does not the proposal to use Logger.addAppender() requires modifications
to used operators code?

Thank you,

Vlad

On 4/10/17 10:58, Munagala Ramanath wrote:


People can currently do this by simply implementing the Appender interface
and adding it
with Logger.addAppender() in the setup method. Why do we need something
more elaborate ?

Ram

On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko 
wrote:

The configuration of a log4j appender via log4j configuration file is a

static configuration that cannot be disabled/enabled and managed
dynamically by an application designer. The programmatic approach will
allow  an application designer to specify which of the available log4j
appenders should be used for the specific application.

It is not necessary Apex should use the predefined log4j appenders only.
The log4j events contain useful but the very limited number of properties
which values can be printed into output log4j sources. But based on the
knowledge of the software product workflow, the custom defined log4j
appender can extend a list of predefined output log events properties
and,
for instance for Apex, return: node, user name, application name,
application id, container id, operator name, etc.

Also the output log events that are generated by a custom defined log4j
appender can be stored and indexed by any type of a full text search
database. It will allow the customers and developers to simplify
collection
of log events statistics and searching/filtering of specific events for
debugging and investigation.

Thanks,
Sergey


On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov 
wrote:

+1 Apex engine does not own log4j config file - it is provided either by

Hadoop or an application. Hadoop log4j config does not necessarily meet
application logging requirements, but if log4j is provided by an
application designer, who can only specify what to log, it may not meet
operations requirements. Dev-ops should have an ability to specify where


to


log depending on the available infrastructure at run-time.

It will be good to have an ability not only specify extra log4j
appenders
at lunch time, but also at run-time, the same way how log4j logger
levels
may be changed.

Thank you,

Vlad

On 4/9/17 23:14, Priyanka Gugale wrote:

We can always write a custom appender and add it by changing root
appender
in log4j config file. Can you explain how adding appender grammatically

would help?

-Priyanka

On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare 

Re: Programmatic log4j appender in Apex

[Munagala Ramanath]

You can do it in a trivial derived class without changing the base class.

Ram

On Mon, Apr 10, 2017 at 11:19 AM, Vlad Rozov 
wrote:

> Does not the proposal to use Logger.addAppender() requires modifications
> to used operators code?
>
> Thank you,
>
> Vlad
>
> On 4/10/17 10:58, Munagala Ramanath wrote:
>
>> People can currently do this by simply implementing the Appender interface
>> and adding it
>> with Logger.addAppender() in the setup method. Why do we need something
>> more elaborate ?
>>
>> Ram
>>
>> On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko 
>> wrote:
>>
>> The configuration of a log4j appender via log4j configuration file is a
>>> static configuration that cannot be disabled/enabled and managed
>>> dynamically by an application designer. The programmatic approach will
>>> allow  an application designer to specify which of the available log4j
>>> appenders should be used for the specific application.
>>>
>>> It is not necessary Apex should use the predefined log4j appenders only.
>>> The log4j events contain useful but the very limited number of properties
>>> which values can be printed into output log4j sources. But based on the
>>> knowledge of the software product workflow, the custom defined log4j
>>> appender can extend a list of predefined output log events properties
>>> and,
>>> for instance for Apex, return: node, user name, application name,
>>> application id, container id, operator name, etc.
>>>
>>> Also the output log events that are generated by a custom defined log4j
>>> appender can be stored and indexed by any type of a full text search
>>> database. It will allow the customers and developers to simplify
>>> collection
>>> of log events statistics and searching/filtering of specific events for
>>> debugging and investigation.
>>>
>>> Thanks,
>>> Sergey
>>>
>>>
>>> On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov 
>>> wrote:
>>>
>>> +1 Apex engine does not own log4j config file - it is provided either by
 Hadoop or an application. Hadoop log4j config does not necessarily meet
 application logging requirements, but if log4j is provided by an
 application designer, who can only specify what to log, it may not meet
 operations requirements. Dev-ops should have an ability to specify where

>>> to
>>>
 log depending on the available infrastructure at run-time.

 It will be good to have an ability not only specify extra log4j
 appenders
 at lunch time, but also at run-time, the same way how log4j logger
 levels
 may be changed.

 Thank you,

 Vlad

 On 4/9/17 23:14, Priyanka Gugale wrote:

 We can always write a custom appender and add it by changing root
>
 appender
>>>
 in log4j config file. Can you explain how adding appender grammatically
> would help?
>
> -Priyanka
>
> On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare  >
> wrote:
>
> Please give some examples and/or use cases of this programmatic log4j
>
>> appender.
>>
>> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko <
>> ser...@datatorrent.com
>> wrote:
>>
>> Hi All,
>>
>>> I'd like to add supporting of a custom defined log4j appender that
>>> can
>>> be
>>> added to Apex Application Master and Containers and be configurable
>>> programmatically.
>>>
>>> Sometimes it is not trivial to control log4j configuration via log4j
>>> properties. And I think the having of the approach to add a log4j
>>>
>>> appender
>>
>> programmatically will allow the customers and developers to plugin
>>>
>> their
>>>
 own custom defined log4j appenders and be much flexible for streaming
>>> and
>>> collection of Apex log events.
>>>
>>> I assume to provide generic approach for definition of the
>>>
>> programmatic
>>>
 log4j appender and to pass all configuration parameters including a
>>>
>> name
>>>
 of
>>
>> the Java class with implementation of the log4j appender via system
>>>
>>> and/or
>>
>> command line properties.
>>>
>>> Thanks,
>>> Sergey
>>>
>>>
>>>
>>
>>
>


-- 

___

Munagala V. Ramanath

Software Engineer

E: r...@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam

www.datatorrent.com  |  apex.apache.org

Re: Programmatic log4j appender in Apex

[Vlad Rozov]

Does not the proposal to use Logger.addAppender() requires modifications 
to used operators code?


Thank you,

Vlad

On 4/10/17 10:58, Munagala Ramanath wrote:

People can currently do this by simply implementing the Appender interface
and adding it
with Logger.addAppender() in the setup method. Why do we need something
more elaborate ?

Ram

On Mon, Apr 10, 2017 at 10:30 AM, Sergey Golovko 
wrote:


The configuration of a log4j appender via log4j configuration file is a
static configuration that cannot be disabled/enabled and managed
dynamically by an application designer. The programmatic approach will
allow  an application designer to specify which of the available log4j
appenders should be used for the specific application.

It is not necessary Apex should use the predefined log4j appenders only.
The log4j events contain useful but the very limited number of properties
which values can be printed into output log4j sources. But based on the
knowledge of the software product workflow, the custom defined log4j
appender can extend a list of predefined output log events properties and,
for instance for Apex, return: node, user name, application name,
application id, container id, operator name, etc.

Also the output log events that are generated by a custom defined log4j
appender can be stored and indexed by any type of a full text search
database. It will allow the customers and developers to simplify collection
of log events statistics and searching/filtering of specific events for
debugging and investigation.

Thanks,
Sergey


On Mon, Apr 10, 2017 at 6:34 AM, Vlad Rozov 
wrote:


+1 Apex engine does not own log4j config file - it is provided either by
Hadoop or an application. Hadoop log4j config does not necessarily meet
application logging requirements, but if log4j is provided by an
application designer, who can only specify what to log, it may not meet
operations requirements. Dev-ops should have an ability to specify where

to

log depending on the available infrastructure at run-time.

It will be good to have an ability not only specify extra log4j appenders
at lunch time, but also at run-time, the same way how log4j logger levels
may be changed.

Thank you,

Vlad

On 4/9/17 23:14, Priyanka Gugale wrote:


We can always write a custom appender and add it by changing root

appender

in log4j config file. Can you explain how adding appender grammatically
would help?

-Priyanka

On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare 
wrote:

Please give some examples and/or use cases of this programmatic log4j

appender.

On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko 

Re: Programmatic log4j appender in Apex

[Vlad Rozov]

+1 Apex engine does not own log4j config file - it is provided either by 
Hadoop or an application. Hadoop log4j config does not necessarily meet 
application logging requirements, but if log4j is provided by an 
application designer, who can only specify what to log, it may not meet 
operations requirements. Dev-ops should have an ability to specify where 
to log depending on the available infrastructure at run-time.


It will be good to have an ability not only specify extra log4j 
appenders  at lunch time, but also at run-time, the same way how log4j 
logger levels may be changed.


Thank you,

Vlad
On 4/9/17 23:14, Priyanka Gugale wrote:

We can always write a custom appender and add it by changing root appender
in log4j config file. Can you explain how adding appender grammatically
would help?

-Priyanka

On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare 
wrote:


Please give some examples and/or use cases of this programmatic log4j
appender.

On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko 
wrote:


Hi All,

I'd like to add supporting of a custom defined log4j appender that can be
added to Apex Application Master and Containers and be configurable
programmatically.

Sometimes it is not trivial to control log4j configuration via log4j
properties. And I think the having of the approach to add a log4j

appender

programmatically will allow the customers and developers to plugin their
own custom defined log4j appenders and be much flexible for streaming and
collection of Apex log events.

I assume to provide generic approach for definition of the programmatic
log4j appender and to pass all configuration parameters including a name

of

the Java class with implementation of the log4j appender via system

and/or

command line properties.

Thanks,
Sergey

Re: Programmatic log4j appender in Apex

[Priyanka Gugale]

We can always write a custom appender and add it by changing root appender
in log4j config file. Can you explain how adding appender grammatically
would help?

-Priyanka

On Sun, Apr 9, 2017 at 11:50 AM, Sanjay Pujare 
wrote:

> Please give some examples and/or use cases of this programmatic log4j
> appender.
>
> On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko 
> wrote:
>
> > Hi All,
> >
> > I'd like to add supporting of a custom defined log4j appender that can be
> > added to Apex Application Master and Containers and be configurable
> > programmatically.
> >
> > Sometimes it is not trivial to control log4j configuration via log4j
> > properties. And I think the having of the approach to add a log4j
> appender
> > programmatically will allow the customers and developers to plugin their
> > own custom defined log4j appenders and be much flexible for streaming and
> > collection of Apex log events.
> >
> > I assume to provide generic approach for definition of the programmatic
> > log4j appender and to pass all configuration parameters including a name
> of
> > the Java class with implementation of the log4j appender via system
> and/or
> > command line properties.
> >
> > Thanks,
> > Sergey
> >
>

Re: Programmatic log4j appender in Apex

[Sanjay Pujare]

Please give some examples and/or use cases of this programmatic log4j
appender.

On Fri, Apr 7, 2017 at 8:40 PM, Sergey Golovko 
wrote:

> Hi All,
>
> I'd like to add supporting of a custom defined log4j appender that can be
> added to Apex Application Master and Containers and be configurable
> programmatically.
>
> Sometimes it is not trivial to control log4j configuration via log4j
> properties. And I think the having of the approach to add a log4j appender
> programmatically will allow the customers and developers to plugin their
> own custom defined log4j appenders and be much flexible for streaming and
> collection of Apex log events.
>
> I assume to provide generic approach for definition of the programmatic
> log4j appender and to pass all configuration parameters including a name of
> the Java class with implementation of the log4j appender via system and/or
> command line properties.
>
> Thanks,
> Sergey
>

Programmatic log4j appender in Apex

[Sergey Golovko]

Hi All,

I'd like to add supporting of a custom defined log4j appender that can be
added to Apex Application Master and Containers and be configurable
programmatically.

Sometimes it is not trivial to control log4j configuration via log4j
properties. And I think the having of the approach to add a log4j appender
programmatically will allow the customers and developers to plugin their
own custom defined log4j appenders and be much flexible for streaming and
collection of Apex log events.

I assume to provide generic approach for definition of the programmatic
log4j appender and to pass all configuration parameters including a name of
the Java class with implementation of the log4j appender via system and/or
command line properties.

Thanks,
Sergey