Re: [DISCUSS] Flume 1.9 release proposal

[Ferenc Szabo]

Thank you for the notification, it will be included. The RC is still in
progress as some minor issue came up during the release process.

On Tue, Dec 4, 2018 at 3:41 PM Ralph Goers 
wrote:

> I am noticing that the release notes on the 1.9 branch have not been
> updated to say changes to the logging configuration will be required in the
> next release as it will be upgraded to Log4j 2.
>
> Ralph
>
> > On Nov 30, 2018, at 8:33 PM, Ralph Goers 
> wrote:
> >
> > Ok, I’m fine with that.
> >
> > Ralph
> >
> >> On Nov 30, 2018, at 3:31 PM, Mike Percy  wrote:
> >>
> >> Hi Ralph,
> >> Let’s update the release notes to indicate that it is coming and do it
> in the next release so people are forewarned of the change. After speaking
> with others it sounds like Flume users are willing to accept the change
> (some have already had to do it because of Solr) so I have changed my view
> regarding whether we should wait for Flume 2.0 to do it. I’m now on board
> doing the log4j2 switch in a 1.10 release if that is what we want to
> release next time.
> >>
> >> Mike
> >>
> >>> On Nov 30, 2018, at 2:20 PM, Ralph Goers 
> wrote:
> >>>
> >>> Mike,
> >>>
> >>> This response confuses me. Are you saying that because the customers
> are good with the config change Flume 1.9 can go out with Log4j 2 or are
> you saying you are good with updating the release notes to say it will
> happen in the next release?
> >>>
> >>> Ralph
> >>>
>  On Nov 29, 2018, at 10:53 PM, Mike Percy  wrote:
> 
>  After speaking with a couple of large users of Flume, because log4j2
> is ABI compatible with log4j1 It sounds like the config change will be
> acceptable to them at upgrade time and so I am +1 on this proposal.
> 
>  Regards,
>  Mike
> 
> > On Nov 28, 2018, at 5:37 AM, Ralph Goers 
> wrote:
> >
> > Again, please update the release notes to indicate the logging
> configuration change will be coming with the next release.
> >
> > Ralph
> >
> >> On Nov 28, 2018, at 1:17 AM, Denes Arvay 
> wrote:
> >>
> >> Hi All,
> >>
> >> I agree that the current content doesn't justify the 2.0 release,
> so I'd
> >> vote for moving forward with the 1.9 as Ferenc recommended, i.e.
> without
> >> the 2 previously mentioned breaking changes.
> >> I also agree with the proposed new features, especially with the
> plugin
> >> system/modularization, which was discussed on the other thread:
> >> https://s.apache.org/2nm9
> >>
> >> So, Ferenc, I think you are good to go with the proposed plan,
> unless
> >> anybody vetoes.
> >>
> >> Regards,
> >> Denes
> >>
> >> On Tue, Nov 27, 2018 at 4:37 PM Ralph Goers <
> ralph.go...@dslextreme.com>
> >> wrote:
> >>
> >>> While Log4j 2 supports using properties files for configuration it
> is not
> >>> syntactically compatible with Log4j 1. However, migrating a Log4j 1
> >>> configuration to Log4j 2 isn’t a difficult task.
> >>>
> >>> Ralph
> >>>
>  On Nov 27, 2018, at 4:38 AM, Tristan Stevens
> >>>  wrote:
> 
>  I did start thinking a while back about a REST API that could be
> used for
>  configuring Flume, after which you could potentially bolt on a
> UI. I
>  stopped work on it because it got tricky around configuring
> sources (and
>  the way in which they relate to channels etc). If someone had
> more time
> >>> I’d
>  be happy to push to a repo the work that I did. This could be a
> really
>  useful 2.0 feature.
> 
>  Regarding move from Log4j1.x to 2, it’d be interesting to see
> what other
>  Apache projects, such as Hadoop, Hive etc did around this (in
> fact, are
>  they even at 2.x yet?). For me, if you need to change config,
> it’s not a
>  minor release, it become major, unless there’s a way you can
> >>> automatically
>  migrate config from one to the other. I’ve not checked, but are
> we sure
>  that .properties file won’t work with log4j2?
> 
>  Tristan
> 
> 
> 
>  On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org)
> wrote:
> 
>  Yeah, I agree with this, especially classloader isolation would
> be great
> >>> to
>  have as well on the plugin side.
> 
>  Mike
> 
>  On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers <
> ralph.go...@dslextreme.com
> 
>  wrote:
> 
> > I think people want to do more than just upgrade Log4j for a
> Flume 2.0
> > release. I would like to make configuration much more pluggable
> for
> > example. There was also talk about splitting all the sources,
> sinks,
> > interceptors, etc out of the core module and making them some
> sort of
> > plugin. At this point those are just at the idea stage.
> >
> >> On Nov 26, 2018, at 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

I am noticing that the release notes on the 1.9 branch have not been updated to 
say changes to the logging configuration will be required in the next release 
as it will be upgraded to Log4j 2.

Ralph

> On Nov 30, 2018, at 8:33 PM, Ralph Goers  wrote:
> 
> Ok, I’m fine with that.
> 
> Ralph
> 
>> On Nov 30, 2018, at 3:31 PM, Mike Percy  wrote:
>> 
>> Hi Ralph,
>> Let’s update the release notes to indicate that it is coming and do it in 
>> the next release so people are forewarned of the change. After speaking with 
>> others it sounds like Flume users are willing to accept the change (some 
>> have already had to do it because of Solr) so I have changed my view 
>> regarding whether we should wait for Flume 2.0 to do it. I’m now on board 
>> doing the log4j2 switch in a 1.10 release if that is what we want to release 
>> next time.
>> 
>> Mike
>> 
>>> On Nov 30, 2018, at 2:20 PM, Ralph Goers  wrote:
>>> 
>>> Mike, 
>>> 
>>> This response confuses me. Are you saying that because the customers are 
>>> good with the config change Flume 1.9 can go out with Log4j 2 or are you 
>>> saying you are good with updating the release notes to say it will happen 
>>> in the next release?
>>> 
>>> Ralph
>>> 
 On Nov 29, 2018, at 10:53 PM, Mike Percy  wrote:
 
 After speaking with a couple of large users of Flume, because log4j2 is 
 ABI compatible with log4j1 It sounds like the config change will be 
 acceptable to them at upgrade time and so I am +1 on this proposal.
 
 Regards,
 Mike
 
> On Nov 28, 2018, at 5:37 AM, Ralph Goers  
> wrote:
> 
> Again, please update the release notes to indicate the logging 
> configuration change will be coming with the next release.
> 
> Ralph
> 
>> On Nov 28, 2018, at 1:17 AM, Denes Arvay  
>> wrote:
>> 
>> Hi All,
>> 
>> I agree that the current content doesn't justify the 2.0 release, so I'd
>> vote for moving forward with the 1.9 as Ferenc recommended, i.e. without
>> the 2 previously mentioned breaking changes.
>> I also agree with the proposed new features, especially with the plugin
>> system/modularization, which was discussed on the other thread:
>> https://s.apache.org/2nm9
>> 
>> So, Ferenc, I think you are good to go with the proposed plan, unless
>> anybody vetoes.
>> 
>> Regards,
>> Denes
>> 
>> On Tue, Nov 27, 2018 at 4:37 PM Ralph Goers 
>> wrote:
>> 
>>> While Log4j 2 supports using properties files for configuration it is 
>>> not
>>> syntactically compatible with Log4j 1. However, migrating a Log4j 1
>>> configuration to Log4j 2 isn’t a difficult task.
>>> 
>>> Ralph
>>> 
 On Nov 27, 2018, at 4:38 AM, Tristan Stevens
>>>  wrote:
 
 I did start thinking a while back about a REST API that could be used 
 for
 configuring Flume, after which you could potentially bolt on a UI. I
 stopped work on it because it got tricky around configuring sources 
 (and
 the way in which they relate to channels etc). If someone had more time
>>> I’d
 be happy to push to a repo the work that I did. This could be a really
 useful 2.0 feature.
 
 Regarding move from Log4j1.x to 2, it’d be interesting to see what 
 other
 Apache projects, such as Hadoop, Hive etc did around this (in fact, are
 they even at 2.x yet?). For me, if you need to change config, it’s not 
 a
 minor release, it become major, unless there’s a way you can
>>> automatically
 migrate config from one to the other. I’ve not checked, but are we sure
 that .properties file won’t work with log4j2?
 
 Tristan
 
 
 
 On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:
 
 Yeah, I agree with this, especially classloader isolation would be 
 great
>>> to
 have as well on the plugin side.
 
 Mike
 
 On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers 
 >>> 
 wrote:
 
> I think people want to do more than just upgrade Log4j for a Flume 2.0
> release. I would like to make configuration much more pluggable for
> example. There was also talk about splitting all the sources, sinks,
> interceptors, etc out of the core module and making them some sort of
> plugin. At this point those are just at the idea stage.
> 
>> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát <
>>> bes...@apache.org>
 
> wrote:
>> 
>> I wonder, is there anything blocking us from releasing 2.0 next 
>> instead
> of 1.x?
>> 
>> 
>> Donat
>> 
>>> On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  
>>> wrote:
>>> 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

Ok, I’m fine with that.

Ralph

> On Nov 30, 2018, at 3:31 PM, Mike Percy  wrote:
> 
> Hi Ralph,
> Let’s update the release notes to indicate that it is coming and do it in the 
> next release so people are forewarned of the change. After speaking with 
> others it sounds like Flume users are willing to accept the change (some have 
> already had to do it because of Solr) so I have changed my view regarding 
> whether we should wait for Flume 2.0 to do it. I’m now on board doing the 
> log4j2 switch in a 1.10 release if that is what we want to release next time.
> 
> Mike
> 
>> On Nov 30, 2018, at 2:20 PM, Ralph Goers  wrote:
>> 
>> Mike, 
>> 
>> This response confuses me. Are you saying that because the customers are 
>> good with the config change Flume 1.9 can go out with Log4j 2 or are you 
>> saying you are good with updating the release notes to say it will happen in 
>> the next release?
>> 
>> Ralph
>> 
>>> On Nov 29, 2018, at 10:53 PM, Mike Percy  wrote:
>>> 
>>> After speaking with a couple of large users of Flume, because log4j2 is ABI 
>>> compatible with log4j1 It sounds like the config change will be acceptable 
>>> to them at upgrade time and so I am +1 on this proposal.
>>> 
>>> Regards,
>>> Mike
>>> 
 On Nov 28, 2018, at 5:37 AM, Ralph Goers  
 wrote:
 
 Again, please update the release notes to indicate the logging 
 configuration change will be coming with the next release.
 
 Ralph
 
> On Nov 28, 2018, at 1:17 AM, Denes Arvay  
> wrote:
> 
> Hi All,
> 
> I agree that the current content doesn't justify the 2.0 release, so I'd
> vote for moving forward with the 1.9 as Ferenc recommended, i.e. without
> the 2 previously mentioned breaking changes.
> I also agree with the proposed new features, especially with the plugin
> system/modularization, which was discussed on the other thread:
> https://s.apache.org/2nm9
> 
> So, Ferenc, I think you are good to go with the proposed plan, unless
> anybody vetoes.
> 
> Regards,
> Denes
> 
> On Tue, Nov 27, 2018 at 4:37 PM Ralph Goers 
> wrote:
> 
>> While Log4j 2 supports using properties files for configuration it is not
>> syntactically compatible with Log4j 1. However, migrating a Log4j 1
>> configuration to Log4j 2 isn’t a difficult task.
>> 
>> Ralph
>> 
>>> On Nov 27, 2018, at 4:38 AM, Tristan Stevens
>>  wrote:
>>> 
>>> I did start thinking a while back about a REST API that could be used 
>>> for
>>> configuring Flume, after which you could potentially bolt on a UI. I
>>> stopped work on it because it got tricky around configuring sources (and
>>> the way in which they relate to channels etc). If someone had more time
>> I’d
>>> be happy to push to a repo the work that I did. This could be a really
>>> useful 2.0 feature.
>>> 
>>> Regarding move from Log4j1.x to 2, it’d be interesting to see what other
>>> Apache projects, such as Hadoop, Hive etc did around this (in fact, are
>>> they even at 2.x yet?). For me, if you need to change config, it’s not a
>>> minor release, it become major, unless there’s a way you can
>> automatically
>>> migrate config from one to the other. I’ve not checked, but are we sure
>>> that .properties file won’t work with log4j2?
>>> 
>>> Tristan
>>> 
>>> 
>>> 
>>> On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:
>>> 
>>> Yeah, I agree with this, especially classloader isolation would be great
>> to
>>> have as well on the plugin side.
>>> 
>>> Mike
>>> 
>>> On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers >> 
>>> wrote:
>>> 
 I think people want to do more than just upgrade Log4j for a Flume 2.0
 release. I would like to make configuration much more pluggable for
 example. There was also talk about splitting all the sources, sinks,
 interceptors, etc out of the core module and making them some sort of
 plugin. At this point those are just at the idea stage.
 
> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát <
>> bes...@apache.org>
>>> 
 wrote:
> 
> I wonder, is there anything blocking us from releasing 2.0 next 
> instead
 of 1.x?
> 
> 
> Donat
> 
>> On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
>> 
>> If only the PMC would be affected by this decision, we could have had
 this
>> discussion on the PMC private list. But this decision impacts
>>> everybody
>> that uses Flume. So let's hear from anybody who cares about this,
 including
>> committers, contributors, and users on whether they are okay with
 switching
>> to log4j2 in a minor release version, knowing that they will need to
 

Re: [DISCUSS] Flume 1.9 release proposal

[Mike Percy]

Hi Ralph,
Let’s update the release notes to indicate that it is coming and do it in the 
next release so people are forewarned of the change. After speaking with others 
it sounds like Flume users are willing to accept the change (some have already 
had to do it because of Solr) so I have changed my view regarding whether we 
should wait for Flume 2.0 to do it. I’m now on board doing the log4j2 switch in 
a 1.10 release if that is what we want to release next time.

Mike

> On Nov 30, 2018, at 2:20 PM, Ralph Goers  wrote:
> 
> Mike, 
> 
> This response confuses me. Are you saying that because the customers are good 
> with the config change Flume 1.9 can go out with Log4j 2 or are you saying 
> you are good with updating the release notes to say it will happen in the 
> next release?
> 
> Ralph
> 
>> On Nov 29, 2018, at 10:53 PM, Mike Percy  wrote:
>> 
>> After speaking with a couple of large users of Flume, because log4j2 is ABI 
>> compatible with log4j1 It sounds like the config change will be acceptable 
>> to them at upgrade time and so I am +1 on this proposal.
>> 
>> Regards,
>> Mike
>> 
>>> On Nov 28, 2018, at 5:37 AM, Ralph Goers  wrote:
>>> 
>>> Again, please update the release notes to indicate the logging 
>>> configuration change will be coming with the next release.
>>> 
>>> Ralph
>>> 
 On Nov 28, 2018, at 1:17 AM, Denes Arvay  
 wrote:
 
 Hi All,
 
 I agree that the current content doesn't justify the 2.0 release, so I'd
 vote for moving forward with the 1.9 as Ferenc recommended, i.e. without
 the 2 previously mentioned breaking changes.
 I also agree with the proposed new features, especially with the plugin
 system/modularization, which was discussed on the other thread:
 https://s.apache.org/2nm9
 
 So, Ferenc, I think you are good to go with the proposed plan, unless
 anybody vetoes.
 
 Regards,
 Denes
 
 On Tue, Nov 27, 2018 at 4:37 PM Ralph Goers 
 wrote:
 
> While Log4j 2 supports using properties files for configuration it is not
> syntactically compatible with Log4j 1. However, migrating a Log4j 1
> configuration to Log4j 2 isn’t a difficult task.
> 
> Ralph
> 
>> On Nov 27, 2018, at 4:38 AM, Tristan Stevens
>  wrote:
>> 
>> I did start thinking a while back about a REST API that could be used for
>> configuring Flume, after which you could potentially bolt on a UI. I
>> stopped work on it because it got tricky around configuring sources (and
>> the way in which they relate to channels etc). If someone had more time
> I’d
>> be happy to push to a repo the work that I did. This could be a really
>> useful 2.0 feature.
>> 
>> Regarding move from Log4j1.x to 2, it’d be interesting to see what other
>> Apache projects, such as Hadoop, Hive etc did around this (in fact, are
>> they even at 2.x yet?). For me, if you need to change config, it’s not a
>> minor release, it become major, unless there’s a way you can
> automatically
>> migrate config from one to the other. I’ve not checked, but are we sure
>> that .properties file won’t work with log4j2?
>> 
>> Tristan
>> 
>> 
>> 
>> On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:
>> 
>> Yeah, I agree with this, especially classloader isolation would be great
> to
>> have as well on the plugin side.
>> 
>> Mike
>> 
>> On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers > 
>> wrote:
>> 
>>> I think people want to do more than just upgrade Log4j for a Flume 2.0
>>> release. I would like to make configuration much more pluggable for
>>> example. There was also talk about splitting all the sources, sinks,
>>> interceptors, etc out of the core module and making them some sort of
>>> plugin. At this point those are just at the idea stage.
>>> 
 On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát <
> bes...@apache.org>
>> 
>>> wrote:
 
 I wonder, is there anything blocking us from releasing 2.0 next instead
>>> of 1.x?
 
 
 Donat
 
> On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
> 
> If only the PMC would be affected by this decision, we could have had
>>> this
> discussion on the PMC private list. But this decision impacts
>> everybody
> that uses Flume. So let's hear from anybody who cares about this,
>>> including
> committers, contributors, and users on whether they are okay with
>>> switching
> to log4j2 in a minor release version, knowing that they will need to
>>> change
> their config files when they upgrade Flume.
> 
> Ferenc, it seems like we will have to ship one or the other with the
>>> binary
> artifacts at release time. It seems to me that we still have to make a

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

Mike, 

This response confuses me. Are you saying that because the customers are good 
with the config change Flume 1.9 can go out with Log4j 2 or are you saying you 
are good with updating the release notes to say it will happen in the next 
release?

Ralph

> On Nov 29, 2018, at 10:53 PM, Mike Percy  wrote:
> 
> After speaking with a couple of large users of Flume, because log4j2 is ABI 
> compatible with log4j1 It sounds like the config change will be acceptable to 
> them at upgrade time and so I am +1 on this proposal.
> 
> Regards,
> Mike
> 
>> On Nov 28, 2018, at 5:37 AM, Ralph Goers  wrote:
>> 
>> Again, please update the release notes to indicate the logging configuration 
>> change will be coming with the next release.
>> 
>> Ralph
>> 
>>> On Nov 28, 2018, at 1:17 AM, Denes Arvay  wrote:
>>> 
>>> Hi All,
>>> 
>>> I agree that the current content doesn't justify the 2.0 release, so I'd
>>> vote for moving forward with the 1.9 as Ferenc recommended, i.e. without
>>> the 2 previously mentioned breaking changes.
>>> I also agree with the proposed new features, especially with the plugin
>>> system/modularization, which was discussed on the other thread:
>>> https://s.apache.org/2nm9
>>> 
>>> So, Ferenc, I think you are good to go with the proposed plan, unless
>>> anybody vetoes.
>>> 
>>> Regards,
>>> Denes
>>> 
>>> On Tue, Nov 27, 2018 at 4:37 PM Ralph Goers 
>>> wrote:
>>> 
 While Log4j 2 supports using properties files for configuration it is not
 syntactically compatible with Log4j 1. However, migrating a Log4j 1
 configuration to Log4j 2 isn’t a difficult task.
 
 Ralph
 
> On Nov 27, 2018, at 4:38 AM, Tristan Stevens
  wrote:
> 
> I did start thinking a while back about a REST API that could be used for
> configuring Flume, after which you could potentially bolt on a UI. I
> stopped work on it because it got tricky around configuring sources (and
> the way in which they relate to channels etc). If someone had more time
 I’d
> be happy to push to a repo the work that I did. This could be a really
> useful 2.0 feature.
> 
> Regarding move from Log4j1.x to 2, it’d be interesting to see what other
> Apache projects, such as Hadoop, Hive etc did around this (in fact, are
> they even at 2.x yet?). For me, if you need to change config, it’s not a
> minor release, it become major, unless there’s a way you can
 automatically
> migrate config from one to the other. I’ve not checked, but are we sure
> that .properties file won’t work with log4j2?
> 
> Tristan
> 
> 
> 
> On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:
> 
> Yeah, I agree with this, especially classloader isolation would be great
 to
> have as well on the plugin side.
> 
> Mike
> 
> On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers  
> wrote:
> 
>> I think people want to do more than just upgrade Log4j for a Flume 2.0
>> release. I would like to make configuration much more pluggable for
>> example. There was also talk about splitting all the sources, sinks,
>> interceptors, etc out of the core module and making them some sort of
>> plugin. At this point those are just at the idea stage.
>> 
>>> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát <
 bes...@apache.org>
> 
>> wrote:
>>> 
>>> I wonder, is there anything blocking us from releasing 2.0 next instead
>> of 1.x?
>>> 
>>> 
>>> Donat
>>> 
 On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
 
 If only the PMC would be affected by this decision, we could have had
>> this
 discussion on the PMC private list. But this decision impacts
> everybody
 that uses Flume. So let's hear from anybody who cares about this,
>> including
 committers, contributors, and users on whether they are okay with
>> switching
 to log4j2 in a minor release version, knowing that they will need to
>> change
 their config files when they upgrade Flume.
 
 Ferenc, it seems like we will have to ship one or the other with the
>> binary
 artifacts at release time. It seems to me that we still have to make a
 choice about the built and shipped default, even if both could work at
 runtime, right?
 
 Thanks,
 Mike
 
 On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo
>> 
 wrote:
 
> I have directed it to the community, the PMC members.
> I am looking for a decision from the PMC members (including You) on
>> whether
> we should continue as planned or not removing log4j2 form the minor
>> release
> branch.
> 
> On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <
>> ralph.go...@dslextreme.com>
> wrote:
> 
>> Was this 

Re: [DISCUSS] Flume 1.9 release proposal

[Mike Percy]

After speaking with a couple of large users of Flume, because log4j2 is ABI 
compatible with log4j1 It sounds like the config change will be acceptable to 
them at upgrade time and so I am +1 on this proposal.

Regards,
Mike

> On Nov 28, 2018, at 5:37 AM, Ralph Goers  wrote:
> 
> Again, please update the release notes to indicate the logging configuration 
> change will be coming with the next release.
> 
> Ralph
> 
>> On Nov 28, 2018, at 1:17 AM, Denes Arvay  wrote:
>> 
>> Hi All,
>> 
>> I agree that the current content doesn't justify the 2.0 release, so I'd
>> vote for moving forward with the 1.9 as Ferenc recommended, i.e. without
>> the 2 previously mentioned breaking changes.
>> I also agree with the proposed new features, especially with the plugin
>> system/modularization, which was discussed on the other thread:
>> https://s.apache.org/2nm9
>> 
>> So, Ferenc, I think you are good to go with the proposed plan, unless
>> anybody vetoes.
>> 
>> Regards,
>> Denes
>> 
>> On Tue, Nov 27, 2018 at 4:37 PM Ralph Goers 
>> wrote:
>> 
>>> While Log4j 2 supports using properties files for configuration it is not
>>> syntactically compatible with Log4j 1. However, migrating a Log4j 1
>>> configuration to Log4j 2 isn’t a difficult task.
>>> 
>>> Ralph
>>> 
 On Nov 27, 2018, at 4:38 AM, Tristan Stevens
>>>  wrote:
 
 I did start thinking a while back about a REST API that could be used for
 configuring Flume, after which you could potentially bolt on a UI. I
 stopped work on it because it got tricky around configuring sources (and
 the way in which they relate to channels etc). If someone had more time
>>> I’d
 be happy to push to a repo the work that I did. This could be a really
 useful 2.0 feature.
 
 Regarding move from Log4j1.x to 2, it’d be interesting to see what other
 Apache projects, such as Hadoop, Hive etc did around this (in fact, are
 they even at 2.x yet?). For me, if you need to change config, it’s not a
 minor release, it become major, unless there’s a way you can
>>> automatically
 migrate config from one to the other. I’ve not checked, but are we sure
 that .properties file won’t work with log4j2?
 
 Tristan
 
 
 
 On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:
 
 Yeah, I agree with this, especially classloader isolation would be great
>>> to
 have as well on the plugin side.
 
 Mike
 
 On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers >>> 
 wrote:
 
> I think people want to do more than just upgrade Log4j for a Flume 2.0
> release. I would like to make configuration much more pluggable for
> example. There was also talk about splitting all the sources, sinks,
> interceptors, etc out of the core module and making them some sort of
> plugin. At this point those are just at the idea stage.
> 
>> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát <
>>> bes...@apache.org>
 
> wrote:
>> 
>> I wonder, is there anything blocking us from releasing 2.0 next instead
> of 1.x?
>> 
>> 
>> Donat
>> 
>>> On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
>>> 
>>> If only the PMC would be affected by this decision, we could have had
> this
>>> discussion on the PMC private list. But this decision impacts
 everybody
>>> that uses Flume. So let's hear from anybody who cares about this,
> including
>>> committers, contributors, and users on whether they are okay with
> switching
>>> to log4j2 in a minor release version, knowing that they will need to
> change
>>> their config files when they upgrade Flume.
>>> 
>>> Ferenc, it seems like we will have to ship one or the other with the
> binary
>>> artifacts at release time. It seems to me that we still have to make a
>>> choice about the built and shipped default, even if both could work at
>>> runtime, right?
>>> 
>>> Thanks,
>>> Mike
>>> 
>>> On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo
> 
>>> wrote:
>>> 
 I have directed it to the community, the PMC members.
 I am looking for a decision from the PMC members (including You) on
> whether
 we should continue as planned or not removing log4j2 form the minor
> release
 branch.
 
 On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <
> ralph.go...@dslextreme.com>
 wrote:
 
> Was this directed at me? I am not sure what decision you are
 referring
 to.
> I stated my position in the email you replied to.
> 
> Ralph
> 
>> On Nov 26, 2018, at 5:29 AM, Ferenc Szabo
>  
> wrote:
>> 
>> With the new release, you can just replace the jars on the
 classpath
> as
> we
>> removed every code dependency and using slf4j. So you do not 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

Again, please update the release notes to indicate the logging configuration 
change will be coming with the next release.

Ralph

> On Nov 28, 2018, at 1:17 AM, Denes Arvay  wrote:
> 
> Hi All,
> 
> I agree that the current content doesn't justify the 2.0 release, so I'd
> vote for moving forward with the 1.9 as Ferenc recommended, i.e. without
> the 2 previously mentioned breaking changes.
> I also agree with the proposed new features, especially with the plugin
> system/modularization, which was discussed on the other thread:
> https://s.apache.org/2nm9
> 
> So, Ferenc, I think you are good to go with the proposed plan, unless
> anybody vetoes.
> 
> Regards,
> Denes
> 
> On Tue, Nov 27, 2018 at 4:37 PM Ralph Goers 
> wrote:
> 
>> While Log4j 2 supports using properties files for configuration it is not
>> syntactically compatible with Log4j 1. However, migrating a Log4j 1
>> configuration to Log4j 2 isn’t a difficult task.
>> 
>> Ralph
>> 
>>> On Nov 27, 2018, at 4:38 AM, Tristan Stevens
>>  wrote:
>>> 
>>> I did start thinking a while back about a REST API that could be used for
>>> configuring Flume, after which you could potentially bolt on a UI. I
>>> stopped work on it because it got tricky around configuring sources (and
>>> the way in which they relate to channels etc). If someone had more time
>> I’d
>>> be happy to push to a repo the work that I did. This could be a really
>>> useful 2.0 feature.
>>> 
>>> Regarding move from Log4j1.x to 2, it’d be interesting to see what other
>>> Apache projects, such as Hadoop, Hive etc did around this (in fact, are
>>> they even at 2.x yet?). For me, if you need to change config, it’s not a
>>> minor release, it become major, unless there’s a way you can
>> automatically
>>> migrate config from one to the other. I’ve not checked, but are we sure
>>> that .properties file won’t work with log4j2?
>>> 
>>> Tristan
>>> 
>>> 
>>> 
>>> On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:
>>> 
>>> Yeah, I agree with this, especially classloader isolation would be great
>> to
>>> have as well on the plugin side.
>>> 
>>> Mike
>>> 
>>> On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers >> 
>>> wrote:
>>> 
 I think people want to do more than just upgrade Log4j for a Flume 2.0
 release. I would like to make configuration much more pluggable for
 example. There was also talk about splitting all the sources, sinks,
 interceptors, etc out of the core module and making them some sort of
 plugin. At this point those are just at the idea stage.
 
> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát <
>> bes...@apache.org>
>>> 
 wrote:
> 
> I wonder, is there anything blocking us from releasing 2.0 next instead
 of 1.x?
> 
> 
> Donat
> 
> On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
>> 
>> If only the PMC would be affected by this decision, we could have had
 this
>> discussion on the PMC private list. But this decision impacts
>>> everybody
>> that uses Flume. So let's hear from anybody who cares about this,
 including
>> committers, contributors, and users on whether they are okay with
 switching
>> to log4j2 in a minor release version, knowing that they will need to
 change
>> their config files when they upgrade Flume.
>> 
>> Ferenc, it seems like we will have to ship one or the other with the
 binary
>> artifacts at release time. It seems to me that we still have to make a
>> choice about the built and shipped default, even if both could work at
>> runtime, right?
>> 
>> Thanks,
>> Mike
>> 
>> On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo
 
>> wrote:
>> 
>>> I have directed it to the community, the PMC members.
>>> I am looking for a decision from the PMC members (including You) on
 whether
>>> we should continue as planned or not removing log4j2 form the minor
 release
>>> branch.
>>> 
>>> On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <
 ralph.go...@dslextreme.com>
>>> wrote:
>>> 
 Was this directed at me? I am not sure what decision you are
>>> referring
>>> to.
 I stated my position in the email you replied to.
 
 Ralph
 
> On Nov 26, 2018, at 5:29 AM, Ferenc Szabo
 >>> 
 wrote:
> 
> With the new release, you can just replace the jars on the
>>> classpath
 as
 we
> removed every code dependency and using slf4j. So you do not need
>>> to
>>> fork
> and change anything.
> 
> Let me know your decision and we will continue with that.
> 
> Regards,
> Ferenc
> 
> On Mon, Nov 26, 2018 at 12:10 PM Apache <
>> ralph.go...@dslextreme.com>
>>> 
 wrote:
> 
>> As I said, I am fine with adding a notice to the release notes
 stating
 the
>> logging 

Re: [DISCUSS] Flume 1.9 release proposal

[Denes Arvay]

Hi All,

I agree that the current content doesn't justify the 2.0 release, so I'd
vote for moving forward with the 1.9 as Ferenc recommended, i.e. without
the 2 previously mentioned breaking changes.
I also agree with the proposed new features, especially with the plugin
system/modularization, which was discussed on the other thread:
https://s.apache.org/2nm9

So, Ferenc, I think you are good to go with the proposed plan, unless
anybody vetoes.

Regards,
Denes

On Tue, Nov 27, 2018 at 4:37 PM Ralph Goers 
wrote:

> While Log4j 2 supports using properties files for configuration it is not
> syntactically compatible with Log4j 1. However, migrating a Log4j 1
> configuration to Log4j 2 isn’t a difficult task.
>
> Ralph
>
> > On Nov 27, 2018, at 4:38 AM, Tristan Stevens
>  wrote:
> >
> > I did start thinking a while back about a REST API that could be used for
> > configuring Flume, after which you could potentially bolt on a UI. I
> > stopped work on it because it got tricky around configuring sources (and
> > the way in which they relate to channels etc). If someone had more time
> I’d
> > be happy to push to a repo the work that I did. This could be a really
> > useful 2.0 feature.
> >
> > Regarding move from Log4j1.x to 2, it’d be interesting to see what other
> > Apache projects, such as Hadoop, Hive etc did around this (in fact, are
> > they even at 2.x yet?). For me, if you need to change config, it’s not a
> > minor release, it become major, unless there’s a way you can
> automatically
> > migrate config from one to the other. I’ve not checked, but are we sure
> > that .properties file won’t work with log4j2?
> >
> > Tristan
> >
> >
> >
> > On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:
> >
> > Yeah, I agree with this, especially classloader isolation would be great
> to
> > have as well on the plugin side.
> >
> > Mike
> >
> > On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers  >
> > wrote:
> >
> >> I think people want to do more than just upgrade Log4j for a Flume 2.0
> >> release. I would like to make configuration much more pluggable for
> >> example. There was also talk about splitting all the sources, sinks,
> >> interceptors, etc out of the core module and making them some sort of
> >> plugin. At this point those are just at the idea stage.
> >>
> >>> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát <
> bes...@apache.org>
> >
> >> wrote:
> >>>
> >>> I wonder, is there anything blocking us from releasing 2.0 next instead
> >> of 1.x?
> >>>
> >>>
> >>> Donat
> >>>
> >>> On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
> 
>  If only the PMC would be affected by this decision, we could have had
> >> this
>  discussion on the PMC private list. But this decision impacts
> > everybody
>  that uses Flume. So let's hear from anybody who cares about this,
> >> including
>  committers, contributors, and users on whether they are okay with
> >> switching
>  to log4j2 in a minor release version, knowing that they will need to
> >> change
>  their config files when they upgrade Flume.
> 
>  Ferenc, it seems like we will have to ship one or the other with the
> >> binary
>  artifacts at release time. It seems to me that we still have to make a
>  choice about the built and shipped default, even if both could work at
>  runtime, right?
> 
>  Thanks,
>  Mike
> 
>  On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo
> >> 
>  wrote:
> 
> > I have directed it to the community, the PMC members.
> > I am looking for a decision from the PMC members (including You) on
> >> whether
> > we should continue as planned or not removing log4j2 form the minor
> >> release
> > branch.
> >
> > On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <
> >> ralph.go...@dslextreme.com>
> > wrote:
> >
> >> Was this directed at me? I am not sure what decision you are
> > referring
> > to.
> >> I stated my position in the email you replied to.
> >>
> >> Ralph
> >>
> >>> On Nov 26, 2018, at 5:29 AM, Ferenc Szabo
> >>  >>
> >> wrote:
> >>>
> >>> With the new release, you can just replace the jars on the
> > classpath
> >> as
> >> we
> >>> removed every code dependency and using slf4j. So you do not need
> > to
> > fork
> >>> and change anything.
> >>>
> >>> Let me know your decision and we will continue with that.
> >>>
> >>> Regards,
> >>> Ferenc
> >>>
> >>> On Mon, Nov 26, 2018 at 12:10 PM Apache <
> ralph.go...@dslextreme.com>
> >
> >> wrote:
> >>>
>  As I said, I am fine with adding a notice to the release notes
> >> stating
> >> the
>  logging will be changed in the next release regardless of its
> >> version
>  number. This way the revert only needs to happen on the release
> >> branch
> >> for
>  this release.
> 
>  Ralph
> 
> > On Nov 25, 2018, at 9:59 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

While Log4j 2 supports using properties files for configuration it is not 
syntactically compatible with Log4j 1. However, migrating a Log4j 1 
configuration to Log4j 2 isn’t a difficult task.

Ralph

> On Nov 27, 2018, at 4:38 AM, Tristan Stevens  
> wrote:
> 
> I did start thinking a while back about a REST API that could be used for
> configuring Flume, after which you could potentially bolt on a UI. I
> stopped work on it because it got tricky around configuring sources (and
> the way in which they relate to channels etc). If someone had more time I’d
> be happy to push to a repo the work that I did. This could be a really
> useful 2.0 feature.
> 
> Regarding move from Log4j1.x to 2, it’d be interesting to see what other
> Apache projects, such as Hadoop, Hive etc did around this (in fact, are
> they even at 2.x yet?). For me, if you need to change config, it’s not a
> minor release, it become major, unless there’s a way you can automatically
> migrate config from one to the other. I’ve not checked, but are we sure
> that .properties file won’t work with log4j2?
> 
> Tristan
> 
> 
> 
> On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:
> 
> Yeah, I agree with this, especially classloader isolation would be great to
> have as well on the plugin side.
> 
> Mike
> 
> On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers 
> wrote:
> 
>> I think people want to do more than just upgrade Log4j for a Flume 2.0
>> release. I would like to make configuration much more pluggable for
>> example. There was also talk about splitting all the sources, sinks,
>> interceptors, etc out of the core module and making them some sort of
>> plugin. At this point those are just at the idea stage.
>> 
>>> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát 
> 
>> wrote:
>>> 
>>> I wonder, is there anything blocking us from releasing 2.0 next instead
>> of 1.x?
>>> 
>>> 
>>> Donat
>>> 
>>> On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
 
 If only the PMC would be affected by this decision, we could have had
>> this
 discussion on the PMC private list. But this decision impacts
> everybody
 that uses Flume. So let's hear from anybody who cares about this,
>> including
 committers, contributors, and users on whether they are okay with
>> switching
 to log4j2 in a minor release version, knowing that they will need to
>> change
 their config files when they upgrade Flume.
 
 Ferenc, it seems like we will have to ship one or the other with the
>> binary
 artifacts at release time. It seems to me that we still have to make a
 choice about the built and shipped default, even if both could work at
 runtime, right?
 
 Thanks,
 Mike
 
 On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo
>> 
 wrote:
 
> I have directed it to the community, the PMC members.
> I am looking for a decision from the PMC members (including You) on
>> whether
> we should continue as planned or not removing log4j2 form the minor
>> release
> branch.
> 
> On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <
>> ralph.go...@dslextreme.com>
> wrote:
> 
>> Was this directed at me? I am not sure what decision you are
> referring
> to.
>> I stated my position in the email you replied to.
>> 
>> Ralph
>> 
>>> On Nov 26, 2018, at 5:29 AM, Ferenc Szabo
>> > 
>> wrote:
>>> 
>>> With the new release, you can just replace the jars on the
> classpath
>> as
>> we
>>> removed every code dependency and using slf4j. So you do not need
> to
> fork
>>> and change anything.
>>> 
>>> Let me know your decision and we will continue with that.
>>> 
>>> Regards,
>>> Ferenc
>>> 
>>> On Mon, Nov 26, 2018 at 12:10 PM Apache 
> 
>> wrote:
>>> 
 As I said, I am fine with adding a notice to the release notes
>> stating
>> the
 logging will be changed in the next release regardless of its
>> version
 number. This way the revert only needs to happen on the release
>> branch
>> for
 this release.
 
 Ralph
 
> On Nov 25, 2018, at 9:59 PM, Mike Percy 
> wrote:
> 
> While the committer veto rules are well documented here
> , and there
> is
> no
> mention of a time limit, I propose we shelve that discussion and
>> work
>> on
> getting to consensus on what we as a community want to include in
>> the
 Flume
> 1.9 release.
> 
> As far as I can tell, the decision we have to make is whether to
>> include
> the FLUME-2050 logging changes in the Flume 1.9 release. We can
> either:
> 
> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to
>> modify
> their log4j configuration files to use the different log4j2 XML
> format,
> with the 

Re: [DISCUSS] Flume 1.9 release proposal

[Tristan Stevens]

I did start thinking a while back about a REST API that could be used for
configuring Flume, after which you could potentially bolt on a UI. I
stopped work on it because it got tricky around configuring sources (and
the way in which they relate to channels etc). If someone had more time I’d
be happy to push to a repo the work that I did. This could be a really
useful 2.0 feature.

Regarding move from Log4j1.x to 2, it’d be interesting to see what other
Apache projects, such as Hadoop, Hive etc did around this (in fact, are
they even at 2.x yet?). For me, if you need to change config, it’s not a
minor release, it become major, unless there’s a way you can automatically
migrate config from one to the other. I’ve not checked, but are we sure
that .properties file won’t work with log4j2?

Tristan



On 26 November 2018 at 19:54:01, Mike Percy (mpe...@apache.org) wrote:

Yeah, I agree with this, especially classloader isolation would be great to
have as well on the plugin side.

Mike

On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers 
wrote:

> I think people want to do more than just upgrade Log4j for a Flume 2.0
> release. I would like to make configuration much more pluggable for
> example. There was also talk about splitting all the sources, sinks,
> interceptors, etc out of the core module and making them some sort of
> plugin. At this point those are just at the idea stage.
>
> > On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát 

> wrote:
> >
> > I wonder, is there anything blocking us from releasing 2.0 next instead
> of 1.x?
> >
> >
> > Donat
> >
> > On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
> >>
> >> If only the PMC would be affected by this decision, we could have had
> this
> >> discussion on the PMC private list. But this decision impacts
everybody
> >> that uses Flume. So let's hear from anybody who cares about this,
> including
> >> committers, contributors, and users on whether they are okay with
> switching
> >> to log4j2 in a minor release version, knowing that they will need to
> change
> >> their config files when they upgrade Flume.
> >>
> >> Ferenc, it seems like we will have to ship one or the other with the
> binary
> >> artifacts at release time. It seems to me that we still have to make a
> >> choice about the built and shipped default, even if both could work at
> >> runtime, right?
> >>
> >> Thanks,
> >> Mike
> >>
> >> On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo
> 
> >> wrote:
> >>
> >>> I have directed it to the community, the PMC members.
> >>> I am looking for a decision from the PMC members (including You) on
> whether
> >>> we should continue as planned or not removing log4j2 form the minor
> release
> >>> branch.
> >>>
> >>> On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <
> ralph.go...@dslextreme.com>
> >>> wrote:
> >>>
>  Was this directed at me? I am not sure what decision you are
referring
> >>> to.
>  I stated my position in the email you replied to.
> 
>  Ralph
> 
> > On Nov 26, 2018, at 5:29 AM, Ferenc Szabo
>  
>  wrote:
> >
> > With the new release, you can just replace the jars on the
classpath
> as
>  we
> > removed every code dependency and using slf4j. So you do not need
to
> >>> fork
> > and change anything.
> >
> > Let me know your decision and we will continue with that.
> >
> > Regards,
> > Ferenc
> >
> > On Mon, Nov 26, 2018 at 12:10 PM Apache 

>  wrote:
> >
> >> As I said, I am fine with adding a notice to the release notes
> stating
>  the
> >> logging will be changed in the next release regardless of its
> version
> >> number. This way the revert only needs to happen on the release
> branch
>  for
> >> this release.
> >>
> >> Ralph
> >>
> >>> On Nov 25, 2018, at 9:59 PM, Mike Percy 
wrote:
> >>>
> >>> While the committer veto rules are well documented here
> >>> , and there
is
> >>> no
> >>> mention of a time limit, I propose we shelve that discussion and
> work
>  on
> >>> getting to consensus on what we as a community want to include in
> the
> >> Flume
> >>> 1.9 release.
> >>>
> >>> As far as I can tell, the decision we have to make is whether to
>  include
> >>> the FLUME-2050 logging changes in the Flume 1.9 release. We can
> >>> either:
> >>>
> >>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to
>  modify
> >>> their log4j configuration files to use the different log4j2 XML
> >>> format,
> >>> with the procedure for doing so documented in the release notes.
> >>> -or-
> >>> 2) Defer that change to a future release where incompatible
changes
> >>> are
> >>> expected, such as Flume 2.0.
> >>>
> >>> Also, maybe there are other options I haven't thought of...?
> >>>
> >>> I would like to get some input from more people on this matter.
How
> >>> do
> >>> 

Re: [DISCUSS] Flume 1.9 release proposal

[Daniel Grigg]

Since users were invited for comment I’ll simply say that for us having to 
change a bit of logging config is the least of worries, maybe an hour to deploy 
to ECS.  Honestly making the code base more accessible to casual contributors, 
a plugin system and more frequent releases of connectors is more important to 
us. We’d almost given up on flume and were about to bite the bullet and migrate 
to Kafka connect until this 1.9 announcement popped up.

Daniel Grigg

Sent from my iPhone

> On 27 Nov 2018, at 06:53, Mike Percy  wrote:
> 
> Yeah, I agree with this, especially classloader isolation would be great to
> have as well on the plugin side.
> 
> Mike
> 
> On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers 
> wrote:
> 
>> I think people want to do more than just upgrade Log4j for a Flume 2.0
>> release. I would like to make configuration much more pluggable for
>> example. There was also talk about splitting all the sources, sinks,
>> interceptors, etc out of the core module and making them some sort of
>> plugin. At this point those are just at the idea stage.
>> 
>>> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát 
>> wrote:
>>> 
>>> I wonder, is there anything blocking us from releasing 2.0 next instead
>> of 1.x?
>>> 
>>> 
>>> Donat
>>> 
 On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
 
 If only the PMC would be affected by this decision, we could have had
>> this
 discussion on the PMC private list. But this decision impacts everybody
 that uses Flume. So let's hear from anybody who cares about this,
>> including
 committers, contributors, and users on whether they are okay with
>> switching
 to log4j2 in a minor release version, knowing that they will need to
>> change
 their config files when they upgrade Flume.
 
 Ferenc, it seems like we will have to ship one or the other with the
>> binary
 artifacts at release time. It seems to me that we still have to make a
 choice about the built and shipped default, even if both could work at
 runtime, right?
 
 Thanks,
 Mike
 
 On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo
>> 
 wrote:
 
> I have directed it to the community, the PMC members.
> I am looking for a decision from the PMC members (including You) on
>> whether
> we should continue as planned or not removing log4j2 form the minor
>> release
> branch.
> 
> On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <
>> ralph.go...@dslextreme.com>
> wrote:
> 
>> Was this directed at me? I am not sure what decision you are referring
> to.
>> I stated my position in the email you replied to.
>> 
>> Ralph
>> 
>>> On Nov 26, 2018, at 5:29 AM, Ferenc Szabo
>> > 
>> wrote:
>>> 
>>> With the new release, you can just replace the jars on the classpath
>> as
>> we
>>> removed every code dependency and using slf4j. So you do not need to
> fork
>>> and change anything.
>>> 
>>> Let me know your decision and we will continue with that.
>>> 
>>> Regards,
>>> Ferenc
>>> 
>>> On Mon, Nov 26, 2018 at 12:10 PM Apache 
>> wrote:
>>> 
 As I said, I am fine with adding a notice to the release notes
>> stating
>> the
 logging will be changed in the next release regardless of its
>> version
 number. This way the revert only needs to happen on the release
>> branch
>> for
 this release.
 
 Ralph
 
> On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
> 
> While the committer veto rules are well documented here
> , and there is
> no
> mention of a time limit, I propose we shelve that discussion and
>> work
>> on
> getting to consensus on what we as a community want to include in
>> the
 Flume
> 1.9 release.
> 
> As far as I can tell, the decision we have to make is whether to
>> include
> the FLUME-2050 logging changes in the Flume 1.9 release. We can
> either:
> 
> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to
>> modify
> their log4j configuration files to use the different log4j2 XML
> format,
> with the procedure for doing so documented in the release notes.
> -or-
> 2) Defer that change to a future release where incompatible changes
> are
> expected, such as Flume 2.0.
> 
> Also, maybe there are other options I haven't thought of...?
> 
> I would like to get some input from more people on this matter. How
> do
> others feel about this?
> 
> Thanks,
> Mike
> 
> 
> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers <
>> ralph.go...@dslextreme.com>
> wrote:
> 
>> I should also point out that the time to raise this was 

Re: [DISCUSS] Flume 1.9 release proposal

[Mike Percy]

Yeah, I agree with this, especially classloader isolation would be great to
have as well on the plugin side.

Mike

On Mon, Nov 26, 2018 at 11:50 AM Ralph Goers 
wrote:

> I think people want to do more than just upgrade Log4j for a Flume 2.0
> release. I would like to make configuration much more pluggable for
> example. There was also talk about splitting all the sources, sinks,
> interceptors, etc out of the core module and making them some sort of
> plugin. At this point those are just at the idea stage.
>
> > On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát 
> wrote:
> >
> > I wonder, is there anything blocking us from releasing 2.0 next instead
> of 1.x?
> >
> >
> > Donat
> >
> > On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
> >>
> >> If only the PMC would be affected by this decision, we could have had
> this
> >> discussion on the PMC private list. But this decision impacts everybody
> >> that uses Flume. So let's hear from anybody who cares about this,
> including
> >> committers, contributors, and users on whether they are okay with
> switching
> >> to log4j2 in a minor release version, knowing that they will need to
> change
> >> their config files when they upgrade Flume.
> >>
> >> Ferenc, it seems like we will have to ship one or the other with the
> binary
> >> artifacts at release time. It seems to me that we still have to make a
> >> choice about the built and shipped default, even if both could work at
> >> runtime, right?
> >>
> >> Thanks,
> >> Mike
> >>
> >> On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo
> 
> >> wrote:
> >>
> >>> I have directed it to the community, the PMC members.
> >>> I am looking for a decision from the PMC members (including You) on
> whether
> >>> we should continue as planned or not removing log4j2 form the minor
> release
> >>> branch.
> >>>
> >>> On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers <
> ralph.go...@dslextreme.com>
> >>> wrote:
> >>>
>  Was this directed at me? I am not sure what decision you are referring
> >>> to.
>  I stated my position in the email you replied to.
> 
>  Ralph
> 
> > On Nov 26, 2018, at 5:29 AM, Ferenc Szabo
>  
>  wrote:
> >
> > With the new release, you can just replace the jars on the classpath
> as
>  we
> > removed every code dependency and using slf4j. So you do not need to
> >>> fork
> > and change anything.
> >
> > Let me know your decision and we will continue with that.
> >
> > Regards,
> > Ferenc
> >
> > On Mon, Nov 26, 2018 at 12:10 PM Apache 
>  wrote:
> >
> >> As I said, I am fine with adding a notice to the release notes
> stating
>  the
> >> logging will be changed in the next release regardless of its
> version
> >> number. This way the revert only needs to happen on the release
> branch
>  for
> >> this release.
> >>
> >> Ralph
> >>
> >>> On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
> >>>
> >>> While the committer veto rules are well documented here
> >>> , and there is
> >>> no
> >>> mention of a time limit, I propose we shelve that discussion and
> work
>  on
> >>> getting to consensus on what we as a community want to include in
> the
> >> Flume
> >>> 1.9 release.
> >>>
> >>> As far as I can tell, the decision we have to make is whether to
>  include
> >>> the FLUME-2050 logging changes in the Flume 1.9 release. We can
> >>> either:
> >>>
> >>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to
>  modify
> >>> their log4j configuration files to use the different log4j2 XML
> >>> format,
> >>> with the procedure for doing so documented in the release notes.
> >>> -or-
> >>> 2) Defer that change to a future release where incompatible changes
> >>> are
> >>> expected, such as Flume 2.0.
> >>>
> >>> Also, maybe there are other options I haven't thought of...?
> >>>
> >>> I would like to get some input from more people on this matter. How
> >>> do
> >>> others feel about this?
> >>>
> >>> Thanks,
> >>> Mike
> >>>
> >>>
> >>> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers <
>  ralph.go...@dslextreme.com>
> >>> wrote:
> >>>
>  I should also point out that the time to raise this was a year ago
>  when
>  the PR for FLUME-2050 was reviewed and committed. As it stands
> now I
> >> could
>  be a jerk and vote -1 on the patch for FLUME-3296 with valid
> >>> technical
>  grounds. If this was causing a true binary incompatibility I would
> >> approve
>  reverting it in a heartbeat, but I just don’t see how having users
>  have
> >> to
>  change logging configuration is “intolerable”, especially with the
>  known
>  security issues in Log4j 1, however unlikely user’s might be to
> >> encounter
>  them.
> 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

I think people want to do more than just upgrade Log4j for a Flume 2.0 release. 
I would like to make configuration much more pluggable for example. There was 
also talk about splitting all the sources, sinks, interceptors, etc out of the 
core module and making them some sort of plugin. At this point those are just 
at the idea stage.

> On Nov 26, 2018, at 12:19 PM, Bessenyei Balázs Donát  
> wrote:
> 
> I wonder, is there anything blocking us from releasing 2.0 next instead of 
> 1.x?
> 
> 
> Donat
> 
> On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
>> 
>> If only the PMC would be affected by this decision, we could have had this
>> discussion on the PMC private list. But this decision impacts everybody
>> that uses Flume. So let's hear from anybody who cares about this, including
>> committers, contributors, and users on whether they are okay with switching
>> to log4j2 in a minor release version, knowing that they will need to change
>> their config files when they upgrade Flume.
>> 
>> Ferenc, it seems like we will have to ship one or the other with the binary
>> artifacts at release time. It seems to me that we still have to make a
>> choice about the built and shipped default, even if both could work at
>> runtime, right?
>> 
>> Thanks,
>> Mike
>> 
>> On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo 
>> wrote:
>> 
>>> I have directed it to the community, the PMC members.
>>> I am looking for a decision from the PMC members (including You) on whether
>>> we should continue as planned or not removing log4j2 form the minor release
>>> branch.
>>> 
>>> On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers 
>>> wrote:
>>> 
 Was this directed at me? I am not sure what decision you are referring
>>> to.
 I stated my position in the email you replied to.
 
 Ralph
 
> On Nov 26, 2018, at 5:29 AM, Ferenc Szabo >>> 
 wrote:
> 
> With the new release, you can just replace the jars on the classpath as
 we
> removed every code dependency and using slf4j. So you do not need to
>>> fork
> and change anything.
> 
> Let me know your decision and we will continue with that.
> 
> Regards,
> Ferenc
> 
> On Mon, Nov 26, 2018 at 12:10 PM Apache 
 wrote:
> 
>> As I said, I am fine with adding a notice to the release notes stating
 the
>> logging will be changed in the next release regardless of its version
>> number. This way the revert only needs to happen on the release branch
 for
>> this release.
>> 
>> Ralph
>> 
>>> On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
>>> 
>>> While the committer veto rules are well documented here
>>> , and there is
>>> no
>>> mention of a time limit, I propose we shelve that discussion and work
 on
>>> getting to consensus on what we as a community want to include in the
>> Flume
>>> 1.9 release.
>>> 
>>> As far as I can tell, the decision we have to make is whether to
 include
>>> the FLUME-2050 logging changes in the Flume 1.9 release. We can
>>> either:
>>> 
>>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to
 modify
>>> their log4j configuration files to use the different log4j2 XML
>>> format,
>>> with the procedure for doing so documented in the release notes.
>>> -or-
>>> 2) Defer that change to a future release where incompatible changes
>>> are
>>> expected, such as Flume 2.0.
>>> 
>>> Also, maybe there are other options I haven't thought of...?
>>> 
>>> I would like to get some input from more people on this matter. How
>>> do
>>> others feel about this?
>>> 
>>> Thanks,
>>> Mike
>>> 
>>> 
>>> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers <
 ralph.go...@dslextreme.com>
>>> wrote:
>>> 
 I should also point out that the time to raise this was a year ago
 when
 the PR for FLUME-2050 was reviewed and committed. As it stands now I
>> could
 be a jerk and vote -1 on the patch for FLUME-3296 with valid
>>> technical
 grounds. If this was causing a true binary incompatibility I would
>> approve
 reverting it in a heartbeat, but I just don’t see how having users
 have
>> to
 change logging configuration is “intolerable”, especially with the
 known
 security issues in Log4j 1, however unlikely user’s might be to
>> encounter
 them.
 
 That said, I wouldn’t veto the revert on the release branch, but I
 would
 suggest that the release notes provide fair warning that the next
>> release
 will upgrade the logging dependency.  It would also be nice if
 releases
 could be more frequent than once a year.
 
 I would also like to say that I’m not doing this for the fun of it.
>>> My
 company uses Flume for some of its 

Re: [DISCUSS] Flume 1.9 release proposal

[Bessenyei Balázs Donát]

I wonder, is there anything blocking us from releasing 2.0 next instead of 1.x?


Donat

On Mon, Nov 26, 2018 at 6:27 PM Mike Percy  wrote:
>
> If only the PMC would be affected by this decision, we could have had this
> discussion on the PMC private list. But this decision impacts everybody
> that uses Flume. So let's hear from anybody who cares about this, including
> committers, contributors, and users on whether they are okay with switching
> to log4j2 in a minor release version, knowing that they will need to change
> their config files when they upgrade Flume.
>
> Ferenc, it seems like we will have to ship one or the other with the binary
> artifacts at release time. It seems to me that we still have to make a
> choice about the built and shipped default, even if both could work at
> runtime, right?
>
> Thanks,
> Mike
>
> On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo 
> wrote:
>
> > I have directed it to the community, the PMC members.
> > I am looking for a decision from the PMC members (including You) on whether
> > we should continue as planned or not removing log4j2 form the minor release
> > branch.
> >
> > On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers 
> > wrote:
> >
> > > Was this directed at me? I am not sure what decision you are referring
> > to.
> > > I stated my position in the email you replied to.
> > >
> > > Ralph
> > >
> > > > On Nov 26, 2018, at 5:29 AM, Ferenc Szabo  > >
> > > wrote:
> > > >
> > > > With the new release, you can just replace the jars on the classpath as
> > > we
> > > > removed every code dependency and using slf4j. So you do not need to
> > fork
> > > > and change anything.
> > > >
> > > > Let me know your decision and we will continue with that.
> > > >
> > > > Regards,
> > > > Ferenc
> > > >
> > > > On Mon, Nov 26, 2018 at 12:10 PM Apache 
> > > wrote:
> > > >
> > > >> As I said, I am fine with adding a notice to the release notes stating
> > > the
> > > >> logging will be changed in the next release regardless of its version
> > > >> number. This way the revert only needs to happen on the release branch
> > > for
> > > >> this release.
> > > >>
> > > >> Ralph
> > > >>
> > > >>> On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
> > > >>>
> > > >>> While the committer veto rules are well documented here
> > > >>> , and there is
> > no
> > > >>> mention of a time limit, I propose we shelve that discussion and work
> > > on
> > > >>> getting to consensus on what we as a community want to include in the
> > > >> Flume
> > > >>> 1.9 release.
> > > >>>
> > > >>> As far as I can tell, the decision we have to make is whether to
> > > include
> > > >>> the FLUME-2050 logging changes in the Flume 1.9 release. We can
> > either:
> > > >>>
> > > >>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to
> > > modify
> > > >>> their log4j configuration files to use the different log4j2 XML
> > format,
> > > >>> with the procedure for doing so documented in the release notes.
> > > >>> -or-
> > > >>> 2) Defer that change to a future release where incompatible changes
> > are
> > > >>> expected, such as Flume 2.0.
> > > >>>
> > > >>> Also, maybe there are other options I haven't thought of...?
> > > >>>
> > > >>> I would like to get some input from more people on this matter. How
> > do
> > > >>> others feel about this?
> > > >>>
> > > >>> Thanks,
> > > >>> Mike
> > > >>>
> > > >>>
> > > >>> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers <
> > > ralph.go...@dslextreme.com>
> > > >>> wrote:
> > > >>>
> > >  I should also point out that the time to raise this was a year ago
> > > when
> > >  the PR for FLUME-2050 was reviewed and committed. As it stands now I
> > > >> could
> > >  be a jerk and vote -1 on the patch for FLUME-3296 with valid
> > technical
> > >  grounds. If this was causing a true binary incompatibility I would
> > > >> approve
> > >  reverting it in a heartbeat, but I just don’t see how having users
> > > have
> > > >> to
> > >  change logging configuration is “intolerable”, especially with the
> > > known
> > >  security issues in Log4j 1, however unlikely user’s might be to
> > > >> encounter
> > >  them.
> > > 
> > >  That said, I wouldn’t veto the revert on the release branch, but I
> > > would
> > >  suggest that the release notes provide fair warning that the next
> > > >> release
> > >  will upgrade the logging dependency.  It would also be nice if
> > > releases
> > >  could be more frequent than once a year.
> > > 
> > >  I would also like to say that I’m not doing this for the fun of it.
> > My
> > >  company uses Flume for some of its most critical processing. We run
> > >  Veracode scans on all of our software and I expect Flume would be
> > > >> flagged
> > >  if I hadn’t repacked it with Log4j 2. It also may not show up since
> > > the
> > > >> CVE
> > >  is marked against Log4j 2 since Log4j 1 is EOL, but the security

Re: [DISCUSS] Flume 1.9 release proposal

[Mike Percy]

If only the PMC would be affected by this decision, we could have had this
discussion on the PMC private list. But this decision impacts everybody
that uses Flume. So let's hear from anybody who cares about this, including
committers, contributors, and users on whether they are okay with switching
to log4j2 in a minor release version, knowing that they will need to change
their config files when they upgrade Flume.

Ferenc, it seems like we will have to ship one or the other with the binary
artifacts at release time. It seems to me that we still have to make a
choice about the built and shipped default, even if both could work at
runtime, right?

Thanks,
Mike

On Mon, Nov 26, 2018 at 5:58 AM Ferenc Szabo 
wrote:

> I have directed it to the community, the PMC members.
> I am looking for a decision from the PMC members (including You) on whether
> we should continue as planned or not removing log4j2 form the minor release
> branch.
>
> On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers 
> wrote:
>
> > Was this directed at me? I am not sure what decision you are referring
> to.
> > I stated my position in the email you replied to.
> >
> > Ralph
> >
> > > On Nov 26, 2018, at 5:29 AM, Ferenc Szabo  >
> > wrote:
> > >
> > > With the new release, you can just replace the jars on the classpath as
> > we
> > > removed every code dependency and using slf4j. So you do not need to
> fork
> > > and change anything.
> > >
> > > Let me know your decision and we will continue with that.
> > >
> > > Regards,
> > > Ferenc
> > >
> > > On Mon, Nov 26, 2018 at 12:10 PM Apache 
> > wrote:
> > >
> > >> As I said, I am fine with adding a notice to the release notes stating
> > the
> > >> logging will be changed in the next release regardless of its version
> > >> number. This way the revert only needs to happen on the release branch
> > for
> > >> this release.
> > >>
> > >> Ralph
> > >>
> > >>> On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
> > >>>
> > >>> While the committer veto rules are well documented here
> > >>> , and there is
> no
> > >>> mention of a time limit, I propose we shelve that discussion and work
> > on
> > >>> getting to consensus on what we as a community want to include in the
> > >> Flume
> > >>> 1.9 release.
> > >>>
> > >>> As far as I can tell, the decision we have to make is whether to
> > include
> > >>> the FLUME-2050 logging changes in the Flume 1.9 release. We can
> either:
> > >>>
> > >>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to
> > modify
> > >>> their log4j configuration files to use the different log4j2 XML
> format,
> > >>> with the procedure for doing so documented in the release notes.
> > >>> -or-
> > >>> 2) Defer that change to a future release where incompatible changes
> are
> > >>> expected, such as Flume 2.0.
> > >>>
> > >>> Also, maybe there are other options I haven't thought of...?
> > >>>
> > >>> I would like to get some input from more people on this matter. How
> do
> > >>> others feel about this?
> > >>>
> > >>> Thanks,
> > >>> Mike
> > >>>
> > >>>
> > >>> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers <
> > ralph.go...@dslextreme.com>
> > >>> wrote:
> > >>>
> >  I should also point out that the time to raise this was a year ago
> > when
> >  the PR for FLUME-2050 was reviewed and committed. As it stands now I
> > >> could
> >  be a jerk and vote -1 on the patch for FLUME-3296 with valid
> technical
> >  grounds. If this was causing a true binary incompatibility I would
> > >> approve
> >  reverting it in a heartbeat, but I just don’t see how having users
> > have
> > >> to
> >  change logging configuration is “intolerable”, especially with the
> > known
> >  security issues in Log4j 1, however unlikely user’s might be to
> > >> encounter
> >  them.
> > 
> >  That said, I wouldn’t veto the revert on the release branch, but I
> > would
> >  suggest that the release notes provide fair warning that the next
> > >> release
> >  will upgrade the logging dependency.  It would also be nice if
> > releases
> >  could be more frequent than once a year.
> > 
> >  I would also like to say that I’m not doing this for the fun of it.
> My
> >  company uses Flume for some of its most critical processing. We run
> >  Veracode scans on all of our software and I expect Flume would be
> > >> flagged
> >  if I hadn’t repacked it with Log4j 2. It also may not show up since
> > the
> > >> CVE
> >  is marked against Log4j 2 since Log4j 1 is EOL, but the security
> > >> scanning
> >  tools should be flagging that as well.
> > 
> >  FWIW I’ve had to hack or enhance a few Flume components to make it
> > work
> >  for my needs but overall it works really, really well. I’d like to
> > >> commit
> >  back some of the changes but the main one - Flume configuration - I
> > >> really
> >  don’t like and need to redo the whole thing.
> > 
> 

Re: [DISCUSS] Flume 1.9 release proposal

[Ferenc Szabo]

I have directed it to the community, the PMC members.
I am looking for a decision from the PMC members (including You) on whether
we should continue as planned or not removing log4j2 form the minor release
branch.

On Mon, Nov 26, 2018 at 2:43 PM Ralph Goers 
wrote:

> Was this directed at me? I am not sure what decision you are referring to.
> I stated my position in the email you replied to.
>
> Ralph
>
> > On Nov 26, 2018, at 5:29 AM, Ferenc Szabo 
> wrote:
> >
> > With the new release, you can just replace the jars on the classpath as
> we
> > removed every code dependency and using slf4j. So you do not need to fork
> > and change anything.
> >
> > Let me know your decision and we will continue with that.
> >
> > Regards,
> > Ferenc
> >
> > On Mon, Nov 26, 2018 at 12:10 PM Apache 
> wrote:
> >
> >> As I said, I am fine with adding a notice to the release notes stating
> the
> >> logging will be changed in the next release regardless of its version
> >> number. This way the revert only needs to happen on the release branch
> for
> >> this release.
> >>
> >> Ralph
> >>
> >>> On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
> >>>
> >>> While the committer veto rules are well documented here
> >>> , and there is no
> >>> mention of a time limit, I propose we shelve that discussion and work
> on
> >>> getting to consensus on what we as a community want to include in the
> >> Flume
> >>> 1.9 release.
> >>>
> >>> As far as I can tell, the decision we have to make is whether to
> include
> >>> the FLUME-2050 logging changes in the Flume 1.9 release. We can either:
> >>>
> >>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to
> modify
> >>> their log4j configuration files to use the different log4j2 XML format,
> >>> with the procedure for doing so documented in the release notes.
> >>> -or-
> >>> 2) Defer that change to a future release where incompatible changes are
> >>> expected, such as Flume 2.0.
> >>>
> >>> Also, maybe there are other options I haven't thought of...?
> >>>
> >>> I would like to get some input from more people on this matter. How do
> >>> others feel about this?
> >>>
> >>> Thanks,
> >>> Mike
> >>>
> >>>
> >>> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers <
> ralph.go...@dslextreme.com>
> >>> wrote:
> >>>
>  I should also point out that the time to raise this was a year ago
> when
>  the PR for FLUME-2050 was reviewed and committed. As it stands now I
> >> could
>  be a jerk and vote -1 on the patch for FLUME-3296 with valid technical
>  grounds. If this was causing a true binary incompatibility I would
> >> approve
>  reverting it in a heartbeat, but I just don’t see how having users
> have
> >> to
>  change logging configuration is “intolerable”, especially with the
> known
>  security issues in Log4j 1, however unlikely user’s might be to
> >> encounter
>  them.
> 
>  That said, I wouldn’t veto the revert on the release branch, but I
> would
>  suggest that the release notes provide fair warning that the next
> >> release
>  will upgrade the logging dependency.  It would also be nice if
> releases
>  could be more frequent than once a year.
> 
>  I would also like to say that I’m not doing this for the fun of it. My
>  company uses Flume for some of its most critical processing. We run
>  Veracode scans on all of our software and I expect Flume would be
> >> flagged
>  if I hadn’t repacked it with Log4j 2. It also may not show up since
> the
> >> CVE
>  is marked against Log4j 2 since Log4j 1 is EOL, but the security
> >> scanning
>  tools should be flagging that as well.
> 
>  FWIW I’ve had to hack or enhance a few Flume components to make it
> work
>  for my needs but overall it works really, really well. I’d like to
> >> commit
>  back some of the changes but the main one - Flume configuration - I
> >> really
>  don’t like and need to redo the whole thing.
> 
>  Ralph
> 
> > On Nov 24, 2018, at 5:11 PM, Mike Percy  wrote:
> >
> > I wasn't aware of this security issue. Do you have a link to the
> >> details?
> >
> > There's no ASF requirement for package names; it's really just a Java
> > language convention, and especially not enforced for compatibility
> >> shims.
> > Even Flume has some of those in the legacy sources.
> >
> > I understand that it's very difficult to provide a compatibility
> layer.
> > Maybe also a boring task. I'm just saying that without log4j1
> backwards
> > compatibility provided by log4j2, there will have to be a really
> >> critical
> > reason to inflict this migration pain on Flume users -- something
> that
> > simply can't be tolerated, even in a minor release, like a new and
> >> highly
> > dangerous security bug. Without such a motivation, I don't see how
> this
> > incompatible dependency change can be justified.
> 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

Was this directed at me? I am not sure what decision you are referring to. I 
stated my position in the email you replied to.

Ralph

> On Nov 26, 2018, at 5:29 AM, Ferenc Szabo  wrote:
> 
> With the new release, you can just replace the jars on the classpath as we
> removed every code dependency and using slf4j. So you do not need to fork
> and change anything.
> 
> Let me know your decision and we will continue with that.
> 
> Regards,
> Ferenc
> 
> On Mon, Nov 26, 2018 at 12:10 PM Apache  wrote:
> 
>> As I said, I am fine with adding a notice to the release notes stating the
>> logging will be changed in the next release regardless of its version
>> number. This way the revert only needs to happen on the release branch for
>> this release.
>> 
>> Ralph
>> 
>>> On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
>>> 
>>> While the committer veto rules are well documented here
>>> , and there is no
>>> mention of a time limit, I propose we shelve that discussion and work on
>>> getting to consensus on what we as a community want to include in the
>> Flume
>>> 1.9 release.
>>> 
>>> As far as I can tell, the decision we have to make is whether to include
>>> the FLUME-2050 logging changes in the Flume 1.9 release. We can either:
>>> 
>>> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to modify
>>> their log4j configuration files to use the different log4j2 XML format,
>>> with the procedure for doing so documented in the release notes.
>>> -or-
>>> 2) Defer that change to a future release where incompatible changes are
>>> expected, such as Flume 2.0.
>>> 
>>> Also, maybe there are other options I haven't thought of...?
>>> 
>>> I would like to get some input from more people on this matter. How do
>>> others feel about this?
>>> 
>>> Thanks,
>>> Mike
>>> 
>>> 
>>> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers 
>>> wrote:
>>> 
 I should also point out that the time to raise this was a year ago when
 the PR for FLUME-2050 was reviewed and committed. As it stands now I
>> could
 be a jerk and vote -1 on the patch for FLUME-3296 with valid technical
 grounds. If this was causing a true binary incompatibility I would
>> approve
 reverting it in a heartbeat, but I just don’t see how having users have
>> to
 change logging configuration is “intolerable”, especially with the known
 security issues in Log4j 1, however unlikely user’s might be to
>> encounter
 them.
 
 That said, I wouldn’t veto the revert on the release branch, but I would
 suggest that the release notes provide fair warning that the next
>> release
 will upgrade the logging dependency.  It would also be nice if releases
 could be more frequent than once a year.
 
 I would also like to say that I’m not doing this for the fun of it. My
 company uses Flume for some of its most critical processing. We run
 Veracode scans on all of our software and I expect Flume would be
>> flagged
 if I hadn’t repacked it with Log4j 2. It also may not show up since the
>> CVE
 is marked against Log4j 2 since Log4j 1 is EOL, but the security
>> scanning
 tools should be flagging that as well.
 
 FWIW I’ve had to hack or enhance a few Flume components to make it work
 for my needs but overall it works really, really well. I’d like to
>> commit
 back some of the changes but the main one - Flume configuration - I
>> really
 don’t like and need to redo the whole thing.
 
 Ralph
 
> On Nov 24, 2018, at 5:11 PM, Mike Percy  wrote:
> 
> I wasn't aware of this security issue. Do you have a link to the
>> details?
> 
> There's no ASF requirement for package names; it's really just a Java
> language convention, and especially not enforced for compatibility
>> shims.
> Even Flume has some of those in the legacy sources.
> 
> I understand that it's very difficult to provide a compatibility layer.
> Maybe also a boring task. I'm just saying that without log4j1 backwards
> compatibility provided by log4j2, there will have to be a really
>> critical
> reason to inflict this migration pain on Flume users -- something that
> simply can't be tolerated, even in a minor release, like a new and
>> highly
> dangerous security bug. Without such a motivation, I don't see how this
> incompatible dependency change can be justified.
> 
> Mike
> 
> On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers <
>> ralph.go...@dslextreme.com>
> wrote:
> 
>> 
>>> On Nov 24, 2018, at 2:35 PM, Mike Percy  wrote:
>>> 
>>> Flume has long had a policy of backwards compatibility with its own
>>> configuration files and people expect things to "just work" when
>> upgrading
>>> Flume. If log4j2 can't parse the log4j1 config file format then it's
>> an
>>> incompatible upgrade and should not be done in a minor Flume release.
>>> 

Re: [DISCUSS] Flume 1.9 release proposal

[Ferenc Szabo]

With the new release, you can just replace the jars on the classpath as we
removed every code dependency and using slf4j. So you do not need to fork
and change anything.

Let me know your decision and we will continue with that.

Regards,
Ferenc

On Mon, Nov 26, 2018 at 12:10 PM Apache  wrote:

> As I said, I am fine with adding a notice to the release notes stating the
> logging will be changed in the next release regardless of its version
> number. This way the revert only needs to happen on the release branch for
> this release.
>
> Ralph
>
> > On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
> >
> > While the committer veto rules are well documented here
> > , and there is no
> > mention of a time limit, I propose we shelve that discussion and work on
> > getting to consensus on what we as a community want to include in the
> Flume
> > 1.9 release.
> >
> > As far as I can tell, the decision we have to make is whether to include
> > the FLUME-2050 logging changes in the Flume 1.9 release. We can either:
> >
> > 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to modify
> > their log4j configuration files to use the different log4j2 XML format,
> > with the procedure for doing so documented in the release notes.
> > -or-
> > 2) Defer that change to a future release where incompatible changes are
> > expected, such as Flume 2.0.
> >
> > Also, maybe there are other options I haven't thought of...?
> >
> > I would like to get some input from more people on this matter. How do
> > others feel about this?
> >
> > Thanks,
> > Mike
> >
> >
> > On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers 
> > wrote:
> >
> >> I should also point out that the time to raise this was a year ago when
> >> the PR for FLUME-2050 was reviewed and committed. As it stands now I
> could
> >> be a jerk and vote -1 on the patch for FLUME-3296 with valid technical
> >> grounds. If this was causing a true binary incompatibility I would
> approve
> >> reverting it in a heartbeat, but I just don’t see how having users have
> to
> >> change logging configuration is “intolerable”, especially with the known
> >> security issues in Log4j 1, however unlikely user’s might be to
> encounter
> >> them.
> >>
> >> That said, I wouldn’t veto the revert on the release branch, but I would
> >> suggest that the release notes provide fair warning that the next
> release
> >> will upgrade the logging dependency.  It would also be nice if releases
> >> could be more frequent than once a year.
> >>
> >> I would also like to say that I’m not doing this for the fun of it. My
> >> company uses Flume for some of its most critical processing. We run
> >> Veracode scans on all of our software and I expect Flume would be
> flagged
> >> if I hadn’t repacked it with Log4j 2. It also may not show up since the
> CVE
> >> is marked against Log4j 2 since Log4j 1 is EOL, but the security
> scanning
> >> tools should be flagging that as well.
> >>
> >> FWIW I’ve had to hack or enhance a few Flume components to make it work
> >> for my needs but overall it works really, really well. I’d like to
> commit
> >> back some of the changes but the main one - Flume configuration - I
> really
> >> don’t like and need to redo the whole thing.
> >>
> >> Ralph
> >>
> >>> On Nov 24, 2018, at 5:11 PM, Mike Percy  wrote:
> >>>
> >>> I wasn't aware of this security issue. Do you have a link to the
> details?
> >>>
> >>> There's no ASF requirement for package names; it's really just a Java
> >>> language convention, and especially not enforced for compatibility
> shims.
> >>> Even Flume has some of those in the legacy sources.
> >>>
> >>> I understand that it's very difficult to provide a compatibility layer.
> >>> Maybe also a boring task. I'm just saying that without log4j1 backwards
> >>> compatibility provided by log4j2, there will have to be a really
> critical
> >>> reason to inflict this migration pain on Flume users -- something that
> >>> simply can't be tolerated, even in a minor release, like a new and
> highly
> >>> dangerous security bug. Without such a motivation, I don't see how this
> >>> incompatible dependency change can be justified.
> >>>
> >>> Mike
> >>>
> >>> On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers <
> ralph.go...@dslextreme.com>
> >>> wrote:
> >>>
> 
> > On Nov 24, 2018, at 2:35 PM, Mike Percy  wrote:
> >
> > Flume has long had a policy of backwards compatibility with its own
> > configuration files and people expect things to "just work" when
>  upgrading
> > Flume. If log4j2 can't parse the log4j1 config file format then it's
> an
> > incompatible upgrade and should not be done in a minor Flume release.
> >
> > If log4j2 wants to be a drop-in replacement for log4j1 then by
> default
> >> it
> > should find and parse the traditional log4j.properties config files,
> at
> > least as a fallback, rather than force users to convert to the new
> XML
> > 

Re: [DISCUSS] Flume 1.9 release proposal

[Apache]

As I said, I am fine with adding a notice to the release notes stating the 
logging will be changed in the next release regardless of its version number. 
This way the revert only needs to happen on the release branch for this release.

Ralph

> On Nov 25, 2018, at 9:59 PM, Mike Percy  wrote:
> 
> While the committer veto rules are well documented here
> , and there is no
> mention of a time limit, I propose we shelve that discussion and work on
> getting to consensus on what we as a community want to include in the Flume
> 1.9 release.
> 
> As far as I can tell, the decision we have to make is whether to include
> the FLUME-2050 logging changes in the Flume 1.9 release. We can either:
> 
> 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to modify
> their log4j configuration files to use the different log4j2 XML format,
> with the procedure for doing so documented in the release notes.
> -or-
> 2) Defer that change to a future release where incompatible changes are
> expected, such as Flume 2.0.
> 
> Also, maybe there are other options I haven't thought of...?
> 
> I would like to get some input from more people on this matter. How do
> others feel about this?
> 
> Thanks,
> Mike
> 
> 
> On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers 
> wrote:
> 
>> I should also point out that the time to raise this was a year ago when
>> the PR for FLUME-2050 was reviewed and committed. As it stands now I could
>> be a jerk and vote -1 on the patch for FLUME-3296 with valid technical
>> grounds. If this was causing a true binary incompatibility I would approve
>> reverting it in a heartbeat, but I just don’t see how having users have to
>> change logging configuration is “intolerable”, especially with the known
>> security issues in Log4j 1, however unlikely user’s might be to encounter
>> them.
>> 
>> That said, I wouldn’t veto the revert on the release branch, but I would
>> suggest that the release notes provide fair warning that the next release
>> will upgrade the logging dependency.  It would also be nice if releases
>> could be more frequent than once a year.
>> 
>> I would also like to say that I’m not doing this for the fun of it. My
>> company uses Flume for some of its most critical processing. We run
>> Veracode scans on all of our software and I expect Flume would be flagged
>> if I hadn’t repacked it with Log4j 2. It also may not show up since the CVE
>> is marked against Log4j 2 since Log4j 1 is EOL, but the security scanning
>> tools should be flagging that as well.
>> 
>> FWIW I’ve had to hack or enhance a few Flume components to make it work
>> for my needs but overall it works really, really well. I’d like to commit
>> back some of the changes but the main one - Flume configuration - I really
>> don’t like and need to redo the whole thing.
>> 
>> Ralph
>> 
>>> On Nov 24, 2018, at 5:11 PM, Mike Percy  wrote:
>>> 
>>> I wasn't aware of this security issue. Do you have a link to the details?
>>> 
>>> There's no ASF requirement for package names; it's really just a Java
>>> language convention, and especially not enforced for compatibility shims.
>>> Even Flume has some of those in the legacy sources.
>>> 
>>> I understand that it's very difficult to provide a compatibility layer.
>>> Maybe also a boring task. I'm just saying that without log4j1 backwards
>>> compatibility provided by log4j2, there will have to be a really critical
>>> reason to inflict this migration pain on Flume users -- something that
>>> simply can't be tolerated, even in a minor release, like a new and highly
>>> dangerous security bug. Without such a motivation, I don't see how this
>>> incompatible dependency change can be justified.
>>> 
>>> Mike
>>> 
>>> On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers 
>>> wrote:
>>> 
 
> On Nov 24, 2018, at 2:35 PM, Mike Percy  wrote:
> 
> Flume has long had a policy of backwards compatibility with its own
> configuration files and people expect things to "just work" when
 upgrading
> Flume. If log4j2 can't parse the log4j1 config file format then it's an
> incompatible upgrade and should not be done in a minor Flume release.
> 
> If log4j2 wants to be a drop-in replacement for log4j1 then by default
>> it
> should find and parse the traditional log4j.properties config files, at
> least as a fallback, rather than force users to convert to the new XML
> format before upgrading.
 
 That is simply not possible:
 1. Log4j 1 requires the use of fully qualified class names. The package
 names log4j 1 used didn’t conform to ASF naming guidelines and don’t
>> line
 up with the package names used in Log4j2.
 2. Log4j 1 did not delineate between what was public and what was
>> private
 so there is code all over the place mucking with the internals of Log4j.
 Log4j 2 implemented a compatibility layer by implementing the classes
>> that
 are being used but by 

Re: [DISCUSS] Flume 1.9 release proposal

[Mike Percy]

While the committer veto rules are well documented here
, and there is no
mention of a time limit, I propose we shelve that discussion and work on
getting to consensus on what we as a community want to include in the Flume
1.9 release.

As far as I can tell, the decision we have to make is whether to include
the FLUME-2050 logging changes in the Flume 1.9 release. We can either:

1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to modify
their log4j configuration files to use the different log4j2 XML format,
with the procedure for doing so documented in the release notes.
-or-
2) Defer that change to a future release where incompatible changes are
expected, such as Flume 2.0.

Also, maybe there are other options I haven't thought of...?

I would like to get some input from more people on this matter. How do
others feel about this?

Thanks,
Mike


On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers 
wrote:

> I should also point out that the time to raise this was a year ago when
> the PR for FLUME-2050 was reviewed and committed. As it stands now I could
> be a jerk and vote -1 on the patch for FLUME-3296 with valid technical
> grounds. If this was causing a true binary incompatibility I would approve
> reverting it in a heartbeat, but I just don’t see how having users have to
> change logging configuration is “intolerable”, especially with the known
> security issues in Log4j 1, however unlikely user’s might be to encounter
> them.
>
> That said, I wouldn’t veto the revert on the release branch, but I would
> suggest that the release notes provide fair warning that the next release
> will upgrade the logging dependency.  It would also be nice if releases
> could be more frequent than once a year.
>
> I would also like to say that I’m not doing this for the fun of it. My
> company uses Flume for some of its most critical processing. We run
> Veracode scans on all of our software and I expect Flume would be flagged
> if I hadn’t repacked it with Log4j 2. It also may not show up since the CVE
> is marked against Log4j 2 since Log4j 1 is EOL, but the security scanning
> tools should be flagging that as well.
>
> FWIW I’ve had to hack or enhance a few Flume components to make it work
> for my needs but overall it works really, really well. I’d like to commit
> back some of the changes but the main one - Flume configuration - I really
> don’t like and need to redo the whole thing.
>
> Ralph
>
> > On Nov 24, 2018, at 5:11 PM, Mike Percy  wrote:
> >
> > I wasn't aware of this security issue. Do you have a link to the details?
> >
> > There's no ASF requirement for package names; it's really just a Java
> > language convention, and especially not enforced for compatibility shims.
> > Even Flume has some of those in the legacy sources.
> >
> > I understand that it's very difficult to provide a compatibility layer.
> > Maybe also a boring task. I'm just saying that without log4j1 backwards
> > compatibility provided by log4j2, there will have to be a really critical
> > reason to inflict this migration pain on Flume users -- something that
> > simply can't be tolerated, even in a minor release, like a new and highly
> > dangerous security bug. Without such a motivation, I don't see how this
> > incompatible dependency change can be justified.
> >
> > Mike
> >
> > On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers 
> > wrote:
> >
> >>
> >>> On Nov 24, 2018, at 2:35 PM, Mike Percy  wrote:
> >>>
> >>> Flume has long had a policy of backwards compatibility with its own
> >>> configuration files and people expect things to "just work" when
> >> upgrading
> >>> Flume. If log4j2 can't parse the log4j1 config file format then it's an
> >>> incompatible upgrade and should not be done in a minor Flume release.
> >>>
> >>> If log4j2 wants to be a drop-in replacement for log4j1 then by default
> it
> >>> should find and parse the traditional log4j.properties config files, at
> >>> least as a fallback, rather than force users to convert to the new XML
> >>> format before upgrading.
> >>
> >> That is simply not possible:
> >> 1. Log4j 1 requires the use of fully qualified class names. The package
> >> names log4j 1 used didn’t conform to ASF naming guidelines and don’t
> line
> >> up with the package names used in Log4j2.
> >> 2. Log4j 1 did not delineate between what was public and what was
> private
> >> so there is code all over the place mucking with the internals of Log4j.
> >> Log4j 2 implemented a compatibility layer by implementing the classes
> that
> >> are being used but by and large they don’t actually do anything.
> >> 3. The Appenders and Filters in Log4j 2 implement different interfaces
> and
> >> cannot use components from Log4j 1.
> >> 4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact,
> many
> >> people didn’t use the Rolling File Appender that shipped with Log4j but
> >> used the one from Log4j extras. So it is hard to.know what Log4j 2 would

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

I should also point out that the time to raise this was a year ago when the PR 
for FLUME-2050 was reviewed and committed. As it stands now I could be a jerk 
and vote -1 on the patch for FLUME-3296 with valid technical grounds. If this 
was causing a true binary incompatibility I would approve reverting it in a 
heartbeat, but I just don’t see how having users have to change logging 
configuration is “intolerable”, especially with the known security issues in 
Log4j 1, however unlikely user’s might be to encounter them.

That said, I wouldn’t veto the revert on the release branch, but I would 
suggest that the release notes provide fair warning that the next release will 
upgrade the logging dependency.  It would also be nice if releases could be 
more frequent than once a year.

I would also like to say that I’m not doing this for the fun of it. My company 
uses Flume for some of its most critical processing. We run Veracode scans on 
all of our software and I expect Flume would be flagged if I hadn’t repacked it 
with Log4j 2. It also may not show up since the CVE is marked against Log4j 2 
since Log4j 1 is EOL, but the security scanning tools should be flagging that 
as well.

FWIW I’ve had to hack or enhance a few Flume components to make it work for my 
needs but overall it works really, really well. I’d like to commit back some of 
the changes but the main one - Flume configuration - I really don’t like and 
need to redo the whole thing. 

Ralph

> On Nov 24, 2018, at 5:11 PM, Mike Percy  wrote:
> 
> I wasn't aware of this security issue. Do you have a link to the details?
> 
> There's no ASF requirement for package names; it's really just a Java
> language convention, and especially not enforced for compatibility shims.
> Even Flume has some of those in the legacy sources.
> 
> I understand that it's very difficult to provide a compatibility layer.
> Maybe also a boring task. I'm just saying that without log4j1 backwards
> compatibility provided by log4j2, there will have to be a really critical
> reason to inflict this migration pain on Flume users -- something that
> simply can't be tolerated, even in a minor release, like a new and highly
> dangerous security bug. Without such a motivation, I don't see how this
> incompatible dependency change can be justified.
> 
> Mike
> 
> On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers 
> wrote:
> 
>> 
>>> On Nov 24, 2018, at 2:35 PM, Mike Percy  wrote:
>>> 
>>> Flume has long had a policy of backwards compatibility with its own
>>> configuration files and people expect things to "just work" when
>> upgrading
>>> Flume. If log4j2 can't parse the log4j1 config file format then it's an
>>> incompatible upgrade and should not be done in a minor Flume release.
>>> 
>>> If log4j2 wants to be a drop-in replacement for log4j1 then by default it
>>> should find and parse the traditional log4j.properties config files, at
>>> least as a fallback, rather than force users to convert to the new XML
>>> format before upgrading.
>> 
>> That is simply not possible:
>> 1. Log4j 1 requires the use of fully qualified class names. The package
>> names log4j 1 used didn’t conform to ASF naming guidelines and don’t line
>> up with the package names used in Log4j2.
>> 2. Log4j 1 did not delineate between what was public and what was private
>> so there is code all over the place mucking with the internals of Log4j.
>> Log4j 2 implemented a compatibility layer by implementing the classes that
>> are being used but by and large they don’t actually do anything.
>> 3. The Appenders and Filters in Log4j 2 implement different interfaces and
>> cannot use components from Log4j 1.
>> 4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact, many
>> people didn’t use the Rolling File Appender that shipped with Log4j but
>> used the one from Log4j extras. So it is hard to.know what Log4j 2 would
>> have needed to be compatible with.
>> 
>> Many organizations (including mine) have security requirements that say
>> they must use software that is supported with security fixes. Log4j 1 has a
>> known security bug that will never be fixed as it reached end-of-life in
>> August of 2015. This means the use of Log4j 1 is not acceptable for any
>> security conscious organization.
>> 
>> While folks have brought up this issue from time to time most seem to
>> adapt quite quickly to the change. Most of the issues are developers
>> wanting to know how to make something they were doing in Log4j 1 work in
>> Log4j 2.
>> 
>> As you can imagine, since Log4j 2 has been GA now for 4 1/2 years and
>> Log4j 1 has been EOL for over 3, making the configuration compatible with
>> Log4j 1 is not a high priority. FWIW, there was an effort to do that a few
>> years ago, and while we were able to get some basic stuff to work making it
>> work in a general way wasn’t possible.
>> 
>> Ralph
>> 
>>> 
>>> Mike
>>> 
>>> On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers 
>>> wrote:
>>> 
 No, you are correct. However, 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

One of the security issues is https://www.cvedetails.com/cve/CVE-2017-5645/ 
. It is reported against log4j 2 
because Log4j 1 is EOL, but the same vulnerability exists there. RedHat has 
created a fix for Log4j 1 for its Linux distribution [1] but I don’t know how 
valuable that would be. This involves using the SocketAppender. The bug in 
Log4j 1 is actually worse than in Log4j 2 as the Log4j 1 SocketAppender does 
not accept a Layout and requires the use of Java Serialization. So the 
SocketServer deserializes Java objects without a whitelist. Log4j 2 handled 
this by deprecating the SerializaedLayout and recommending that JSONLayout be 
used instead. We had a bit of a discussion with the folks at Logstash and 
unfortunately they completely misunderstood the problem and the resolution.

There was another security bug that was addressed in Log4j 2 that also exists 
in Log4j 1, although I don’t have the CVE number handy, but you can find the 
details about it in the Log4j private list. This one had to do with using an 
XML configuration. It is possible to steal files from the local file system by 
using XML entity tags. Log4j 2 disables this capability by default. I believe 
Log4j 1 didn’t. I don’t consider this to be a very high security risk because 
you would need access to modify the config file to do it, but security people 
don’t think like I do.

Ralph


1. https://www.redhat.com/archives/rhsa-announce/2017-August/msg00038.html

> On Nov 24, 2018, at 5:11 PM, Mike Percy  wrote:
> 
> I wasn't aware of this security issue. Do you have a link to the details?
> 
> There's no ASF requirement for package names; it's really just a Java
> language convention, and especially not enforced for compatibility shims.
> Even Flume has some of those in the legacy sources.
> 
> I understand that it's very difficult to provide a compatibility layer.
> Maybe also a boring task. I'm just saying that without log4j1 backwards
> compatibility provided by log4j2, there will have to be a really critical
> reason to inflict this migration pain on Flume users -- something that
> simply can't be tolerated, even in a minor release, like a new and highly
> dangerous security bug. Without such a motivation, I don't see how this
> incompatible dependency change can be justified.
> 
> Mike
> 
> On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers 
> wrote:
> 
>> 
>>> On Nov 24, 2018, at 2:35 PM, Mike Percy  wrote:
>>> 
>>> Flume has long had a policy of backwards compatibility with its own
>>> configuration files and people expect things to "just work" when
>> upgrading
>>> Flume. If log4j2 can't parse the log4j1 config file format then it's an
>>> incompatible upgrade and should not be done in a minor Flume release.
>>> 
>>> If log4j2 wants to be a drop-in replacement for log4j1 then by default it
>>> should find and parse the traditional log4j.properties config files, at
>>> least as a fallback, rather than force users to convert to the new XML
>>> format before upgrading.
>> 
>> That is simply not possible:
>> 1. Log4j 1 requires the use of fully qualified class names. The package
>> names log4j 1 used didn’t conform to ASF naming guidelines and don’t line
>> up with the package names used in Log4j2.
>> 2. Log4j 1 did not delineate between what was public and what was private
>> so there is code all over the place mucking with the internals of Log4j.
>> Log4j 2 implemented a compatibility layer by implementing the classes that
>> are being used but by and large they don’t actually do anything.
>> 3. The Appenders and Filters in Log4j 2 implement different interfaces and
>> cannot use components from Log4j 1.
>> 4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact, many
>> people didn’t use the Rolling File Appender that shipped with Log4j but
>> used the one from Log4j extras. So it is hard to.know what Log4j 2 would
>> have needed to be compatible with.
>> 
>> Many organizations (including mine) have security requirements that say
>> they must use software that is supported with security fixes. Log4j 1 has a
>> known security bug that will never be fixed as it reached end-of-life in
>> August of 2015. This means the use of Log4j 1 is not acceptable for any
>> security conscious organization.
>> 
>> While folks have brought up this issue from time to time most seem to
>> adapt quite quickly to the change. Most of the issues are developers
>> wanting to know how to make something they were doing in Log4j 1 work in
>> Log4j 2.
>> 
>> As you can imagine, since Log4j 2 has been GA now for 4 1/2 years and
>> Log4j 1 has been EOL for over 3, making the configuration compatible with
>> Log4j 1 is not a high priority. FWIW, there was an effort to do that a few
>> years ago, and while we were able to get some basic stuff to work making it
>> work in a general way wasn’t possible.
>> 
>> Ralph
>> 
>>> 
>>> Mike
>>> 
>>> On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers 
>>> wrote:
>>> 
 

Re: [DISCUSS] Flume 1.9 release proposal

[Mike Percy]

I wasn't aware of this security issue. Do you have a link to the details?

There's no ASF requirement for package names; it's really just a Java
language convention, and especially not enforced for compatibility shims.
Even Flume has some of those in the legacy sources.

I understand that it's very difficult to provide a compatibility layer.
Maybe also a boring task. I'm just saying that without log4j1 backwards
compatibility provided by log4j2, there will have to be a really critical
reason to inflict this migration pain on Flume users -- something that
simply can't be tolerated, even in a minor release, like a new and highly
dangerous security bug. Without such a motivation, I don't see how this
incompatible dependency change can be justified.

Mike

On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers 
wrote:

>
> > On Nov 24, 2018, at 2:35 PM, Mike Percy  wrote:
> >
> > Flume has long had a policy of backwards compatibility with its own
> > configuration files and people expect things to "just work" when
> upgrading
> > Flume. If log4j2 can't parse the log4j1 config file format then it's an
> > incompatible upgrade and should not be done in a minor Flume release.
> >
> > If log4j2 wants to be a drop-in replacement for log4j1 then by default it
> > should find and parse the traditional log4j.properties config files, at
> > least as a fallback, rather than force users to convert to the new XML
> > format before upgrading.
>
> That is simply not possible:
> 1. Log4j 1 requires the use of fully qualified class names. The package
> names log4j 1 used didn’t conform to ASF naming guidelines and don’t line
> up with the package names used in Log4j2.
> 2. Log4j 1 did not delineate between what was public and what was private
> so there is code all over the place mucking with the internals of Log4j.
> Log4j 2 implemented a compatibility layer by implementing the classes that
> are being used but by and large they don’t actually do anything.
> 3. The Appenders and Filters in Log4j 2 implement different interfaces and
> cannot use components from Log4j 1.
> 4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact, many
> people didn’t use the Rolling File Appender that shipped with Log4j but
> used the one from Log4j extras. So it is hard to.know what Log4j 2 would
> have needed to be compatible with.
>
> Many organizations (including mine) have security requirements that say
> they must use software that is supported with security fixes. Log4j 1 has a
> known security bug that will never be fixed as it reached end-of-life in
> August of 2015. This means the use of Log4j 1 is not acceptable for any
> security conscious organization.
>
> While folks have brought up this issue from time to time most seem to
> adapt quite quickly to the change. Most of the issues are developers
> wanting to know how to make something they were doing in Log4j 1 work in
> Log4j 2.
>
> As you can imagine, since Log4j 2 has been GA now for 4 1/2 years and
> Log4j 1 has been EOL for over 3, making the configuration compatible with
> Log4j 1 is not a high priority. FWIW, there was an effort to do that a few
> years ago, and while we were able to get some basic stuff to work making it
> work in a general way wasn’t possible.
>
> Ralph
>
> >
> > Mike
> >
> > On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers 
> > wrote:
> >
> >> No, you are correct. However, requiring a change to logging
> configuration
> >> has never been considered a binary compatibility break in any project I
> >> have ever worked on.
> >>
> >> Ralph
> >>
> >>> On Nov 23, 2018, at 10:05 AM, Ferenc Szabo  >
> >> wrote:
> >>>
> >>> As you mentioned, you have the freedom to use Log4j 2 and the same time
> >> we
> >>> have to keep the out of the box experience the same in a minor version.
> >>> Users should be able to upgrade flume without changing any of their
> >>> configurations.
> >>> If they have a log4j.properties (Log4j 1) then they would not be able
> to
> >>> use it after the upgrade without changing it.
> >>>
> >>> Or am I missing a feature that would solve this case?
> >>>
> >>> On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers <
> ralph.go...@dslextreme.com>
> >>> wrote:
> >>>
>  Also, please put details in the Jira issues. It is much easier to find
> >> out
>  why something was done by searching Jira later on then searching the
>  mailing list.
> 
>  Ralph
> 
> > On Nov 23, 2018, at 9:47 AM, Ralph Goers  >
>  wrote:
> >
> > I do not understand this at all. Log4j 2 provides runtime
> compatibility
>  with Log4j 1. What is the problem that requires a revert?
> >
> > I have been running Flume with Log4j 2 since 1.6 so I don’t
> understand
>  what the problem could possibly be.
> >
> > Ralph
> >
> >> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo 
>  wrote:
> >>
> >> Hi everyone
> >>
> >> I am about to branch the 1.9 release from trunk.
> >>
> >> On the 1.9 branch we will revert the 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

> On Nov 24, 2018, at 2:35 PM, Mike Percy  wrote:
> 
> Flume has long had a policy of backwards compatibility with its own
> configuration files and people expect things to "just work" when upgrading
> Flume. If log4j2 can't parse the log4j1 config file format then it's an
> incompatible upgrade and should not be done in a minor Flume release.
> 
> If log4j2 wants to be a drop-in replacement for log4j1 then by default it
> should find and parse the traditional log4j.properties config files, at
> least as a fallback, rather than force users to convert to the new XML
> format before upgrading.

That is simply not possible:
1. Log4j 1 requires the use of fully qualified class names. The package names 
log4j 1 used didn’t conform to ASF naming guidelines and don’t line up with the 
package names used in Log4j2.
2. Log4j 1 did not delineate between what was public and what was private so 
there is code all over the place mucking with the internals of Log4j. Log4j 2 
implemented a compatibility layer by implementing the classes that are being 
used but by and large they don’t actually do anything. 
3. The Appenders and Filters in Log4j 2 implement different interfaces and 
cannot use components from Log4j 1.
4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact, many 
people didn’t use the Rolling File Appender that shipped with Log4j but used 
the one from Log4j extras. So it is hard to.know what Log4j 2 would have needed 
to be compatible with.

Many organizations (including mine) have security requirements that say they 
must use software that is supported with security fixes. Log4j 1 has a known 
security bug that will never be fixed as it reached end-of-life in August of 
2015. This means the use of Log4j 1 is not acceptable for any security 
conscious organization.

While folks have brought up this issue from time to time most seem to adapt 
quite quickly to the change. Most of the issues are developers wanting to know 
how to make something they were doing in Log4j 1 work in Log4j 2. 

As you can imagine, since Log4j 2 has been GA now for 4 1/2 years and Log4j 1 
has been EOL for over 3, making the configuration compatible with Log4j 1 is 
not a high priority. FWIW, there was an effort to do that a few years ago, and 
while we were able to get some basic stuff to work making it work in a general 
way wasn’t possible. 

Ralph

> 
> Mike
> 
> On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers 
> wrote:
> 
>> No, you are correct. However, requiring a change to logging configuration
>> has never been considered a binary compatibility break in any project I
>> have ever worked on.
>> 
>> Ralph
>> 
>>> On Nov 23, 2018, at 10:05 AM, Ferenc Szabo 
>> wrote:
>>> 
>>> As you mentioned, you have the freedom to use Log4j 2 and the same time
>> we
>>> have to keep the out of the box experience the same in a minor version.
>>> Users should be able to upgrade flume without changing any of their
>>> configurations.
>>> If they have a log4j.properties (Log4j 1) then they would not be able to
>>> use it after the upgrade without changing it.
>>> 
>>> Or am I missing a feature that would solve this case?
>>> 
>>> On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers 
>>> wrote:
>>> 
 Also, please put details in the Jira issues. It is much easier to find
>> out
 why something was done by searching Jira later on then searching the
 mailing list.
 
 Ralph
 
> On Nov 23, 2018, at 9:47 AM, Ralph Goers 
 wrote:
> 
> I do not understand this at all. Log4j 2 provides runtime compatibility
 with Log4j 1. What is the problem that requires a revert?
> 
> I have been running Flume with Log4j 2 since 1.6 so I don’t understand
 what the problem could possibly be.
> 
> Ralph
> 
>> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo 
 wrote:
>> 
>> Hi everyone
>> 
>> I am about to branch the 1.9 release from trunk.
>> 
>> On the 1.9 branch we will revert the following breaking changes:
>> - FLUME-2957. Remove Guava from our public API:
>> 
>> 
 
>> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
>> - part of FLUME-2050. Upgrade to Log4j 2.10.0:
>> as the new release should work with the previous configurations we
>> have
>> to release it with log4j 1.x
>> For the log4j2 upgrade, we will provide a guide, how to replace the
>> jars
>> if users would like to start using it in the 1.9 release on the wiki
 page.
>> 
>> Because of these changes the first release candidate might be
>> postponed
 to
>> Monday.
>> 
>> Regards,
>> Ferenc Szabo
>> 
>> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com <
>> ema...@cloudera.com
> 
>> wrote:
>> 
>>> Hi Ferenc,
>>> 
>>> +1
>>> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able
 to
>>> finish it
>>> till the suggested deadline.
>>> I am also happy 

Re: [DISCUSS] Flume 1.9 release proposal

[Mike Percy]

Flume has long had a policy of backwards compatibility with its own
configuration files and people expect things to "just work" when upgrading
Flume. If log4j2 can't parse the log4j1 config file format then it's an
incompatible upgrade and should not be done in a minor Flume release.

If log4j2 wants to be a drop-in replacement for log4j1 then by default it
should find and parse the traditional log4j.properties config files, at
least as a fallback, rather than force users to convert to the new XML
format before upgrading.

Mike

On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers 
wrote:

> No, you are correct. However, requiring a change to logging configuration
> has never been considered a binary compatibility break in any project I
> have ever worked on.
>
> Ralph
>
> > On Nov 23, 2018, at 10:05 AM, Ferenc Szabo 
> wrote:
> >
> > As you mentioned, you have the freedom to use Log4j 2 and the same time
> we
> > have to keep the out of the box experience the same in a minor version.
> > Users should be able to upgrade flume without changing any of their
> > configurations.
> > If they have a log4j.properties (Log4j 1) then they would not be able to
> > use it after the upgrade without changing it.
> >
> > Or am I missing a feature that would solve this case?
> >
> > On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers 
> > wrote:
> >
> >> Also, please put details in the Jira issues. It is much easier to find
> out
> >> why something was done by searching Jira later on then searching the
> >> mailing list.
> >>
> >> Ralph
> >>
> >>> On Nov 23, 2018, at 9:47 AM, Ralph Goers 
> >> wrote:
> >>>
> >>> I do not understand this at all. Log4j 2 provides runtime compatibility
> >> with Log4j 1. What is the problem that requires a revert?
> >>>
> >>> I have been running Flume with Log4j 2 since 1.6 so I don’t understand
> >> what the problem could possibly be.
> >>>
> >>> Ralph
> >>>
>  On Nov 23, 2018, at 8:50 AM, Ferenc Szabo 
> >> wrote:
> 
>  Hi everyone
> 
>  I am about to branch the 1.9 release from trunk.
> 
>  On the 1.9 branch we will revert the following breaking changes:
>  - FLUME-2957. Remove Guava from our public API:
> 
> 
> >>
> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
>  - part of FLUME-2050. Upgrade to Log4j 2.10.0:
>  as the new release should work with the previous configurations we
> have
>  to release it with log4j 1.x
>  For the log4j2 upgrade, we will provide a guide, how to replace the
> jars
>  if users would like to start using it in the 1.9 release on the wiki
> >> page.
> 
>  Because of these changes the first release candidate might be
> postponed
> >> to
>  Monday.
> 
>  Regards,
>  Ferenc Szabo
> 
>  On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com <
> ema...@cloudera.com
> >>>
>  wrote:
> 
> > Hi Ferenc,
> >
> > +1
> > I am working on FLUME-3281 Update to Kafka 2.0 client, should be able
> >> to
> > finish it
> > till the suggested deadline.
> > I am also happy to do some reviews.
> >
> > Regards
> > Endre
> >
> > On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
> >> Hello Flume Community,
> >>
> >> 1.8 was released about a year ago and since that quite a few bug
> >> fixes,
> >> improvements, features and documentation were introduced.
> >> I would like to propose to publish the next minor release of Flume
> >> to make these changes available to the users.
> >>
> >> I would be more than happy to be the Release Manager with the help
> of
> >> Denes Arvay for anything that requires PMC access - if both the
> >> community
> >> and he are
> >> OK with it.
> >>
> >> Among others the following changes will be included in the next
> >> release:
> >>
> >> Fixed bugs:
> >> - FLUME-3117 Application can be dead loop when call System.exit() in
> >> methodconfigure
> >> - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider
> >> in
> >> JMSSource
> >> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
> > correctly
> >> - FLUME-3056 TestApplication hangs indefinitely
> >> - FLUME-2976 Exception when JMS source tries to connect to a
> Weblogic
> >> server without authentication
> >> - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor
> in
> > case
> >> of failure
> >> - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
> > being
> >> deleted from the TAILDIR source
> >> - FLUME-2894 Flume components should stop in the correct order
> >> (graceful
> >> shutdown)
> >> - FLUME-2973 Deadlock in hdfs sink
> >> - FLUME-3278 Handling -D keystore parameters in Kafka components
> >> - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
> >>
> >>
> >> Improvements:
> >> - FLUME-3186 Make asyncHbaseClient configuration 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

No, you are correct. However, requiring a change to logging configuration has 
never been considered a binary compatibility break in any project I have ever 
worked on.

Ralph

> On Nov 23, 2018, at 10:05 AM, Ferenc Szabo  
> wrote:
> 
> As you mentioned, you have the freedom to use Log4j 2 and the same time we
> have to keep the out of the box experience the same in a minor version.
> Users should be able to upgrade flume without changing any of their
> configurations.
> If they have a log4j.properties (Log4j 1) then they would not be able to
> use it after the upgrade without changing it.
> 
> Or am I missing a feature that would solve this case?
> 
> On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers 
> wrote:
> 
>> Also, please put details in the Jira issues. It is much easier to find out
>> why something was done by searching Jira later on then searching the
>> mailing list.
>> 
>> Ralph
>> 
>>> On Nov 23, 2018, at 9:47 AM, Ralph Goers 
>> wrote:
>>> 
>>> I do not understand this at all. Log4j 2 provides runtime compatibility
>> with Log4j 1. What is the problem that requires a revert?
>>> 
>>> I have been running Flume with Log4j 2 since 1.6 so I don’t understand
>> what the problem could possibly be.
>>> 
>>> Ralph
>>> 
 On Nov 23, 2018, at 8:50 AM, Ferenc Szabo 
>> wrote:
 
 Hi everyone
 
 I am about to branch the 1.9 release from trunk.
 
 On the 1.9 branch we will revert the following breaking changes:
 - FLUME-2957. Remove Guava from our public API:
 
 
>> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
 - part of FLUME-2050. Upgrade to Log4j 2.10.0:
 as the new release should work with the previous configurations we have
 to release it with log4j 1.x
 For the log4j2 upgrade, we will provide a guide, how to replace the jars
 if users would like to start using it in the 1.9 release on the wiki
>> page.
 
 Because of these changes the first release candidate might be postponed
>> to
 Monday.
 
 Regards,
 Ferenc Szabo
 
 On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com >> 
 wrote:
 
> Hi Ferenc,
> 
> +1
> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able
>> to
> finish it
> till the suggested deadline.
> I am also happy to do some reviews.
> 
> Regards
> Endre
> 
> On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
>> Hello Flume Community,
>> 
>> 1.8 was released about a year ago and since that quite a few bug
>> fixes,
>> improvements, features and documentation were introduced.
>> I would like to propose to publish the next minor release of Flume
>> to make these changes available to the users.
>> 
>> I would be more than happy to be the Release Manager with the help of
>> Denes Arvay for anything that requires PMC access - if both the
>> community
>> and he are
>> OK with it.
>> 
>> Among others the following changes will be included in the next
>> release:
>> 
>> Fixed bugs:
>> - FLUME-3117 Application can be dead loop when call System.exit() in
>> methodconfigure
>> - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider
>> in
>> JMSSource
>> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
> correctly
>> - FLUME-3056 TestApplication hangs indefinitely
>> - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
>> server without authentication
>> - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
> case
>> of failure
>> - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
> being
>> deleted from the TAILDIR source
>> - FLUME-2894 Flume components should stop in the correct order
>> (graceful
>> shutdown)
>> - FLUME-2973 Deadlock in hdfs sink
>> - FLUME-3278 Handling -D keystore parameters in Kafka components
>> - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
>> 
>> 
>> Improvements:
>> - FLUME-3186 Make asyncHbaseClient configuration parameters available
> from
>> flume config
>> - FLUME-3239 Do not rename files in SpoolDirectorySource
>> - FLUME-3227 Add Rate Limiter to StressSource
>> - FLUME-2050 Upgrade to log4j2 (when GA)
>> - FLUME-3246 Validate flume configuration to prevent larger source
>> batch
>> size than the channel transaction capacity
>> - FLUME-3050 add counters for error conditions and expose to monitor
>> URL
>> - FLUME-3269 Support JSSE keystore/trustore -D system properties
>> - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
>> recoverLease
>> - FLUME-3276 Components supporting SSL/TLS should be able to specify
> cipher
>> suite list
>> - FLUME-3275 Components supporting SSL/TLS should be able to specify
>> protocol list
>> 
>> 
>> New Features:

Re: [DISCUSS] Flume 1.9 release proposal

[Ferenc Szabo]

Sure, we will wait. Thank you for your remarks!

On Fri, Nov 23, 2018 at 6:04 PM Ralph Goers 
wrote:

> I should also note that since today is a semi-holiday in the US please do
> not commit anything with regards to this until this is cleared up as I may
> or may not be able to respond immediately.
>
> Ralph
>
> > On Nov 23, 2018, at 9:47 AM, Ralph Goers 
> wrote:
> >
> > I do not understand this at all. Log4j 2 provides runtime compatibility
> with Log4j 1. What is the problem that requires a revert?
> >
> > I have been running Flume with Log4j 2 since 1.6 so I don’t understand
> what the problem could possibly be.
> >
> > Ralph
> >
> >> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo 
> wrote:
> >>
> >> Hi everyone
> >>
> >> I am about to branch the 1.9 release from trunk.
> >>
> >> On the 1.9 branch we will revert the following breaking changes:
> >> - FLUME-2957. Remove Guava from our public API:
> >>
> >>
> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
> >> - part of FLUME-2050. Upgrade to Log4j 2.10.0:
> >> as the new release should work with the previous configurations we have
> >> to release it with log4j 1.x
> >> For the log4j2 upgrade, we will provide a guide, how to replace the jars
> >> if users would like to start using it in the 1.9 release on the wiki
> page.
> >>
> >> Because of these changes the first release candidate might be postponed
> to
> >> Monday.
> >>
> >> Regards,
> >> Ferenc Szabo
> >>
> >> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com  >
> >> wrote:
> >>
> >>> Hi Ferenc,
> >>>
> >>> +1
> >>> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able
> to
> >>> finish it
> >>> till the suggested deadline.
> >>> I am also happy to do some reviews.
> >>>
> >>> Regards
> >>> Endre
> >>>
> >>> On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
>  Hello Flume Community,
> 
>  1.8 was released about a year ago and since that quite a few bug
> fixes,
>  improvements, features and documentation were introduced.
>  I would like to propose to publish the next minor release of Flume
>  to make these changes available to the users.
> 
>  I would be more than happy to be the Release Manager with the help of
>  Denes Arvay for anything that requires PMC access - if both the
> community
>  and he are
>  OK with it.
> 
>  Among others the following changes will be included in the next
> release:
> 
>  Fixed bugs:
>  - FLUME-3117 Application can be dead loop when call System.exit() in
>  methodconfigure
>  - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider
> in
>  JMSSource
>  - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
> >>> correctly
>  - FLUME-3056 TestApplication hangs indefinitely
>  - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
>  server without authentication
>  - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
> >>> case
>  of failure
>  - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
> >>> being
>  deleted from the TAILDIR source
>  - FLUME-2894 Flume components should stop in the correct order
> (graceful
>  shutdown)
>  - FLUME-2973 Deadlock in hdfs sink
>  - FLUME-3278 Handling -D keystore parameters in Kafka components
>  - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
> 
> 
>  Improvements:
>  - FLUME-3186 Make asyncHbaseClient configuration parameters available
> >>> from
>  flume config
>  - FLUME-3239 Do not rename files in SpoolDirectorySource
>  - FLUME-3227 Add Rate Limiter to StressSource
>  - FLUME-2050 Upgrade to log4j2 (when GA)
>  - FLUME-3246 Validate flume configuration to prevent larger source
> batch
>  size than the channel transaction capacity
>  - FLUME-3050 add counters for error conditions and expose to monitor
> URL
>  - FLUME-3269 Support JSSE keystore/trustore -D system properties
>  - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
>  recoverLease
>  - FLUME-3276 Components supporting SSL/TLS should be able to specify
> >>> cipher
>  suite list
>  - FLUME-3275 Components supporting SSL/TLS should be able to specify
>  protocol list
> 
> 
>  New Features:
>  - FLUME-3142 Adding HBase2 sink
>  - FLUME-2442 Need an alternative to providing clear text passwords in
> >>> flume
>  config
> 
> 
>  There are 26 open tickets targeted for 1.9 in patch available state:
>  https://s.apache.org/flume-1.9-target-tickets
> 
>  We also have a number of open pull requests on Github:
>  https://github.com/apache/flume/pulls
> 
>  There are a few tickets in progress that would be good to include in
> the
>  release so
>  I would say that we focus on reviewing and testing them in the next 2
>  weeks.
> 
>  I would like to propose 

Re: [DISCUSS] Flume 1.9 release proposal

[Ferenc Szabo]

As you mentioned, you have the freedom to use Log4j 2 and the same time we
have to keep the out of the box experience the same in a minor version.
Users should be able to upgrade flume without changing any of their
configurations.
If they have a log4j.properties (Log4j 1) then they would not be able to
use it after the upgrade without changing it.

Or am I missing a feature that would solve this case?

On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers 
wrote:

> Also, please put details in the Jira issues. It is much easier to find out
> why something was done by searching Jira later on then searching the
> mailing list.
>
> Ralph
>
> > On Nov 23, 2018, at 9:47 AM, Ralph Goers 
> wrote:
> >
> > I do not understand this at all. Log4j 2 provides runtime compatibility
> with Log4j 1. What is the problem that requires a revert?
> >
> > I have been running Flume with Log4j 2 since 1.6 so I don’t understand
> what the problem could possibly be.
> >
> > Ralph
> >
> >> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo 
> wrote:
> >>
> >> Hi everyone
> >>
> >> I am about to branch the 1.9 release from trunk.
> >>
> >> On the 1.9 branch we will revert the following breaking changes:
> >> - FLUME-2957. Remove Guava from our public API:
> >>
> >>
> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
> >> - part of FLUME-2050. Upgrade to Log4j 2.10.0:
> >> as the new release should work with the previous configurations we have
> >> to release it with log4j 1.x
> >> For the log4j2 upgrade, we will provide a guide, how to replace the jars
> >> if users would like to start using it in the 1.9 release on the wiki
> page.
> >>
> >> Because of these changes the first release candidate might be postponed
> to
> >> Monday.
> >>
> >> Regards,
> >> Ferenc Szabo
> >>
> >> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com  >
> >> wrote:
> >>
> >>> Hi Ferenc,
> >>>
> >>> +1
> >>> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able
> to
> >>> finish it
> >>> till the suggested deadline.
> >>> I am also happy to do some reviews.
> >>>
> >>> Regards
> >>> Endre
> >>>
> >>> On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
>  Hello Flume Community,
> 
>  1.8 was released about a year ago and since that quite a few bug
> fixes,
>  improvements, features and documentation were introduced.
>  I would like to propose to publish the next minor release of Flume
>  to make these changes available to the users.
> 
>  I would be more than happy to be the Release Manager with the help of
>  Denes Arvay for anything that requires PMC access - if both the
> community
>  and he are
>  OK with it.
> 
>  Among others the following changes will be included in the next
> release:
> 
>  Fixed bugs:
>  - FLUME-3117 Application can be dead loop when call System.exit() in
>  methodconfigure
>  - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider
> in
>  JMSSource
>  - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
> >>> correctly
>  - FLUME-3056 TestApplication hangs indefinitely
>  - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
>  server without authentication
>  - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
> >>> case
>  of failure
>  - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
> >>> being
>  deleted from the TAILDIR source
>  - FLUME-2894 Flume components should stop in the correct order
> (graceful
>  shutdown)
>  - FLUME-2973 Deadlock in hdfs sink
>  - FLUME-3278 Handling -D keystore parameters in Kafka components
>  - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
> 
> 
>  Improvements:
>  - FLUME-3186 Make asyncHbaseClient configuration parameters available
> >>> from
>  flume config
>  - FLUME-3239 Do not rename files in SpoolDirectorySource
>  - FLUME-3227 Add Rate Limiter to StressSource
>  - FLUME-2050 Upgrade to log4j2 (when GA)
>  - FLUME-3246 Validate flume configuration to prevent larger source
> batch
>  size than the channel transaction capacity
>  - FLUME-3050 add counters for error conditions and expose to monitor
> URL
>  - FLUME-3269 Support JSSE keystore/trustore -D system properties
>  - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
>  recoverLease
>  - FLUME-3276 Components supporting SSL/TLS should be able to specify
> >>> cipher
>  suite list
>  - FLUME-3275 Components supporting SSL/TLS should be able to specify
>  protocol list
> 
> 
>  New Features:
>  - FLUME-3142 Adding HBase2 sink
>  - FLUME-2442 Need an alternative to providing clear text passwords in
> >>> flume
>  config
> 
> 
>  There are 26 open tickets targeted for 1.9 in patch available state:
>  https://s.apache.org/flume-1.9-target-tickets
> 
>  We also have 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

I should also note that since today is a semi-holiday in the US please do not 
commit anything with regards to this until this is cleared up as I may or may 
not be able to respond immediately. 

Ralph

> On Nov 23, 2018, at 9:47 AM, Ralph Goers  wrote:
> 
> I do not understand this at all. Log4j 2 provides runtime compatibility with 
> Log4j 1. What is the problem that requires a revert? 
> 
> I have been running Flume with Log4j 2 since 1.6 so I don’t understand what 
> the problem could possibly be.
> 
> Ralph
> 
>> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo  wrote:
>> 
>> Hi everyone
>> 
>> I am about to branch the 1.9 release from trunk.
>> 
>> On the 1.9 branch we will revert the following breaking changes:
>> - FLUME-2957. Remove Guava from our public API:
>> 
>> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
>> - part of FLUME-2050. Upgrade to Log4j 2.10.0:
>> as the new release should work with the previous configurations we have
>> to release it with log4j 1.x
>> For the log4j2 upgrade, we will provide a guide, how to replace the jars
>> if users would like to start using it in the 1.9 release on the wiki page.
>> 
>> Because of these changes the first release candidate might be postponed to
>> Monday.
>> 
>> Regards,
>> Ferenc Szabo
>> 
>> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com 
>> wrote:
>> 
>>> Hi Ferenc,
>>> 
>>> +1
>>> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able to
>>> finish it
>>> till the suggested deadline.
>>> I am also happy to do some reviews.
>>> 
>>> Regards
>>> Endre
>>> 
>>> On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
 Hello Flume Community,
 
 1.8 was released about a year ago and since that quite a few bug fixes,
 improvements, features and documentation were introduced.
 I would like to propose to publish the next minor release of Flume
 to make these changes available to the users.
 
 I would be more than happy to be the Release Manager with the help of
 Denes Arvay for anything that requires PMC access - if both the community
 and he are
 OK with it.
 
 Among others the following changes will be included in the next release:
 
 Fixed bugs:
 - FLUME-3117 Application can be dead loop when call System.exit() in
 methodconfigure
 - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider in
 JMSSource
 - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
>>> correctly
 - FLUME-3056 TestApplication hangs indefinitely
 - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
 server without authentication
 - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
>>> case
 of failure
 - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
>>> being
 deleted from the TAILDIR source
 - FLUME-2894 Flume components should stop in the correct order (graceful
 shutdown)
 - FLUME-2973 Deadlock in hdfs sink
 - FLUME-3278 Handling -D keystore parameters in Kafka components
 - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
 
 
 Improvements:
 - FLUME-3186 Make asyncHbaseClient configuration parameters available
>>> from
 flume config
 - FLUME-3239 Do not rename files in SpoolDirectorySource
 - FLUME-3227 Add Rate Limiter to StressSource
 - FLUME-2050 Upgrade to log4j2 (when GA)
 - FLUME-3246 Validate flume configuration to prevent larger source batch
 size than the channel transaction capacity
 - FLUME-3050 add counters for error conditions and expose to monitor URL
 - FLUME-3269 Support JSSE keystore/trustore -D system properties
 - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
 recoverLease
 - FLUME-3276 Components supporting SSL/TLS should be able to specify
>>> cipher
 suite list
 - FLUME-3275 Components supporting SSL/TLS should be able to specify
 protocol list
 
 
 New Features:
 - FLUME-3142 Adding HBase2 sink
 - FLUME-2442 Need an alternative to providing clear text passwords in
>>> flume
 config
 
 
 There are 26 open tickets targeted for 1.9 in patch available state:
 https://s.apache.org/flume-1.9-target-tickets
 
 We also have a number of open pull requests on Github:
 https://github.com/apache/flume/pulls
 
 There are a few tickets in progress that would be good to include in the
 release so
 I would say that we focus on reviewing and testing them in the next 2
 weeks.
 
 I would like to propose to target the 23rd of November for the first
 release candidate.
 That would mean that the branch date would be a few days before that.
 Any significant code change should get in by that date.
 
 If nobody has any concerns then I am going to create an umbrella ticket
>>> to
 track the release process.
 
 Some 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

Also, please put details in the Jira issues. It is much easier to find out why 
something was done by searching Jira later on then searching the mailing list.

Ralph

> On Nov 23, 2018, at 9:47 AM, Ralph Goers  wrote:
> 
> I do not understand this at all. Log4j 2 provides runtime compatibility with 
> Log4j 1. What is the problem that requires a revert? 
> 
> I have been running Flume with Log4j 2 since 1.6 so I don’t understand what 
> the problem could possibly be.
> 
> Ralph
> 
>> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo  wrote:
>> 
>> Hi everyone
>> 
>> I am about to branch the 1.9 release from trunk.
>> 
>> On the 1.9 branch we will revert the following breaking changes:
>> - FLUME-2957. Remove Guava from our public API:
>> 
>> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
>> - part of FLUME-2050. Upgrade to Log4j 2.10.0:
>> as the new release should work with the previous configurations we have
>> to release it with log4j 1.x
>> For the log4j2 upgrade, we will provide a guide, how to replace the jars
>> if users would like to start using it in the 1.9 release on the wiki page.
>> 
>> Because of these changes the first release candidate might be postponed to
>> Monday.
>> 
>> Regards,
>> Ferenc Szabo
>> 
>> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com 
>> wrote:
>> 
>>> Hi Ferenc,
>>> 
>>> +1
>>> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able to
>>> finish it
>>> till the suggested deadline.
>>> I am also happy to do some reviews.
>>> 
>>> Regards
>>> Endre
>>> 
>>> On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
 Hello Flume Community,
 
 1.8 was released about a year ago and since that quite a few bug fixes,
 improvements, features and documentation were introduced.
 I would like to propose to publish the next minor release of Flume
 to make these changes available to the users.
 
 I would be more than happy to be the Release Manager with the help of
 Denes Arvay for anything that requires PMC access - if both the community
 and he are
 OK with it.
 
 Among others the following changes will be included in the next release:
 
 Fixed bugs:
 - FLUME-3117 Application can be dead loop when call System.exit() in
 methodconfigure
 - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider in
 JMSSource
 - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
>>> correctly
 - FLUME-3056 TestApplication hangs indefinitely
 - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
 server without authentication
 - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
>>> case
 of failure
 - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
>>> being
 deleted from the TAILDIR source
 - FLUME-2894 Flume components should stop in the correct order (graceful
 shutdown)
 - FLUME-2973 Deadlock in hdfs sink
 - FLUME-3278 Handling -D keystore parameters in Kafka components
 - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
 
 
 Improvements:
 - FLUME-3186 Make asyncHbaseClient configuration parameters available
>>> from
 flume config
 - FLUME-3239 Do not rename files in SpoolDirectorySource
 - FLUME-3227 Add Rate Limiter to StressSource
 - FLUME-2050 Upgrade to log4j2 (when GA)
 - FLUME-3246 Validate flume configuration to prevent larger source batch
 size than the channel transaction capacity
 - FLUME-3050 add counters for error conditions and expose to monitor URL
 - FLUME-3269 Support JSSE keystore/trustore -D system properties
 - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
 recoverLease
 - FLUME-3276 Components supporting SSL/TLS should be able to specify
>>> cipher
 suite list
 - FLUME-3275 Components supporting SSL/TLS should be able to specify
 protocol list
 
 
 New Features:
 - FLUME-3142 Adding HBase2 sink
 - FLUME-2442 Need an alternative to providing clear text passwords in
>>> flume
 config
 
 
 There are 26 open tickets targeted for 1.9 in patch available state:
 https://s.apache.org/flume-1.9-target-tickets
 
 We also have a number of open pull requests on Github:
 https://github.com/apache/flume/pulls
 
 There are a few tickets in progress that would be good to include in the
 release so
 I would say that we focus on reviewing and testing them in the next 2
 weeks.
 
 I would like to propose to target the 23rd of November for the first
 release candidate.
 That would mean that the branch date would be a few days before that.
 Any significant code change should get in by that date.
 
 If nobody has any concerns then I am going to create an umbrella ticket
>>> to
 track the release process.
 
 Some movement can be expected in the related 

Re: [DISCUSS] Flume 1.9 release proposal

[Ralph Goers]

I do not understand this at all. Log4j 2 provides runtime compatibility with 
Log4j 1. What is the problem that requires a revert? 

I have been running Flume with Log4j 2 since 1.6 so I don’t understand what the 
problem could possibly be.

Ralph

> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo  wrote:
> 
> Hi everyone
> 
> I am about to branch the 1.9 release from trunk.
> 
> On the 1.9 branch we will revert the following breaking changes:
> - FLUME-2957. Remove Guava from our public API:
> 
> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
> - part of FLUME-2050. Upgrade to Log4j 2.10.0:
>  as the new release should work with the previous configurations we have
> to release it with log4j 1.x
>  For the log4j2 upgrade, we will provide a guide, how to replace the jars
> if users would like to start using it in the 1.9 release on the wiki page.
> 
> Because of these changes the first release candidate might be postponed to
> Monday.
> 
> Regards,
> Ferenc Szabo
> 
> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com 
> wrote:
> 
>> Hi Ferenc,
>> 
>> +1
>> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able to
>> finish it
>> till the suggested deadline.
>> I am also happy to do some reviews.
>> 
>> Regards
>> Endre
>> 
>> On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
>>> Hello Flume Community,
>>> 
>>> 1.8 was released about a year ago and since that quite a few bug fixes,
>>> improvements, features and documentation were introduced.
>>> I would like to propose to publish the next minor release of Flume
>>> to make these changes available to the users.
>>> 
>>> I would be more than happy to be the Release Manager with the help of
>>> Denes Arvay for anything that requires PMC access - if both the community
>>> and he are
>>> OK with it.
>>> 
>>> Among others the following changes will be included in the next release:
>>> 
>>> Fixed bugs:
>>> - FLUME-3117 Application can be dead loop when call System.exit() in
>>> methodconfigure
>>> - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider in
>>> JMSSource
>>> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
>> correctly
>>> - FLUME-3056 TestApplication hangs indefinitely
>>> - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
>>> server without authentication
>>> - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
>> case
>>> of failure
>>> - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
>> being
>>> deleted from the TAILDIR source
>>> - FLUME-2894 Flume components should stop in the correct order (graceful
>>> shutdown)
>>> - FLUME-2973 Deadlock in hdfs sink
>>> - FLUME-3278 Handling -D keystore parameters in Kafka components
>>> - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
>>> 
>>> 
>>> Improvements:
>>> - FLUME-3186 Make asyncHbaseClient configuration parameters available
>> from
>>> flume config
>>> - FLUME-3239 Do not rename files in SpoolDirectorySource
>>> - FLUME-3227 Add Rate Limiter to StressSource
>>> - FLUME-2050 Upgrade to log4j2 (when GA)
>>> - FLUME-3246 Validate flume configuration to prevent larger source batch
>>> size than the channel transaction capacity
>>> - FLUME-3050 add counters for error conditions and expose to monitor URL
>>> - FLUME-3269 Support JSSE keystore/trustore -D system properties
>>> - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
>>> recoverLease
>>> - FLUME-3276 Components supporting SSL/TLS should be able to specify
>> cipher
>>> suite list
>>> - FLUME-3275 Components supporting SSL/TLS should be able to specify
>>> protocol list
>>> 
>>> 
>>> New Features:
>>> - FLUME-3142 Adding HBase2 sink
>>> - FLUME-2442 Need an alternative to providing clear text passwords in
>> flume
>>> config
>>> 
>>> 
>>> There are 26 open tickets targeted for 1.9 in patch available state:
>>> https://s.apache.org/flume-1.9-target-tickets
>>> 
>>> We also have a number of open pull requests on Github:
>>> https://github.com/apache/flume/pulls
>>> 
>>> There are a few tickets in progress that would be good to include in the
>>> release so
>>> I would say that we focus on reviewing and testing them in the next 2
>>> weeks.
>>> 
>>> I would like to propose to target the 23rd of November for the first
>>> release candidate.
>>> That would mean that the branch date would be a few days before that.
>>> Any significant code change should get in by that date.
>>> 
>>> If nobody has any concerns then I am going to create an umbrella ticket
>> to
>>> track the release process.
>>> 
>>> Some movement can be expected in the related JIRA tickets.
>>> 
>>> Kind regards,
>>> Ferenc
>>> 
>> 

Re: [DISCUSS] Flume 1.9 release proposal

[Denes Arvay]

Hi Ferenc,

Thank you for your effort on the release so far.
I definitely agree that those breaking changes should be deferred to the
2.0 release, so +1 from me for your plan.

Feel free to ping me if you need any assistance on creating the RC.

Regards,
Denes


On Fri, Nov 23, 2018, 16:50 Ferenc Szabo  wrote:

> Hi everyone
>
> I am about to branch the 1.9 release from trunk.
>
> On the 1.9 branch we will revert the following breaking changes:
> - FLUME-2957. Remove Guava from our public API:
>
>
> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
> - part of FLUME-2050. Upgrade to Log4j 2.10.0:
>   as the new release should work with the previous configurations we have
> to release it with log4j 1.x
>   For the log4j2 upgrade, we will provide a guide, how to replace the jars
> if users would like to start using it in the 1.9 release on the wiki page.
>
> Because of these changes the first release candidate might be postponed to
> Monday.
>
> Regards,
> Ferenc Szabo
>
> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com 
> wrote:
>
> > Hi Ferenc,
> >
> > +1
> > I am working on FLUME-3281 Update to Kafka 2.0 client, should be able to
> > finish it
> > till the suggested deadline.
> > I am also happy to do some reviews.
> >
> > Regards
> > Endre
> >
> > On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
> > > Hello Flume Community,
> > >
> > > 1.8 was released about a year ago and since that quite a few bug fixes,
> > > improvements, features and documentation were introduced.
> > > I would like to propose to publish the next minor release of Flume
> > > to make these changes available to the users.
> > >
> > > I would be more than happy to be the Release Manager with the help of
> > > Denes Arvay for anything that requires PMC access - if both the
> community
> > > and he are
> > > OK with it.
> > >
> > > Among others the following changes will be included in the next
> release:
> > >
> > > Fixed bugs:
> > > - FLUME-3117 Application can be dead loop when call System.exit() in
> > > methodconfigure
> > > - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider in
> > > JMSSource
> > > - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
> > correctly
> > > - FLUME-3056 TestApplication hangs indefinitely
> > > - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
> > > server without authentication
> > > - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
> > case
> > > of failure
> > > - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
> > being
> > > deleted from the TAILDIR source
> > > - FLUME-2894 Flume components should stop in the correct order
> (graceful
> > > shutdown)
> > > - FLUME-2973 Deadlock in hdfs sink
> > > - FLUME-3278 Handling -D keystore parameters in Kafka components
> > > - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
> > >
> > >
> > > Improvements:
> > > - FLUME-3186 Make asyncHbaseClient configuration parameters available
> > from
> > > flume config
> > > - FLUME-3239 Do not rename files in SpoolDirectorySource
> > > - FLUME-3227 Add Rate Limiter to StressSource
> > > - FLUME-2050 Upgrade to log4j2 (when GA)
> > > - FLUME-3246 Validate flume configuration to prevent larger source
> batch
> > > size than the channel transaction capacity
> > > - FLUME-3050 add counters for error conditions and expose to monitor
> URL
> > > - FLUME-3269 Support JSSE keystore/trustore -D system properties
> > > - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
> > > recoverLease
> > > - FLUME-3276 Components supporting SSL/TLS should be able to specify
> > cipher
> > > suite list
> > > - FLUME-3275 Components supporting SSL/TLS should be able to specify
> > > protocol list
> > >
> > >
> > > New Features:
> > > - FLUME-3142 Adding HBase2 sink
> > > - FLUME-2442 Need an alternative to providing clear text passwords in
> > flume
> > > config
> > >
> > >
> > > There are 26 open tickets targeted for 1.9 in patch available state:
> > >  https://s.apache.org/flume-1.9-target-tickets
> > >
> > > We also have a number of open pull requests on Github:
> > > https://github.com/apache/flume/pulls
> > >
> > > There are a few tickets in progress that would be good to include in
> the
> > > release so
> > > I would say that we focus on reviewing and testing them in the next 2
> > > weeks.
> > >
> > > I would like to propose to target the 23rd of November for the first
> > > release candidate.
> > > That would mean that the branch date would be a few days before that.
> > > Any significant code change should get in by that date.
> > >
> > > If nobody has any concerns then I am going to create an umbrella ticket
> > to
> > > track the release process.
> > >
> > > Some movement can be expected in the related JIRA tickets.
> > >
> > > Kind regards,
> > > Ferenc
> > >
> >
>

Re: [DISCUSS] Flume 1.9 release proposal

[Ferenc Szabo]

Hi everyone

I am about to branch the 1.9 release from trunk.

On the 1.9 branch we will revert the following breaking changes:
- FLUME-2957. Remove Guava from our public API:

https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088
- part of FLUME-2050. Upgrade to Log4j 2.10.0:
  as the new release should work with the previous configurations we have
to release it with log4j 1.x
  For the log4j2 upgrade, we will provide a guide, how to replace the jars
if users would like to start using it in the 1.9 release on the wiki page.

Because of these changes the first release candidate might be postponed to
Monday.

Regards,
Ferenc Szabo

On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com 
wrote:

> Hi Ferenc,
>
> +1
> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able to
> finish it
> till the suggested deadline.
> I am also happy to do some reviews.
>
> Regards
> Endre
>
> On 2018/11/06 21:23:17, Ferenc Szabo  wrote:
> > Hello Flume Community,
> >
> > 1.8 was released about a year ago and since that quite a few bug fixes,
> > improvements, features and documentation were introduced.
> > I would like to propose to publish the next minor release of Flume
> > to make these changes available to the users.
> >
> > I would be more than happy to be the Release Manager with the help of
> > Denes Arvay for anything that requires PMC access - if both the community
> > and he are
> > OK with it.
> >
> > Among others the following changes will be included in the next release:
> >
> > Fixed bugs:
> > - FLUME-3117 Application can be dead loop when call System.exit() in
> > methodconfigure
> > - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider in
> > JMSSource
> > - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
> correctly
> > - FLUME-3056 TestApplication hangs indefinitely
> > - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
> > server without authentication
> > - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
> case
> > of failure
> > - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
> being
> > deleted from the TAILDIR source
> > - FLUME-2894 Flume components should stop in the correct order (graceful
> > shutdown)
> > - FLUME-2973 Deadlock in hdfs sink
> > - FLUME-3278 Handling -D keystore parameters in Kafka components
> > - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
> >
> >
> > Improvements:
> > - FLUME-3186 Make asyncHbaseClient configuration parameters available
> from
> > flume config
> > - FLUME-3239 Do not rename files in SpoolDirectorySource
> > - FLUME-3227 Add Rate Limiter to StressSource
> > - FLUME-2050 Upgrade to log4j2 (when GA)
> > - FLUME-3246 Validate flume configuration to prevent larger source batch
> > size than the channel transaction capacity
> > - FLUME-3050 add counters for error conditions and expose to monitor URL
> > - FLUME-3269 Support JSSE keystore/trustore -D system properties
> > - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
> > recoverLease
> > - FLUME-3276 Components supporting SSL/TLS should be able to specify
> cipher
> > suite list
> > - FLUME-3275 Components supporting SSL/TLS should be able to specify
> > protocol list
> >
> >
> > New Features:
> > - FLUME-3142 Adding HBase2 sink
> > - FLUME-2442 Need an alternative to providing clear text passwords in
> flume
> > config
> >
> >
> > There are 26 open tickets targeted for 1.9 in patch available state:
> >  https://s.apache.org/flume-1.9-target-tickets
> >
> > We also have a number of open pull requests on Github:
> > https://github.com/apache/flume/pulls
> >
> > There are a few tickets in progress that would be good to include in the
> > release so
> > I would say that we focus on reviewing and testing them in the next 2
> > weeks.
> >
> > I would like to propose to target the 23rd of November for the first
> > release candidate.
> > That would mean that the branch date would be a few days before that.
> > Any significant code change should get in by that date.
> >
> > If nobody has any concerns then I am going to create an umbrella ticket
> to
> > track the release process.
> >
> > Some movement can be expected in the related JIRA tickets.
> >
> > Kind regards,
> > Ferenc
> >
>

Re: [DISCUSS] Flume 1.9 release proposal

[emajor]

Hi Ferenc,

+1
I am working on FLUME-3281 Update to Kafka 2.0 client, should be able to finish 
it 
till the suggested deadline.
I am also happy to do some reviews.

Regards
Endre

On 2018/11/06 21:23:17, Ferenc Szabo  wrote: 
> Hello Flume Community,
> 
> 1.8 was released about a year ago and since that quite a few bug fixes,
> improvements, features and documentation were introduced.
> I would like to propose to publish the next minor release of Flume
> to make these changes available to the users.
> 
> I would be more than happy to be the Release Manager with the help of
> Denes Arvay for anything that requires PMC access - if both the community
> and he are
> OK with it.
> 
> Among others the following changes will be included in the next release:
> 
> Fixed bugs:
> - FLUME-3117 Application can be dead loop when call System.exit() in
> methodconfigure
> - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider in
> JMSSource
> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december correctly
> - FLUME-3056 TestApplication hangs indefinitely
> - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
> server without authentication
> - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in case
> of failure
> - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are being
> deleted from the TAILDIR source
> - FLUME-2894 Flume components should stop in the correct order (graceful
> shutdown)
> - FLUME-2973 Deadlock in hdfs sink
> - FLUME-3278 Handling -D keystore parameters in Kafka components
> - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
> 
> 
> Improvements:
> - FLUME-3186 Make asyncHbaseClient configuration parameters available from
> flume config
> - FLUME-3239 Do not rename files in SpoolDirectorySource
> - FLUME-3227 Add Rate Limiter to StressSource
> - FLUME-2050 Upgrade to log4j2 (when GA)
> - FLUME-3246 Validate flume configuration to prevent larger source batch
> size than the channel transaction capacity
> - FLUME-3050 add counters for error conditions and expose to monitor URL
> - FLUME-3269 Support JSSE keystore/trustore -D system properties
> - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
> recoverLease
> - FLUME-3276 Components supporting SSL/TLS should be able to specify cipher
> suite list
> - FLUME-3275 Components supporting SSL/TLS should be able to specify
> protocol list
> 
> 
> New Features:
> - FLUME-3142 Adding HBase2 sink
> - FLUME-2442 Need an alternative to providing clear text passwords in flume
> config
> 
> 
> There are 26 open tickets targeted for 1.9 in patch available state:
>  https://s.apache.org/flume-1.9-target-tickets
> 
> We also have a number of open pull requests on Github:
> https://github.com/apache/flume/pulls
> 
> There are a few tickets in progress that would be good to include in the
> release so
> I would say that we focus on reviewing and testing them in the next 2
> weeks.
> 
> I would like to propose to target the 23rd of November for the first
> release candidate.
> That would mean that the branch date would be a few days before that.
> Any significant code change should get in by that date.
> 
> If nobody has any concerns then I am going to create an umbrella ticket to
> track the release process.
> 
> Some movement can be expected in the related JIRA tickets.
> 
> Kind regards,
> Ferenc
> 

Re: [DISCUSS] Flume 1.9 release proposal

[Peter Turcsanyi]

Hi Ferenc,

+1
Please let me know if I can help you in any way.

At the moment I am working on FLUME-3133 (PR is expected tomorrow) which
can be added to the release too.
Then I will continue with reviewing the PR-s.

Regards,
Peter

On Wed, Nov 7, 2018 at 8:56 AM Bessenyei Balázs Donát 
wrote:

> Hi Ferenc,
>
> Thanks for volunteering to RM!
>
> +1 from me on this proposal.
>
> I am also happy to help with anything that requires PMC access, just let me
> know if there is anything I can do to help.
>
>
> Donat
>
> On Wed, Nov 7, 2018, 03:08 Mike Percy 
> > Hi Ferenc,
> > Thanks for volunteering to RM!
> >
> > +1 from me on this proposal.
> >
> > I am also happy to help with anything that requires PMC access, just let
> me
> > know if there is anything I can do to help.
> >
> > Regards,
> > Mike
> >
> >
> > On Tue, Nov 6, 2018 at 1:23 PM Ferenc Szabo 
> wrote:
> >
> > > Hello Flume Community,
> > >
> > > 1.8 was released about a year ago and since that quite a few bug fixes,
> > > improvements, features and documentation were introduced.
> > > I would like to propose to publish the next minor release of Flume
> > > to make these changes available to the users.
> > >
> > > I would be more than happy to be the Release Manager with the help of
> > > Denes Arvay for anything that requires PMC access - if both the
> community
> > > and he are
> > > OK with it.
> > >
> > > Among others the following changes will be included in the next
> release:
> > >
> > > Fixed bugs:
> > > - FLUME-3117 Application can be dead loop when call System.exit() in
> > > methodconfigure
> > > - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider in
> > > JMSSource
> > > - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
> > correctly
> > > - FLUME-3056 TestApplication hangs indefinitely
> > > - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
> > > server without authentication
> > > - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
> > case
> > > of failure
> > > - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
> > being
> > > deleted from the TAILDIR source
> > > - FLUME-2894 Flume components should stop in the correct order
> (graceful
> > > shutdown)
> > > - FLUME-2973 Deadlock in hdfs sink
> > > - FLUME-3278 Handling -D keystore parameters in Kafka components
> > > - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
> > >
> > >
> > > Improvements:
> > > - FLUME-3186 Make asyncHbaseClient configuration parameters available
> > from
> > > flume config
> > > - FLUME-3239 Do not rename files in SpoolDirectorySource
> > > - FLUME-3227 Add Rate Limiter to StressSource
> > > - FLUME-2050 Upgrade to log4j2 (when GA)
> > > - FLUME-3246 Validate flume configuration to prevent larger source
> batch
> > > size than the channel transaction capacity
> > > - FLUME-3050 add counters for error conditions and expose to monitor
> URL
> > > - FLUME-3269 Support JSSE keystore/trustore -D system properties
> > > - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
> > > recoverLease
> > > - FLUME-3276 Components supporting SSL/TLS should be able to specify
> > cipher
> > > suite list
> > > - FLUME-3275 Components supporting SSL/TLS should be able to specify
> > > protocol list
> > >
> > >
> > > New Features:
> > > - FLUME-3142 Adding HBase2 sink
> > > - FLUME-2442 Need an alternative to providing clear text passwords in
> > flume
> > > config
> > >
> > >
> > > There are 26 open tickets targeted for 1.9 in patch available state:
> > >  https://s.apache.org/flume-1.9-target-tickets
> > >
> > > We also have a number of open pull requests on Github:
> > > https://github.com/apache/flume/pulls
> > >
> > > There are a few tickets in progress that would be good to include in
> the
> > > release so
> > > I would say that we focus on reviewing and testing them in the next 2
> > > weeks.
> > >
> > > I would like to propose to target the 23rd of November for the first
> > > release candidate.
> > > That would mean that the branch date would be a few days before that.
> > > Any significant code change should get in by that date.
> > >
> > > If nobody has any concerns then I am going to create an umbrella ticket
> > to
> > > track the release process.
> > >
> > > Some movement can be expected in the related JIRA tickets.
> > >
> > > Kind regards,
> > > Ferenc
> > >
> >
>

Re: [DISCUSS] Flume 1.9 release proposal

[Bessenyei Balázs Donát]

Hi Ferenc,

Thanks for volunteering to RM!

+1 from me on this proposal.

I am also happy to help with anything that requires PMC access, just let me
know if there is anything I can do to help.


Donat

On Wed, Nov 7, 2018, 03:08 Mike Percy  Hi Ferenc,
> Thanks for volunteering to RM!
>
> +1 from me on this proposal.
>
> I am also happy to help with anything that requires PMC access, just let me
> know if there is anything I can do to help.
>
> Regards,
> Mike
>
>
> On Tue, Nov 6, 2018 at 1:23 PM Ferenc Szabo  wrote:
>
> > Hello Flume Community,
> >
> > 1.8 was released about a year ago and since that quite a few bug fixes,
> > improvements, features and documentation were introduced.
> > I would like to propose to publish the next minor release of Flume
> > to make these changes available to the users.
> >
> > I would be more than happy to be the Release Manager with the help of
> > Denes Arvay for anything that requires PMC access - if both the community
> > and he are
> > OK with it.
> >
> > Among others the following changes will be included in the next release:
> >
> > Fixed bugs:
> > - FLUME-3117 Application can be dead loop when call System.exit() in
> > methodconfigure
> > - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider in
> > JMSSource
> > - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december
> correctly
> > - FLUME-3056 TestApplication hangs indefinitely
> > - FLUME-2976 Exception when JMS source tries to connect to a Weblogic
> > server without authentication
> > - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor in
> case
> > of failure
> > - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are
> being
> > deleted from the TAILDIR source
> > - FLUME-2894 Flume components should stop in the correct order (graceful
> > shutdown)
> > - FLUME-2973 Deadlock in hdfs sink
> > - FLUME-3278 Handling -D keystore parameters in Kafka components
> > - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient
> >
> >
> > Improvements:
> > - FLUME-3186 Make asyncHbaseClient configuration parameters available
> from
> > flume config
> > - FLUME-3239 Do not rename files in SpoolDirectorySource
> > - FLUME-3227 Add Rate Limiter to StressSource
> > - FLUME-2050 Upgrade to log4j2 (when GA)
> > - FLUME-3246 Validate flume configuration to prevent larger source batch
> > size than the channel transaction capacity
> > - FLUME-3050 add counters for error conditions and expose to monitor URL
> > - FLUME-3269 Support JSSE keystore/trustore -D system properties
> > - FLUME-3223 Flume HDFS Sink should retry close prior to performing a
> > recoverLease
> > - FLUME-3276 Components supporting SSL/TLS should be able to specify
> cipher
> > suite list
> > - FLUME-3275 Components supporting SSL/TLS should be able to specify
> > protocol list
> >
> >
> > New Features:
> > - FLUME-3142 Adding HBase2 sink
> > - FLUME-2442 Need an alternative to providing clear text passwords in
> flume
> > config
> >
> >
> > There are 26 open tickets targeted for 1.9 in patch available state:
> >  https://s.apache.org/flume-1.9-target-tickets
> >
> > We also have a number of open pull requests on Github:
> > https://github.com/apache/flume/pulls
> >
> > There are a few tickets in progress that would be good to include in the
> > release so
> > I would say that we focus on reviewing and testing them in the next 2
> > weeks.
> >
> > I would like to propose to target the 23rd of November for the first
> > release candidate.
> > That would mean that the branch date would be a few days before that.
> > Any significant code change should get in by that date.
> >
> > If nobody has any concerns then I am going to create an umbrella ticket
> to
> > track the release process.
> >
> > Some movement can be expected in the related JIRA tickets.
> >
> > Kind regards,
> > Ferenc
> >
>