With the new release, you can just replace the jars on the classpath as we removed every code dependency and using slf4j. So you do not need to fork and change anything.
Let me know your decision and we will continue with that. Regards, Ferenc On Mon, Nov 26, 2018 at 12:10 PM Apache <ralph.go...@dslextreme.com> wrote: > As I said, I am fine with adding a notice to the release notes stating the > logging will be changed in the next release regardless of its version > number. This way the revert only needs to happen on the release branch for > this release. > > Ralph > > > On Nov 25, 2018, at 9:59 PM, Mike Percy <mpe...@apache.org> wrote: > > > > While the committer veto rules are well documented here > > <https://www.apache.org/foundation/voting.html#Veto>, and there is no > > mention of a time limit, I propose we shelve that discussion and work on > > getting to consensus on what we as a community want to include in the > Flume > > 1.9 release. > > > > As far as I can tell, the decision we have to make is whether to include > > the FLUME-2050 logging changes in the Flume 1.9 release. We can either: > > > > 1) Require users upgrading from Flume 1.8 to Flume 1.9 to have to modify > > their log4j configuration files to use the different log4j2 XML format, > > with the procedure for doing so documented in the release notes. > > -or- > > 2) Defer that change to a future release where incompatible changes are > > expected, such as Flume 2.0. > > > > Also, maybe there are other options I haven't thought of...? > > > > I would like to get some input from more people on this matter. How do > > others feel about this? > > > > Thanks, > > Mike > > > > > > On Sat, Nov 24, 2018 at 9:36 PM Ralph Goers <ralph.go...@dslextreme.com> > > wrote: > > > >> I should also point out that the time to raise this was a year ago when > >> the PR for FLUME-2050 was reviewed and committed. As it stands now I > could > >> be a jerk and vote -1 on the patch for FLUME-3296 with valid technical > >> grounds. If this was causing a true binary incompatibility I would > approve > >> reverting it in a heartbeat, but I just don’t see how having users have > to > >> change logging configuration is “intolerable”, especially with the known > >> security issues in Log4j 1, however unlikely user’s might be to > encounter > >> them. > >> > >> That said, I wouldn’t veto the revert on the release branch, but I would > >> suggest that the release notes provide fair warning that the next > release > >> will upgrade the logging dependency. It would also be nice if releases > >> could be more frequent than once a year. > >> > >> I would also like to say that I’m not doing this for the fun of it. My > >> company uses Flume for some of its most critical processing. We run > >> Veracode scans on all of our software and I expect Flume would be > flagged > >> if I hadn’t repacked it with Log4j 2. It also may not show up since the > CVE > >> is marked against Log4j 2 since Log4j 1 is EOL, but the security > scanning > >> tools should be flagging that as well. > >> > >> FWIW I’ve had to hack or enhance a few Flume components to make it work > >> for my needs but overall it works really, really well. I’d like to > commit > >> back some of the changes but the main one - Flume configuration - I > really > >> don’t like and need to redo the whole thing. > >> > >> Ralph > >> > >>> On Nov 24, 2018, at 5:11 PM, Mike Percy <mpe...@apache.org> wrote: > >>> > >>> I wasn't aware of this security issue. Do you have a link to the > details? > >>> > >>> There's no ASF requirement for package names; it's really just a Java > >>> language convention, and especially not enforced for compatibility > shims. > >>> Even Flume has some of those in the legacy sources. > >>> > >>> I understand that it's very difficult to provide a compatibility layer. > >>> Maybe also a boring task. I'm just saying that without log4j1 backwards > >>> compatibility provided by log4j2, there will have to be a really > critical > >>> reason to inflict this migration pain on Flume users -- something that > >>> simply can't be tolerated, even in a minor release, like a new and > highly > >>> dangerous security bug. Without such a motivation, I don't see how this > >>> incompatible dependency change can be justified. > >>> > >>> Mike > >>> > >>> On Sat, Nov 24, 2018 at 3:20 PM Ralph Goers < > ralph.go...@dslextreme.com> > >>> wrote: > >>> > >>>> > >>>>> On Nov 24, 2018, at 2:35 PM, Mike Percy <mpe...@apache.org> wrote: > >>>>> > >>>>> Flume has long had a policy of backwards compatibility with its own > >>>>> configuration files and people expect things to "just work" when > >>>> upgrading > >>>>> Flume. If log4j2 can't parse the log4j1 config file format then it's > an > >>>>> incompatible upgrade and should not be done in a minor Flume release. > >>>>> > >>>>> If log4j2 wants to be a drop-in replacement for log4j1 then by > default > >> it > >>>>> should find and parse the traditional log4j.properties config files, > at > >>>>> least as a fallback, rather than force users to convert to the new > XML > >>>>> format before upgrading. > >>>> > >>>> That is simply not possible: > >>>> 1. Log4j 1 requires the use of fully qualified class names. The > package > >>>> names log4j 1 used didn’t conform to ASF naming guidelines and don’t > >> line > >>>> up with the package names used in Log4j2. > >>>> 2. Log4j 1 did not delineate between what was public and what was > >> private > >>>> so there is code all over the place mucking with the internals of > Log4j. > >>>> Log4j 2 implemented a compatibility layer by implementing the classes > >> that > >>>> are being used but by and large they don’t actually do anything. > >>>> 3. The Appenders and Filters in Log4j 2 implement different interfaces > >> and > >>>> cannot use components from Log4j 1. > >>>> 4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact, > >> many > >>>> people didn’t use the Rolling File Appender that shipped with Log4j > but > >>>> used the one from Log4j extras. So it is hard to.know what Log4j 2 > would > >>>> have needed to be compatible with. > >>>> > >>>> Many organizations (including mine) have security requirements that > say > >>>> they must use software that is supported with security fixes. Log4j 1 > >> has a > >>>> known security bug that will never be fixed as it reached end-of-life > in > >>>> August of 2015. This means the use of Log4j 1 is not acceptable for > any > >>>> security conscious organization. > >>>> > >>>> While folks have brought up this issue from time to time most seem to > >>>> adapt quite quickly to the change. Most of the issues are developers > >>>> wanting to know how to make something they were doing in Log4j 1 work > in > >>>> Log4j 2. > >>>> > >>>> As you can imagine, since Log4j 2 has been GA now for 4 1/2 years and > >>>> Log4j 1 has been EOL for over 3, making the configuration compatible > >> with > >>>> Log4j 1 is not a high priority. FWIW, there was an effort to do that a > >> few > >>>> years ago, and while we were able to get some basic stuff to work > >> making it > >>>> work in a general way wasn’t possible. > >>>> > >>>> Ralph > >>>> > >>>>> > >>>>> Mike > >>>>> > >>>>> On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers < > >> ralph.go...@dslextreme.com> > >>>>> wrote: > >>>>> > >>>>>> No, you are correct. However, requiring a change to logging > >>>> configuration > >>>>>> has never been considered a binary compatibility break in any > project > >> I > >>>>>> have ever worked on. > >>>>>> > >>>>>> Ralph > >>>>>> > >>>>>>> On Nov 23, 2018, at 10:05 AM, Ferenc Szabo > >> <fsz...@cloudera.com.INVALID > >>>>> > >>>>>> wrote: > >>>>>>> > >>>>>>> As you mentioned, you have the freedom to use Log4j 2 and the same > >> time > >>>>>> we > >>>>>>> have to keep the out of the box experience the same in a minor > >> version. > >>>>>>> Users should be able to upgrade flume without changing any of their > >>>>>>> configurations. > >>>>>>> If they have a log4j.properties (Log4j 1) then they would not be > able > >>>> to > >>>>>>> use it after the upgrade without changing it. > >>>>>>> > >>>>>>> Or am I missing a feature that would solve this case? > >>>>>>> > >>>>>>> On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers < > >>>> ralph.go...@dslextreme.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Also, please put details in the Jira issues. It is much easier to > >> find > >>>>>> out > >>>>>>>> why something was done by searching Jira later on then searching > the > >>>>>>>> mailing list. > >>>>>>>> > >>>>>>>> Ralph > >>>>>>>> > >>>>>>>>> On Nov 23, 2018, at 9:47 AM, Ralph Goers < > >> ralph.go...@dslextreme.com > >>>>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> I do not understand this at all. Log4j 2 provides runtime > >>>> compatibility > >>>>>>>> with Log4j 1. What is the problem that requires a revert? > >>>>>>>>> > >>>>>>>>> I have been running Flume with Log4j 2 since 1.6 so I don’t > >>>> understand > >>>>>>>> what the problem could possibly be. > >>>>>>>>> > >>>>>>>>> Ralph > >>>>>>>>> > >>>>>>>>>> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo < > szabofe...@apache.org> > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi everyone > >>>>>>>>>> > >>>>>>>>>> I am about to branch the 1.9 release from trunk. > >>>>>>>>>> > >>>>>>>>>> On the 1.9 branch we will revert the following breaking changes: > >>>>>>>>>> - FLUME-2957. Remove Guava from our public API: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >> > https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088 > >>>>>>>>>> - part of FLUME-2050. Upgrade to Log4j 2.10.0: > >>>>>>>>>> as the new release should work with the previous configurations > we > >>>>>> have > >>>>>>>>>> to release it with log4j 1.x > >>>>>>>>>> For the log4j2 upgrade, we will provide a guide, how to replace > >> the > >>>>>> jars > >>>>>>>>>> if users would like to start using it in the 1.9 release on the > >> wiki > >>>>>>>> page. > >>>>>>>>>> > >>>>>>>>>> Because of these changes the first release candidate might be > >>>>>> postponed > >>>>>>>> to > >>>>>>>>>> Monday. > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Ferenc Szabo > >>>>>>>>>> > >>>>>>>>>> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com < > >>>>>> ema...@cloudera.com > >>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Ferenc, > >>>>>>>>>>> > >>>>>>>>>>> +1 > >>>>>>>>>>> I am working on FLUME-3281 Update to Kafka 2.0 client, should > be > >>>> able > >>>>>>>> to > >>>>>>>>>>> finish it > >>>>>>>>>>> till the suggested deadline. > >>>>>>>>>>> I am also happy to do some reviews. > >>>>>>>>>>> > >>>>>>>>>>> Regards > >>>>>>>>>>> Endre > >>>>>>>>>>> > >>>>>>>>>>> On 2018/11/06 21:23:17, Ferenc Szabo <szabofe...@apache.org> > >>>> wrote: > >>>>>>>>>>>> Hello Flume Community, > >>>>>>>>>>>> > >>>>>>>>>>>> 1.8 was released about a year ago and since that quite a few > bug > >>>>>>>> fixes, > >>>>>>>>>>>> improvements, features and documentation were introduced. > >>>>>>>>>>>> I would like to propose to publish the next minor release of > >> Flume > >>>>>>>>>>>> to make these changes available to the users. > >>>>>>>>>>>> > >>>>>>>>>>>> I would be more than happy to be the Release Manager with the > >> help > >>>>>> of > >>>>>>>>>>>> Denes Arvay for anything that requires PMC access - if both > the > >>>>>>>> community > >>>>>>>>>>>> and he are > >>>>>>>>>>>> OK with it. > >>>>>>>>>>>> > >>>>>>>>>>>> Among others the following changes will be included in the > next > >>>>>>>> release: > >>>>>>>>>>>> > >>>>>>>>>>>> Fixed bugs: > >>>>>>>>>>>> - FLUME-3117 Application can be dead loop when call > >> System.exit() > >>>> in > >>>>>>>>>>>> methodconfigure > >>>>>>>>>>>> - FLUME-3237 Handling RuntimeExceptions coming from the JMS > >>>> provider > >>>>>>>> in > >>>>>>>>>>>> JMSSource > >>>>>>>>>>>> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in > december > >>>>>>>>>>> correctly > >>>>>>>>>>>> - FLUME-3056 TestApplication hangs indefinitely > >>>>>>>>>>>> - FLUME-2976 Exception when JMS source tries to connect to a > >>>>>> Weblogic > >>>>>>>>>>>> server without authentication > >>>>>>>>>>>> - FLUME-3270 Close JMS resources in JMSMessageConsumer > >> constructor > >>>>>> in > >>>>>>>>>>> case > >>>>>>>>>>>> of failure > >>>>>>>>>>>> - FLUME-3222 java.nio.file.NoSuchFileException thrown when > files > >>>> are > >>>>>>>>>>> being > >>>>>>>>>>>> deleted from the TAILDIR source > >>>>>>>>>>>> - FLUME-2894 Flume components should stop in the correct order > >>>>>>>> (graceful > >>>>>>>>>>>> shutdown) > >>>>>>>>>>>> - FLUME-2973 Deadlock in hdfs sink > >>>>>>>>>>>> - FLUME-3278 Handling -D keystore parameters in Kafka > components > >>>>>>>>>>>> - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Improvements: > >>>>>>>>>>>> - FLUME-3186 Make asyncHbaseClient configuration parameters > >>>>>> available > >>>>>>>>>>> from > >>>>>>>>>>>> flume config > >>>>>>>>>>>> - FLUME-3239 Do not rename files in SpoolDirectorySource > >>>>>>>>>>>> - FLUME-3227 Add Rate Limiter to StressSource > >>>>>>>>>>>> - FLUME-2050 Upgrade to log4j2 (when GA) > >>>>>>>>>>>> - FLUME-3246 Validate flume configuration to prevent larger > >> source > >>>>>>>> batch > >>>>>>>>>>>> size than the channel transaction capacity > >>>>>>>>>>>> - FLUME-3050 add counters for error conditions and expose to > >>>> monitor > >>>>>>>> URL > >>>>>>>>>>>> - FLUME-3269 Support JSSE keystore/trustore -D system > properties > >>>>>>>>>>>> - FLUME-3223 Flume HDFS Sink should retry close prior to > >>>> performing > >>>>>> a > >>>>>>>>>>>> recoverLease > >>>>>>>>>>>> - FLUME-3276 Components supporting SSL/TLS should be able to > >>>> specify > >>>>>>>>>>> cipher > >>>>>>>>>>>> suite list > >>>>>>>>>>>> - FLUME-3275 Components supporting SSL/TLS should be able to > >>>> specify > >>>>>>>>>>>> protocol list > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> New Features: > >>>>>>>>>>>> - FLUME-3142 Adding HBase2 sink > >>>>>>>>>>>> - FLUME-2442 Need an alternative to providing clear text > >> passwords > >>>>>> in > >>>>>>>>>>> flume > >>>>>>>>>>>> config > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> There are 26 open tickets targeted for 1.9 in patch available > >>>> state: > >>>>>>>>>>>> https://s.apache.org/flume-1.9-target-tickets > >>>>>>>>>>>> > >>>>>>>>>>>> We also have a number of open pull requests on Github: > >>>>>>>>>>>> https://github.com/apache/flume/pulls > >>>>>>>>>>>> > >>>>>>>>>>>> There are a few tickets in progress that would be good to > >> include > >>>> in > >>>>>>>> the > >>>>>>>>>>>> release so > >>>>>>>>>>>> I would say that we focus on reviewing and testing them in the > >>>> next > >>>>>> 2 > >>>>>>>>>>>> weeks. > >>>>>>>>>>>> > >>>>>>>>>>>> I would like to propose to target the 23rd of November for the > >>>> first > >>>>>>>>>>>> release candidate. > >>>>>>>>>>>> That would mean that the branch date would be a few days > before > >>>>>> that. > >>>>>>>>>>>> Any significant code change should get in by that date. > >>>>>>>>>>>> > >>>>>>>>>>>> If nobody has any concerns then I am going to create an > umbrella > >>>>>>>> ticket > >>>>>>>>>>> to > >>>>>>>>>>>> track the release process. > >>>>>>>>>>>> > >>>>>>>>>>>> Some movement can be expected in the related JIRA tickets. > >>>>>>>>>>>> > >>>>>>>>>>>> Kind regards, > >>>>>>>>>>>> Ferenc > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>> > >>>> > >>>> > >> > >> > >> > > >
