> On Nov 24, 2018, at 2:35 PM, Mike Percy <mpe...@apache.org> wrote: > > Flume has long had a policy of backwards compatibility with its own > configuration files and people expect things to "just work" when upgrading > Flume. If log4j2 can't parse the log4j1 config file format then it's an > incompatible upgrade and should not be done in a minor Flume release. > > If log4j2 wants to be a drop-in replacement for log4j1 then by default it > should find and parse the traditional log4j.properties config files, at > least as a fallback, rather than force users to convert to the new XML > format before upgrading.
That is simply not possible: 1. Log4j 1 requires the use of fully qualified class names. The package names log4j 1 used didn’t conform to ASF naming guidelines and don’t line up with the package names used in Log4j2. 2. Log4j 1 did not delineate between what was public and what was private so there is code all over the place mucking with the internals of Log4j. Log4j 2 implemented a compatibility layer by implementing the classes that are being used but by and large they don’t actually do anything. 3. The Appenders and Filters in Log4j 2 implement different interfaces and cannot use components from Log4j 1. 4. The Appenders in Log4j 2 are not identical with Log4j 1. In fact, many people didn’t use the Rolling File Appender that shipped with Log4j but used the one from Log4j extras. So it is hard to.know what Log4j 2 would have needed to be compatible with. Many organizations (including mine) have security requirements that say they must use software that is supported with security fixes. Log4j 1 has a known security bug that will never be fixed as it reached end-of-life in August of 2015. This means the use of Log4j 1 is not acceptable for any security conscious organization. While folks have brought up this issue from time to time most seem to adapt quite quickly to the change. Most of the issues are developers wanting to know how to make something they were doing in Log4j 1 work in Log4j 2. As you can imagine, since Log4j 2 has been GA now for 4 1/2 years and Log4j 1 has been EOL for over 3, making the configuration compatible with Log4j 1 is not a high priority. FWIW, there was an effort to do that a few years ago, and while we were able to get some basic stuff to work making it work in a general way wasn’t possible. Ralph > > Mike > > On Fri, Nov 23, 2018 at 9:39 AM Ralph Goers <ralph.go...@dslextreme.com> > wrote: > >> No, you are correct. However, requiring a change to logging configuration >> has never been considered a binary compatibility break in any project I >> have ever worked on. >> >> Ralph >> >>> On Nov 23, 2018, at 10:05 AM, Ferenc Szabo <fsz...@cloudera.com.INVALID> >> wrote: >>> >>> As you mentioned, you have the freedom to use Log4j 2 and the same time >> we >>> have to keep the out of the box experience the same in a minor version. >>> Users should be able to upgrade flume without changing any of their >>> configurations. >>> If they have a log4j.properties (Log4j 1) then they would not be able to >>> use it after the upgrade without changing it. >>> >>> Or am I missing a feature that would solve this case? >>> >>> On Fri, Nov 23, 2018 at 5:49 PM Ralph Goers <ralph.go...@dslextreme.com> >>> wrote: >>> >>>> Also, please put details in the Jira issues. It is much easier to find >> out >>>> why something was done by searching Jira later on then searching the >>>> mailing list. >>>> >>>> Ralph >>>> >>>>> On Nov 23, 2018, at 9:47 AM, Ralph Goers <ralph.go...@dslextreme.com> >>>> wrote: >>>>> >>>>> I do not understand this at all. Log4j 2 provides runtime compatibility >>>> with Log4j 1. What is the problem that requires a revert? >>>>> >>>>> I have been running Flume with Log4j 2 since 1.6 so I don’t understand >>>> what the problem could possibly be. >>>>> >>>>> Ralph >>>>> >>>>>> On Nov 23, 2018, at 8:50 AM, Ferenc Szabo <szabofe...@apache.org> >>>> wrote: >>>>>> >>>>>> Hi everyone >>>>>> >>>>>> I am about to branch the 1.9 release from trunk. >>>>>> >>>>>> On the 1.9 branch we will revert the following breaking changes: >>>>>> - FLUME-2957. Remove Guava from our public API: >>>>>> >>>>>> >>>> >> https://github.com/apache/flume/commit/7f85df9e473ee675d461d5b76650694c5a6c0088 >>>>>> - part of FLUME-2050. Upgrade to Log4j 2.10.0: >>>>>> as the new release should work with the previous configurations we >> have >>>>>> to release it with log4j 1.x >>>>>> For the log4j2 upgrade, we will provide a guide, how to replace the >> jars >>>>>> if users would like to start using it in the 1.9 release on the wiki >>>> page. >>>>>> >>>>>> Because of these changes the first release candidate might be >> postponed >>>> to >>>>>> Monday. >>>>>> >>>>>> Regards, >>>>>> Ferenc Szabo >>>>>> >>>>>> On Wed, Nov 7, 2018 at 5:04 PM ema...@cloudera.com < >> ema...@cloudera.com >>>>> >>>>>> wrote: >>>>>> >>>>>>> Hi Ferenc, >>>>>>> >>>>>>> +1 >>>>>>> I am working on FLUME-3281 Update to Kafka 2.0 client, should be able >>>> to >>>>>>> finish it >>>>>>> till the suggested deadline. >>>>>>> I am also happy to do some reviews. >>>>>>> >>>>>>> Regards >>>>>>> Endre >>>>>>> >>>>>>> On 2018/11/06 21:23:17, Ferenc Szabo <szabofe...@apache.org> wrote: >>>>>>>> Hello Flume Community, >>>>>>>> >>>>>>>> 1.8 was released about a year ago and since that quite a few bug >>>> fixes, >>>>>>>> improvements, features and documentation were introduced. >>>>>>>> I would like to propose to publish the next minor release of Flume >>>>>>>> to make these changes available to the users. >>>>>>>> >>>>>>>> I would be more than happy to be the Release Manager with the help >> of >>>>>>>> Denes Arvay for anything that requires PMC access - if both the >>>> community >>>>>>>> and he are >>>>>>>> OK with it. >>>>>>>> >>>>>>>> Among others the following changes will be included in the next >>>> release: >>>>>>>> >>>>>>>> Fixed bugs: >>>>>>>> - FLUME-3117 Application can be dead loop when call System.exit() in >>>>>>>> methodconfigure >>>>>>>> - FLUME-3237 Handling RuntimeExceptions coming from the JMS provider >>>> in >>>>>>>> JMSSource >>>>>>>> - FLUME-3201 Fix SyslogUtil to handle RFC3164 format in december >>>>>>> correctly >>>>>>>> - FLUME-3056 TestApplication hangs indefinitely >>>>>>>> - FLUME-2976 Exception when JMS source tries to connect to a >> Weblogic >>>>>>>> server without authentication >>>>>>>> - FLUME-3270 Close JMS resources in JMSMessageConsumer constructor >> in >>>>>>> case >>>>>>>> of failure >>>>>>>> - FLUME-3222 java.nio.file.NoSuchFileException thrown when files are >>>>>>> being >>>>>>>> deleted from the TAILDIR source >>>>>>>> - FLUME-2894 Flume components should stop in the correct order >>>> (graceful >>>>>>>> shutdown) >>>>>>>> - FLUME-2973 Deadlock in hdfs sink >>>>>>>> - FLUME-3278 Handling -D keystore parameters in Kafka components >>>>>>>> - FLUME-3265 Cannot set batch-size for LoadBalancingRpcClient >>>>>>>> >>>>>>>> >>>>>>>> Improvements: >>>>>>>> - FLUME-3186 Make asyncHbaseClient configuration parameters >> available >>>>>>> from >>>>>>>> flume config >>>>>>>> - FLUME-3239 Do not rename files in SpoolDirectorySource >>>>>>>> - FLUME-3227 Add Rate Limiter to StressSource >>>>>>>> - FLUME-2050 Upgrade to log4j2 (when GA) >>>>>>>> - FLUME-3246 Validate flume configuration to prevent larger source >>>> batch >>>>>>>> size than the channel transaction capacity >>>>>>>> - FLUME-3050 add counters for error conditions and expose to monitor >>>> URL >>>>>>>> - FLUME-3269 Support JSSE keystore/trustore -D system properties >>>>>>>> - FLUME-3223 Flume HDFS Sink should retry close prior to performing >> a >>>>>>>> recoverLease >>>>>>>> - FLUME-3276 Components supporting SSL/TLS should be able to specify >>>>>>> cipher >>>>>>>> suite list >>>>>>>> - FLUME-3275 Components supporting SSL/TLS should be able to specify >>>>>>>> protocol list >>>>>>>> >>>>>>>> >>>>>>>> New Features: >>>>>>>> - FLUME-3142 Adding HBase2 sink >>>>>>>> - FLUME-2442 Need an alternative to providing clear text passwords >> in >>>>>>> flume >>>>>>>> config >>>>>>>> >>>>>>>> >>>>>>>> There are 26 open tickets targeted for 1.9 in patch available state: >>>>>>>> https://s.apache.org/flume-1.9-target-tickets >>>>>>>> >>>>>>>> We also have a number of open pull requests on Github: >>>>>>>> https://github.com/apache/flume/pulls >>>>>>>> >>>>>>>> There are a few tickets in progress that would be good to include in >>>> the >>>>>>>> release so >>>>>>>> I would say that we focus on reviewing and testing them in the next >> 2 >>>>>>>> weeks. >>>>>>>> >>>>>>>> I would like to propose to target the 23rd of November for the first >>>>>>>> release candidate. >>>>>>>> That would mean that the branch date would be a few days before >> that. >>>>>>>> Any significant code change should get in by that date. >>>>>>>> >>>>>>>> If nobody has any concerns then I am going to create an umbrella >>>> ticket >>>>>>> to >>>>>>>> track the release process. >>>>>>>> >>>>>>>> Some movement can be expected in the related JIRA tickets. >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> Ferenc >>>>>>>> >>>>>>> >>>>> >>>> >>>> >>>> >> >> >>
