Re: Migrate to Log4j 2

[Lior Zeno]

Sure, point taken. The only issue is having both versions in the same
binary. But it's not the case here.
I am not sure of it is a good practice to continue supporting deprecated
software, but that's another discussion I guess.

On Oct 22, 2016 01:24, "Ralph Goers"  wrote:

I should point out that Log4j 2 has shipped a Flume Appender since before
it went GA. The Flume Log4j Appender is for users who use Log4j 1.x to
connect to Flume. There really is no reason to drop that even if Flume
itself starts using Log4j 2. Since Flume is a standalone app there really
isn’t anything that requires a break in compatibility.

Ralph

> On Oct 21, 2016, at 11:24 AM, Attila Simon  wrote:
>
> Hi Lior,
>
> Thanks a lot! Based on these it is indeed flume 2.0 and I have no
objection
> having it than (don't know when it will be but hope we can speed up the
> release cycles).
>
> Cheers,
> Attila
>
>
> *Attila Simon*
> Software Engineer
> Email:   s...@cloudera.com
>
> [image: Cloudera Inc.]
>
> On Thu, Oct 20, 2016 at 12:11 PM, Lior Zeno  wrote:
>
>> Sure.
>>
>> First, the Log4jAppender will be deprecated. Log4j2 already provides a
>> Flume appender [1].
>> Second, since we use almost exclusively the SLF4j API in Flume, the code
>> will only slightly change. The major difference is with the configuration
>> files which have changed from 1.x to 2.x [2].
>>
>> [1]
>> https://logging.apache.org/log4j/2.x/manual/appenders.html#FlumeAppender
>> [2] https://logging.apache.org/log4j/2.x/manual/migration.html.
>>
>> On Thu, Oct 20, 2016 at 11:44 AM, Attila Simon  wrote:
>>
>>> Hi Lior,
>>>
>>> Could you please explain a bit what will break?
>>>
>>> Cheers,
>>> Attila
>>>
>>> On Tuesday, 18 October 2016, Lior Zeno  wrote:
>>>
 Hi All,

 Log4j (v1) has been EOL'ed over a year now (
 https://blogs.apache.org/foundation/entry/apache_
>>> logging_services_project_
 announces)
 and is no longer officially supported.

 I propose we migrate to Log4j 2. We can begin with using the Log4j 1.x
 bridge, and then incrementally move the whole codebase.

 Since this is a breaking change, I think we should schedule this to
>> Flume
 2.0.0.

 Thoughts?

 Thanks

>>>
>>>
>>> --
>>>
>>> *Attila Simon*
>>> Software Engineer
>>> Email:   s...@cloudera.com
>>>
>>> [image: Cloudera Inc.]
>>>
>>

Re: Migrate to Log4j 2

[Ralph Goers]

I should point out that Log4j 2 has shipped a Flume Appender since before it 
went GA. The Flume Log4j Appender is for users who use Log4j 1.x to connect to 
Flume. There really is no reason to drop that even if Flume itself starts using 
Log4j 2. Since Flume is a standalone app there really isn’t anything that 
requires a break in compatibility.

Ralph

> On Oct 21, 2016, at 11:24 AM, Attila Simon  wrote:
> 
> Hi Lior,
> 
> Thanks a lot! Based on these it is indeed flume 2.0 and I have no objection
> having it than (don't know when it will be but hope we can speed up the
> release cycles).
> 
> Cheers,
> Attila
> 
> 
> *Attila Simon*
> Software Engineer
> Email:   s...@cloudera.com
> 
> [image: Cloudera Inc.]
> 
> On Thu, Oct 20, 2016 at 12:11 PM, Lior Zeno  wrote:
> 
>> Sure.
>> 
>> First, the Log4jAppender will be deprecated. Log4j2 already provides a
>> Flume appender [1].
>> Second, since we use almost exclusively the SLF4j API in Flume, the code
>> will only slightly change. The major difference is with the configuration
>> files which have changed from 1.x to 2.x [2].
>> 
>> [1]
>> https://logging.apache.org/log4j/2.x/manual/appenders.html#FlumeAppender
>> [2] https://logging.apache.org/log4j/2.x/manual/migration.html.
>> 
>> On Thu, Oct 20, 2016 at 11:44 AM, Attila Simon  wrote:
>> 
>>> Hi Lior,
>>> 
>>> Could you please explain a bit what will break?
>>> 
>>> Cheers,
>>> Attila
>>> 
>>> On Tuesday, 18 October 2016, Lior Zeno  wrote:
>>> 
 Hi All,
 
 Log4j (v1) has been EOL'ed over a year now (
 https://blogs.apache.org/foundation/entry/apache_
>>> logging_services_project_
 announces)
 and is no longer officially supported.
 
 I propose we migrate to Log4j 2. We can begin with using the Log4j 1.x
 bridge, and then incrementally move the whole codebase.
 
 Since this is a breaking change, I think we should schedule this to
>> Flume
 2.0.0.
 
 Thoughts?
 
 Thanks
 
>>> 
>>> 
>>> --
>>> 
>>> *Attila Simon*
>>> Software Engineer
>>> Email:   s...@cloudera.com
>>> 
>>> [image: Cloudera Inc.]
>>> 
>> 

Re: jira reference is missing from git commit messages

[Attila Simon]

Hi Mike,

I'm a big fan of having the site in scm so I guess I can volunteer
after Donat finishes that movement.
I have no strong opinion on whether we should have a jira or not (thus
the proposal of FLUME-PR70). I just "I found this habit very useful".
I have tooling which depends on the commit titles are unique with high
probability. Most likely these tools can be upgraded with some extra
effort.
Maybe having jira for each change is an overkill for small changes.
What do you think what can be considered as a small change? eg the
ones for which the "how to contribute" guide doesn't require review
board?

Cheers,
Attila

On Fri, Oct 21, 2016 at 12:54 PM, Mike Percy  wrote:
>
> Hi Attila,
> Thanks for raising this concern of yours. Please see inline.
>
> On Fri, Oct 21, 2016 at 12:17 PM, Attila Simon  wrote:
>
> > The how to contribute page asks to have a jira first for each change. I
> > guess with allowing pull requests we have to update the how to contribute
> > page as well (which only describes attaching patch and using review board).
> >
>
> Yes, it appeared that we had consensus for allowing PR on a couple separate
> dev@ threads a while back [1], [2].
>
> The Flume contributor docs are currently a bit stale. I have recently
> updated the How to Release cwiki page but if you want to volunteer to
> update some of the other docs that would be very welcome! Please let me
> know if you want to help and I can give you cwiki edit access if you send
> me your account id. Side note: Donat has recently proposed moving those
> docs from cwiki to the Git repo which I think would be a big improvement.
>
> I would like to ask committers to reestablish the habit of having a jira
> > for each commit and start the message with that jira.
> >
>
> Forcing people to file a JIRA when they are already doing a PR feels like
> pointless extra paperwork to me. There is certainly a place for JIRA in a
> software project, but I think that is to track unfixed bugs, ongoing tasks
> / projects, etc.
>
>
> > If we would like to relax the have jira for each change (for pull requests)
> > then I would suggest putting the request id as the first thing in the
> > commit message.
> >
>
> Why?
>
> If you click on this:
> https://github.com/apache/flume/commit/87d4c2c13862144eb578b211bcf800b2206834ff
>
> You will see that the text "Closes #70" creates a clickable hyperlink to
> the PR. Is this not sufficient for tracking purposes?
>
> Mike
>
> [1] https://s.apache.org/k31f
> [2] https://s.apache.org/Skm2

Re: Migrate to Log4j 2

[Ralph Goers]

FYI, I have been using Log4j 2 in my Flume deployments for over a year, despite 
whatever Flume or its dependencies are using for an API.  Having Flume come 
preconfigured for Log4j 2 would be nice.  It will take some work to convert the 
code to use the Log4j 2 API. At a minimum all the log4j imports have to be 
modified. You only have to modify the actual logging statements if you want to 
take advantage of the API improvements.

Ralph

> On Oct 20, 2016, at 1:44 AM, Attila Simon  wrote:
> 
> Hi Lior,
> 
> Could you please explain a bit what will break?
> 
> Cheers,
> Attila
> 
> On Tuesday, 18 October 2016, Lior Zeno  wrote:
> 
>> Hi All,
>> 
>> Log4j (v1) has been EOL'ed over a year now (
>> https://blogs.apache.org/foundation/entry/apache_logging_services_project_
>> announces)
>> and is no longer officially supported.
>> 
>> I propose we migrate to Log4j 2. We can begin with using the Log4j 1.x
>> bridge, and then incrementally move the whole codebase.
>> 
>> Since this is a breaking change, I think we should schedule this to Flume
>> 2.0.0.
>> 
>> Thoughts?
>> 
>> Thanks
>> 
> 
> 
> -- 
> 
> *Attila Simon*
> Software Engineer
> Email:   s...@cloudera.com
> 
> [image: Cloudera Inc.]

[jira] [Comment Edited] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595616#comment-15595616
 ] 

Attila Simon edited comment on FLUME-3015 at 10/21/16 4:54 PM:
---

TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be 
flaky. If you found the root cause for this ticket please double check whether 
it also applicable to 
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks. Stack traces and 
exception is the similar to NoFK.


was (Author: sati):
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be 
flaky. If you found the root cause for this ticket please double check whether 
it also applicable to 
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks

> Fix flaky 
> org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
> 
>
> Key: FLUME-3015
> URL: https://issues.apache.org/jira/browse/FLUME-3015
> Project: Flume
>  Issue Type: Bug
>  Components: Test
>Affects Versions: v1.7.0
>Reporter: Attila Simon
>
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
>  org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> {noformat}
>  
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
>   Time elapsed: 310978 sec  <<< ERROR!
>  java.util.concurrent.ExecutionException: 
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
>  Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Failed to 
> persist event
>   at 
> 

[jira] [Created] (FLUME-3017) Fix flaky org.apache.flume.sink.hbase.TestHBaseSink

[Attila Simon (JIRA)]

Attila Simon created FLUME-3017:
---

 Summary: Fix flaky org.apache.flume.sink.hbase.TestHBaseSink
 Key: FLUME-3017
 URL: https://issues.apache.org/jira/browse/FLUME-3017
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


{noformat}
ERROR [main] hbase.MiniHBaseCluster (MiniHBaseCluster.java:init(230)) - Error 
starting cluster
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/procedure2/Procedure
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:219)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:214)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:94)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1036)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:996)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
at 
org.apache.flume.sink.hbase.TestHBaseSink.setUpOnce(TestHBaseSink.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.hbase.procedure2.Procedure
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 37 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3016) Fix flaky org.apache.flume.sink.hbase.TestAsyncHBaseSink

[Attila Simon (JIRA)]

Attila Simon created FLUME-3016:
---

 Summary: Fix flaky org.apache.flume.sink.hbase.TestAsyncHBaseSink 
 Key: FLUME-3016
 URL: https://issues.apache.org/jira/browse/FLUME-3016
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


{noformat}
2016-09-26 05:11:33,321 ERROR [main] hbase.MiniHBaseCluster 
(MiniHBaseCluster.java:init(230)) - Error starting cluster
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/procedure2/Procedure
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getConstructor(Class.java:1825)
at 
org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:219)
at 
org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:155)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:214)
at 
org.apache.hadoop.hbase.MiniHBaseCluster.(MiniHBaseCluster.java:94)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1036)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:996)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862)
at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
at 
org.apache.flume.sink.hbase.TestAsyncHBaseSink.setUp(TestAsyncHBaseSink.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.hbase.procedure2.Procedure
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15595616#comment-15595616
 ] 

Attila Simon commented on FLUME-3015:
-

TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks might also be 
flaky. If you found the root cause for this ticket please double check whether 
it also applicable to 
TestJdbcChannelProvider.testEventWithSimulatedSourceAndSinks

> Fix flaky 
> org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
> 
>
> Key: FLUME-3015
> URL: https://issues.apache.org/jira/browse/FLUME-3015
> Project: Flume
>  Issue Type: Bug
>  Components: Test
>Affects Versions: v1.7.0
>Reporter: Attila Simon
>
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
>  org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
> {noformat}
>  
> testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
>   Time elapsed: 310978 sec  <<< ERROR!
>  java.util.concurrent.ExecutionException: 
> org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:188)
>   at 
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
>   at 
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
>  Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Failed to 
> persist event
>   at 
> org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:312)
>   at 
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:368)
>   at 
> org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:345)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> 

[jira] [Created] (FLUME-3015) Fix flaky org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks

[Attila Simon (JIRA)]

Attila Simon created FLUME-3015:
---

 Summary: Fix flaky 
org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK.testEventWithSimulatedSourceAndSinks
 Key: FLUME-3015
 URL: https://issues.apache.org/jira/browse/FLUME-3015
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK):
 org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event

{noformat}
 
testEventWithSimulatedSourceAndSinks(org.apache.flume.channel.jdbc.TestJdbcChannelProviderNoFK)
  Time elapsed: 310978 sec  <<< ERROR!
 java.util.concurrent.ExecutionException: 
org.apache.flume.channel.jdbc.JdbcChannelException: Failed to persist event
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at 
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest.testEventWithSimulatedSourceAndSinks(BaseJdbcChannelProviderTest.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
 Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Failed to 
persist event
at 
org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:312)
at 
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:368)
at 
org.apache.flume.channel.jdbc.BaseJdbcChannelProviderTest$MockSource.call(BaseJdbcChannelProviderTest.java:345)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
 Caused by: org.apache.flume.channel.jdbc.JdbcChannelException: Unable to 
persist event: org.apache.flume.channel.jdbc.impl.PersistableEvent@4fbafc05
at 
org.apache.flume.channel.jdbc.impl.DerbySchemaHandler.storeEvent(DerbySchemaHandler.java:733)
at 
org.apache.flume.channel.jdbc.impl.JdbcChannelProviderImpl.persistEvent(JdbcChannelProviderImpl.java:305)
 

[jira] [Created] (FLUME-3014) fix flaky test org.apache.flume.source.TestSyslogUdpSource.testSourceCounter

[Attila Simon (JIRA)]

Attila Simon created FLUME-3014:
---

 Summary: fix flaky test 
org.apache.flume.source.TestSyslogUdpSource.testSourceCounter
 Key: FLUME-3014
 URL: https://issues.apache.org/jira/browse/FLUME-3014
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


{noformat}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.flume.source.TestSyslogUdpSource.testSourceCounter(TestSyslogUdpSource.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3013) Fix flaky testShutdown(org.apache.flume.source.TestExecSource)

[Attila Simon (JIRA)]

Attila Simon created FLUME-3013:
---

 Summary: Fix flaky 
testShutdown(org.apache.flume.source.TestExecSource)
 Key: FLUME-3013
 URL: https://issues.apache.org/jira/browse/FLUME-3013
 Project: Flume
  Issue Type: Bug
  Components: Test
Affects Versions: v1.7.0
Reporter: Attila Simon


{noformat}
java.lang.IllegalStateException: Command [ps -ef] exited with 143
at org.apache.flume.source.TestExecSource.exec(TestExecSource.java:460)
at
org.apache.flume.source.TestExecSource.testShutdown(TestExecSource.java:406)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

testShutdown(org.apache.flume.source.TestExecSource)  Time elapsed: 621 sec
 <<< ERROR!
java.lang.NullPointerException
at org.apache.flume.source.ExecSource.stop(ExecSource.java:202)
at org.apache.flume.source.TestExecSource.tearDown(TestExecSource.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at

Re: Enabling Travis-CI on Flume

[Mike Percy]

Hi Lior!

No, my message was not directed at you, or any person in particular. I
intended this message for those paying attention to this topic to try and
set expectations for how decision making for things like this usually (and
hopefully) works in Apache: If you are willing to do the work to get
something done, then it will probably get done the way you want!

(Assuming the end result is something that others in the community want --
in this case, it's very basic pre-commit checks. I think we have consensus
that pre-commit checks are something that would be a net benefit to
everyone.)

Sorry for any confusion! And don't let me stop people from voicing their
views and concerns.

Best,
Mike

On Fri, Oct 21, 2016 at 4:44 PM, Lior Zeno  wrote:

> Mike, I was not holding Donat back. I was just suggesting ways to configure
> Jenins, per Donat's request. I'm sorry if my former post delivered the
> wrong message.
>
> On Fri, Oct 21, 2016 at 6:29 PM, Mike Percy  wrote:
>
> > Personally I prefer Jenkins over TravisCI for various reasons however if
> > Donat is willing to do the work of adding pre-commit checks on PRs via
> > Travis then I say let him do it, in the Apache spirit of "let they that
> do
> > the work make the decisions".
> >
> > If someone actually spends the time to set up Jenkins and configure it to
> > do the same thing, then great, let's switch when it's ready.
> >
> > Note that only ASF committers have access to Jenkins so non-committers
> will
> > need to work with a committer to get it done if they want to help.
> >
> > Mike
> >
> > On Fri, Oct 21, 2016 at 3:46 PM, Lior Zeno  wrote:
> >
> > > There are many ways to do it, for example:
> > > https://www.theguild.nl/building-github-pull-requests-using-
> > > jenkins-pipelines/
> > > or https://www.theguild.nl/building-github-pull-requests-with-jenkins/
> > for
> > > earlier versions of Jenkins.
> > > I do not really care if it would be Jenkins or Travis, but I do think
> > that
> > > we can get Jenkins configured faster since we already have it. I can
> help
> > > with the configuration.
> > >
> > > On Fri, Oct 21, 2016 at 5:17 PM, Balazs Donat Bessenyei <
> > > bes...@cloudera.com
> > > > wrote:
> > >
> > > > As I haven't received any objections to enabling Travis, I'm going to
> > > > ask INFRA to enable it for Flume soon.
> > > >
> > > > This change would help submitting and reviewing pull requests.
> > > >
> > > > If someone figures out how we could use Jenkins for this purpose, we
> > > > can always disable Travis.
> > > >
> > > > PS. there are more projects using Travis:
> > > > https://issues.apache.org/jira/browse/INFRA-12757?jql=
> > > > project%20%3D%20INFRA%20AND%20text%20~%20travis%20ORDER%
> > > > 20BY%20updated%20DESC
> > > >
> > > > On Fri, Oct 14, 2016 at 5:41 PM, Attila Simon 
> > wrote:
> > > > > Denes I'm happy to help you in this endeavor of setting up jenkins
> > job
> > > > for
> > > > > verifying pull requests.
> > > > >
> > > > >
> > > > > *Attila Simon*
> > > > > Software Engineer
> > > > > Email:   s...@cloudera.com
> > > > >
> > > > > [image: Cloudera Inc.]
> > > > >
> > > > > On Fri, Oct 14, 2016 at 2:47 PM, Denes Arvay 
> > > wrote:
> > > > >
> > > > >> I'd also vote for Jenkins with github PRs.
> > > > >> I just checked Mesos and the PRs are checked by Travis, or at
> least
> > > they
> > > > >> experienced with it, there's a short discussion regarding to
> Travis
> > at
> > > > >> https://github.com/apache/mesos/pull/165
> > > > >>
> > > > >> As for the jenkins pull request job I'd be happy to set it up or
> > help
> > > > >> setting it up.
> > > > >>
> > > > >> Denes
> > > > >>
> > > > >> On Fri, Oct 14, 2016 at 2:15 PM Lior Zeno 
> > wrote:
> > > > >>
> > > > >> Are we switching to PRs from patches + RB? In Apache Mesos, they
> > have
> > > a
> > > > >>
> > > > >> review bot that can leave a comment on the patch, we could try and
> > > port
> > > > it
> > > > >>
> > > > >> to Flume. I think they use Jenkins too.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Oct 14, 2016 at 3:11 PM, Balazs Donat Bessenyei <
> > > > >> bes...@cloudera.com
> > > > >>
> > > > >> > wrote:
> > > > >>
> > > > >>
> > > > >>
> > > > >> > If the same function can be achieved with Jenkins and it's easy
> > > > >>
> > > > >> > (+quick) to set up, I'm totally happy with that.
> > > > >>
> > > > >> >
> > > > >>
> > > > >> > What do we have to do to enable Jenkins builds on PR-s?
> > > > >>
> > > > >> >
> > > > >>
> > > > >> > On Fri, Oct 14, 2016 at 2:05 PM, Lior Zeno 
> > > > wrote:
> > > > >>
> > > > >> > > There are ways to do the same with Jenkins, for instance, see
> > this
> > > > SO
> > > > >>
> > > > >> > > thread
> > > > >>
> > > > >> > > http://stackoverflow.com/questions/37661602/how-to-set-
> > > > >>
> > > > >> > up-a-github-pull-request-build-in-a-jenkinsfile
> > > > >>
> > > > >> > >
> > 

Re: Moving Developer Section of cwiki to the git repository

[Mike Percy]

Thanks, Donat. +1 on moving the docs.

Regarding the web site source code, it can reside in svn or git however it
should continue using the standard ASF site infrastructure [1] for hosting
content which consists only of static rendered HTML content. Our options
for pushing that rendered HTML are either svnpubsub[2] or gitpubsub[3]
which are basically the same thing - commit HTML and it gets pushed live to
the web site.

If someone wants to redesign the site or improve the site template, I would
certainly welcome that.

As a comparison point with another ASF project I am involved in, on Apache
Kudu we use the software used for GitHub pages (Jekyll) to render the site.
The web site source code is in the gh-pages branch in Git [4]. The site is
mostly written in Markdown and we wrote a script [5] that locally renders
the site to static HTML content and checks in the rendered content to the
Kudu gitpubsub repository [6].

In Flume, we basically do the same thing but but instead of Jekyll we use
Sphinx and the site source code is primarily ReStructuredText. The site
lives in its own svn repo [7]. We push rendered HTML tent to an svnpubsub
repository for Flume [8] which is controlled by the ASF CMS [9]... IIRC,
the CMS takes care of invoking the rendering part instead of having a local
script to do it. This is kind of documented in the How to Release guide for
Flume [10].

If someone just wanted to change the look & feel of the web site they could
easily do it with a little HTML/CSS knowledge and some reading up on
Sphinx. The web site template could be changed to something nicer (it's
using some kind of stock template right now). The existing .rst content
would not need to be modified at all, probably.

If someone wanted to switch the whole web site to some other system, like
Jekyll, obviously it's a bigger change but if they were determined then I
don't think anyone would try to stop them, assuming it's an improvement!

Hope this helps,
Mike

[1] https://www.apache.org/dev/project-site.html
[2] http://svn.apache.org/viewvc/subversion/trunk/tools/
server-side/svnpubsub/
[3] https://www.apache.org/dev/gitpubsub.html
[4] https://github.com/apache/kudu/tree/gh-pages
[5] https://github.com/apache/kudu/blob/gh-pages/site_tool
[6] https://git-wip-us.apache.org/repos/asf?p=kudu-site.git;a=
shortlog;h=refs/heads/asf-site
[7] https://svn.apache.org/repos/asf/flume/site/trunk/
[8] https://svn.apache.org/repos/infra/websites/production/flume/
[9] https://www.apache.org/dev/cms.html
[10]
https://cwiki.apache.org/confluence/display/FLUME/How+to+Release#HowtoRelease-Updatethewebsite

On Thu, Oct 20, 2016 at 2:59 PM, Balazs Donat Bessenyei  wrote:

> As nobody has objected in ~48 hours, I'll open a pull request about this
> soon.
>
> Regarding the whole site thing: AFAIK, the site contents have to
> reside in SVN, because that's how ASF infrastructure works now.
>
> First, I'll work on moving the Developer Section to
> https://github.com/apache/flume , then we'll see what else we can do.
>
>
> Thank you,
>
> Donat
>
> On Thu, Oct 20, 2016 at 12:41 PM, UMESH CHAUDHARY 
> wrote:
> > +1 for moving wiki and web contents into GitHub. It would be easier to
> > manage. Also, +1 for improving our website.
> >
> > On Thu, 20 Oct 2016 at 15:42 Lior Zeno  wrote:
> >
> > +1 for moving the whole site as well. I wish we could improve our
> frontend
> > to become a bit more appealing.
> >
> > On Thu, Oct 20, 2016 at 11:53 AM, Attila Simon 
> wrote:
> >
> >> +1
> >> Essentially moving whole site (all web content) to scm would help
> >> contribution.
> >>
> >> Cheers,
> >> Attila
> >>
> >> On Tuesday, 18 October 2016, Balazs Donat Bessenyei <
> bes...@cloudera.com>
> >> wrote:
> >>
> >> > Hi All,
> >> >
> >> > As it's kind of difficult to get permissions in the wiki to edit pages
> >> > like https://cwiki.apache.org/confluence/display/FLUME/How+to+
> Contribute
> >> > , I suggest moving contents to the git repository into files like
> >> > CONTRIBUTING.md, etc.
> >> >
> >> > I'd be happy to create the files and move the current texts.
> >> >
> >> > Please, let me know your thoughts.
> >> >
> >> >
> >> > Thank you,
> >> >
> >> > Donat
> >> >
> >>
> >>
> >> --
> >>
> >> *Attila Simon*
> >> Software Engineer
> >> Email:   s...@cloudera.com
> >>
> >> [image: Cloudera Inc.]
> >>
>

Re: Review Request 53056: FLUME-2945: Bump java target version to 1.8

[Lior Zeno]

> On Oct. 21, 2016, 10:57 a.m., Attila Simon wrote:
> > Hi Lior,
> > conf/flume-env.sh.template#22 is still referenceing to java-6-sun. Could 
> > you please include that as well.

Thanks. BTW, do you think we should continue recommending Oracle's JVM? The 
default-jdk/jre in variuos distributions is OpenJDK (e.g. Ubuntu).


- Lior


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53056/#review153527
---


On Oct. 21, 2016, 3:56 p.m., Lior Zeno wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53056/
> ---
> 
> (Updated Oct. 21, 2016, 3:56 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> We should move to Java 8 as a minimum requirement.
> 
> 1. Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html.
> 2. Many projects are adopting java 8 as a minimum requirement, for instance:
> 
> * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722
> * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624
> * elasticsearch 5: 
> https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html
> 
> 
> Diffs
> -
> 
>   README.md 9ebb2a3 
>   conf/flume-env.sh.template 07182ca 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3 
>   flume-ng-sources/flume-taildir-source/pom.xml 96c2468 
>   pom.xml f62c99a 
> 
> Diff: https://reviews.apache.org/r/53056/diff/
> 
> 
> Testing
> ---
> 
> The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91".
> All unit tests, except for the known flaky ones, successfully passed.
> 
> 
> Thanks,
> 
> Lior Zeno
> 
>

Re: Review Request 53056: FLUME-2945: Bump java target version to 1.8

[Lior Zeno]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53056/
---

(Updated Oct. 21, 2016, 3:56 p.m.)


Review request for Flume.


Changes
---

Update flume env file


Repository: flume-git


Description
---

We should move to Java 8 as a minimum requirement.

1. Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html.
2. Many projects are adopting java 8 as a minimum requirement, for instance:

* Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722
* Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624
* elasticsearch 5: 
https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html


Diffs (updated)
-

  README.md 9ebb2a3 
  conf/flume-env.sh.template 07182ca 
  flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3 
  flume-ng-sources/flume-taildir-source/pom.xml 96c2468 
  pom.xml f62c99a 

Diff: https://reviews.apache.org/r/53056/diff/


Testing
---

The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91".
All unit tests, except for the known flaky ones, successfully passed.


Thanks,

Lior Zeno

Re: Enabling Travis-CI on Flume

[Lior Zeno]

Mike, I was not holding Donat back. I was just suggesting ways to configure
Jenins, per Donat's request. I'm sorry if my former post delivered the
wrong message.

On Fri, Oct 21, 2016 at 6:29 PM, Mike Percy  wrote:

> Personally I prefer Jenkins over TravisCI for various reasons however if
> Donat is willing to do the work of adding pre-commit checks on PRs via
> Travis then I say let him do it, in the Apache spirit of "let they that do
> the work make the decisions".
>
> If someone actually spends the time to set up Jenkins and configure it to
> do the same thing, then great, let's switch when it's ready.
>
> Note that only ASF committers have access to Jenkins so non-committers will
> need to work with a committer to get it done if they want to help.
>
> Mike
>
> On Fri, Oct 21, 2016 at 3:46 PM, Lior Zeno  wrote:
>
> > There are many ways to do it, for example:
> > https://www.theguild.nl/building-github-pull-requests-using-
> > jenkins-pipelines/
> > or https://www.theguild.nl/building-github-pull-requests-with-jenkins/
> for
> > earlier versions of Jenkins.
> > I do not really care if it would be Jenkins or Travis, but I do think
> that
> > we can get Jenkins configured faster since we already have it. I can help
> > with the configuration.
> >
> > On Fri, Oct 21, 2016 at 5:17 PM, Balazs Donat Bessenyei <
> > bes...@cloudera.com
> > > wrote:
> >
> > > As I haven't received any objections to enabling Travis, I'm going to
> > > ask INFRA to enable it for Flume soon.
> > >
> > > This change would help submitting and reviewing pull requests.
> > >
> > > If someone figures out how we could use Jenkins for this purpose, we
> > > can always disable Travis.
> > >
> > > PS. there are more projects using Travis:
> > > https://issues.apache.org/jira/browse/INFRA-12757?jql=
> > > project%20%3D%20INFRA%20AND%20text%20~%20travis%20ORDER%
> > > 20BY%20updated%20DESC
> > >
> > > On Fri, Oct 14, 2016 at 5:41 PM, Attila Simon 
> wrote:
> > > > Denes I'm happy to help you in this endeavor of setting up jenkins
> job
> > > for
> > > > verifying pull requests.
> > > >
> > > >
> > > > *Attila Simon*
> > > > Software Engineer
> > > > Email:   s...@cloudera.com
> > > >
> > > > [image: Cloudera Inc.]
> > > >
> > > > On Fri, Oct 14, 2016 at 2:47 PM, Denes Arvay 
> > wrote:
> > > >
> > > >> I'd also vote for Jenkins with github PRs.
> > > >> I just checked Mesos and the PRs are checked by Travis, or at least
> > they
> > > >> experienced with it, there's a short discussion regarding to Travis
> at
> > > >> https://github.com/apache/mesos/pull/165
> > > >>
> > > >> As for the jenkins pull request job I'd be happy to set it up or
> help
> > > >> setting it up.
> > > >>
> > > >> Denes
> > > >>
> > > >> On Fri, Oct 14, 2016 at 2:15 PM Lior Zeno 
> wrote:
> > > >>
> > > >> Are we switching to PRs from patches + RB? In Apache Mesos, they
> have
> > a
> > > >>
> > > >> review bot that can leave a comment on the patch, we could try and
> > port
> > > it
> > > >>
> > > >> to Flume. I think they use Jenkins too.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Oct 14, 2016 at 3:11 PM, Balazs Donat Bessenyei <
> > > >> bes...@cloudera.com
> > > >>
> > > >> > wrote:
> > > >>
> > > >>
> > > >>
> > > >> > If the same function can be achieved with Jenkins and it's easy
> > > >>
> > > >> > (+quick) to set up, I'm totally happy with that.
> > > >>
> > > >> >
> > > >>
> > > >> > What do we have to do to enable Jenkins builds on PR-s?
> > > >>
> > > >> >
> > > >>
> > > >> > On Fri, Oct 14, 2016 at 2:05 PM, Lior Zeno 
> > > wrote:
> > > >>
> > > >> > > There are ways to do the same with Jenkins, for instance, see
> this
> > > SO
> > > >>
> > > >> > > thread
> > > >>
> > > >> > > http://stackoverflow.com/questions/37661602/how-to-set-
> > > >>
> > > >> > up-a-github-pull-request-build-in-a-jenkinsfile
> > > >>
> > > >> > >
> > > >>
> > > >> > > On Fri, Oct 14, 2016 at 11:09 AM, Balazs Donat Bessenyei <
> > > >>
> > > >> > > bes...@cloudera.com> wrote:
> > > >>
> > > >> > >
> > > >>
> > > >> > >> My primary reason for Travis (vs. Jenkins) was that I have
> > > experience
> > > >>
> > > >> > with
> > > >>
> > > >> > >> it.
> > > >>
> > > >> > >>
> > > >>
> > > >> > >> And it leaves these happy little checkmarks:
> > > >>
> > > >> > >> https://github.com/sebastianbergmann/phpunit/pull/1051/commits
> > on
> > > the
> > > >>
> > > >> > >> commits and messages as seen on
> > > >>
> > > >> > >> https://github.com/apache/hive/pull/107 .
> > > >>
> > > >> > >>
> > > >>
> > > >> > >> Jenkins is probably configurable to achieve similar function.
> > > However,
> > > >>
> > > >> > >> I have no idea how to do such. (And could not find an example
> > when
> > > I
> > > >>
> > > >> > >> did a quick search.)
> > > >>
> > > >> > >>
> > > >>
> > > >> > >> Are there any disadvantages of enabling Travis on Flume?
> > > >>
> > > >> > >>
> > > >>
> > > >> > 

Re: Enabling Travis-CI on Flume

[Mike Percy]

Personally I prefer Jenkins over TravisCI for various reasons however if
Donat is willing to do the work of adding pre-commit checks on PRs via
Travis then I say let him do it, in the Apache spirit of "let they that do
the work make the decisions".

If someone actually spends the time to set up Jenkins and configure it to
do the same thing, then great, let's switch when it's ready.

Note that only ASF committers have access to Jenkins so non-committers will
need to work with a committer to get it done if they want to help.

Mike

On Fri, Oct 21, 2016 at 3:46 PM, Lior Zeno  wrote:

> There are many ways to do it, for example:
> https://www.theguild.nl/building-github-pull-requests-using-
> jenkins-pipelines/
> or https://www.theguild.nl/building-github-pull-requests-with-jenkins/ for
> earlier versions of Jenkins.
> I do not really care if it would be Jenkins or Travis, but I do think that
> we can get Jenkins configured faster since we already have it. I can help
> with the configuration.
>
> On Fri, Oct 21, 2016 at 5:17 PM, Balazs Donat Bessenyei <
> bes...@cloudera.com
> > wrote:
>
> > As I haven't received any objections to enabling Travis, I'm going to
> > ask INFRA to enable it for Flume soon.
> >
> > This change would help submitting and reviewing pull requests.
> >
> > If someone figures out how we could use Jenkins for this purpose, we
> > can always disable Travis.
> >
> > PS. there are more projects using Travis:
> > https://issues.apache.org/jira/browse/INFRA-12757?jql=
> > project%20%3D%20INFRA%20AND%20text%20~%20travis%20ORDER%
> > 20BY%20updated%20DESC
> >
> > On Fri, Oct 14, 2016 at 5:41 PM, Attila Simon  wrote:
> > > Denes I'm happy to help you in this endeavor of setting up jenkins job
> > for
> > > verifying pull requests.
> > >
> > >
> > > *Attila Simon*
> > > Software Engineer
> > > Email:   s...@cloudera.com
> > >
> > > [image: Cloudera Inc.]
> > >
> > > On Fri, Oct 14, 2016 at 2:47 PM, Denes Arvay 
> wrote:
> > >
> > >> I'd also vote for Jenkins with github PRs.
> > >> I just checked Mesos and the PRs are checked by Travis, or at least
> they
> > >> experienced with it, there's a short discussion regarding to Travis at
> > >> https://github.com/apache/mesos/pull/165
> > >>
> > >> As for the jenkins pull request job I'd be happy to set it up or help
> > >> setting it up.
> > >>
> > >> Denes
> > >>
> > >> On Fri, Oct 14, 2016 at 2:15 PM Lior Zeno  wrote:
> > >>
> > >> Are we switching to PRs from patches + RB? In Apache Mesos, they have
> a
> > >>
> > >> review bot that can leave a comment on the patch, we could try and
> port
> > it
> > >>
> > >> to Flume. I think they use Jenkins too.
> > >>
> > >>
> > >>
> > >> On Fri, Oct 14, 2016 at 3:11 PM, Balazs Donat Bessenyei <
> > >> bes...@cloudera.com
> > >>
> > >> > wrote:
> > >>
> > >>
> > >>
> > >> > If the same function can be achieved with Jenkins and it's easy
> > >>
> > >> > (+quick) to set up, I'm totally happy with that.
> > >>
> > >> >
> > >>
> > >> > What do we have to do to enable Jenkins builds on PR-s?
> > >>
> > >> >
> > >>
> > >> > On Fri, Oct 14, 2016 at 2:05 PM, Lior Zeno 
> > wrote:
> > >>
> > >> > > There are ways to do the same with Jenkins, for instance, see this
> > SO
> > >>
> > >> > > thread
> > >>
> > >> > > http://stackoverflow.com/questions/37661602/how-to-set-
> > >>
> > >> > up-a-github-pull-request-build-in-a-jenkinsfile
> > >>
> > >> > >
> > >>
> > >> > > On Fri, Oct 14, 2016 at 11:09 AM, Balazs Donat Bessenyei <
> > >>
> > >> > > bes...@cloudera.com> wrote:
> > >>
> > >> > >
> > >>
> > >> > >> My primary reason for Travis (vs. Jenkins) was that I have
> > experience
> > >>
> > >> > with
> > >>
> > >> > >> it.
> > >>
> > >> > >>
> > >>
> > >> > >> And it leaves these happy little checkmarks:
> > >>
> > >> > >> https://github.com/sebastianbergmann/phpunit/pull/1051/commits
> on
> > the
> > >>
> > >> > >> commits and messages as seen on
> > >>
> > >> > >> https://github.com/apache/hive/pull/107 .
> > >>
> > >> > >>
> > >>
> > >> > >> Jenkins is probably configurable to achieve similar function.
> > However,
> > >>
> > >> > >> I have no idea how to do such. (And could not find an example
> when
> > I
> > >>
> > >> > >> did a quick search.)
> > >>
> > >> > >>
> > >>
> > >> > >> Are there any disadvantages of enabling Travis on Flume?
> > >>
> > >> > >>
> > >>
> > >> > >>
> > >>
> > >> > >> Thank you,
> > >>
> > >> > >>
> > >>
> > >> > >> Donat
> > >>
> > >> > >>
> > >>
> > >> > >> On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno 
> > >> wrote:
> > >>
> > >> > >> > Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0,
> we
> > >> will
> > >>
> > >> > be
> > >>
> > >> > >> > able to define the build step via Jenkinsfile which becomes
> very
> > >>
> > >> > similar
> > >>
> > >> > >> to
> > >>
> > >> > >> > Travis.
> > >>
> > >> > >> > Is there any reason to prefer Travis over Jenkins in our 

Re: Enabling Travis-CI on Flume

[Lior Zeno]

There are many ways to do it, for example:
https://www.theguild.nl/building-github-pull-requests-using-jenkins-pipelines/
or https://www.theguild.nl/building-github-pull-requests-with-jenkins/ for
earlier versions of Jenkins.
I do not really care if it would be Jenkins or Travis, but I do think that
we can get Jenkins configured faster since we already have it. I can help
with the configuration.

On Fri, Oct 21, 2016 at 5:17 PM, Balazs Donat Bessenyei  wrote:

> As I haven't received any objections to enabling Travis, I'm going to
> ask INFRA to enable it for Flume soon.
>
> This change would help submitting and reviewing pull requests.
>
> If someone figures out how we could use Jenkins for this purpose, we
> can always disable Travis.
>
> PS. there are more projects using Travis:
> https://issues.apache.org/jira/browse/INFRA-12757?jql=
> project%20%3D%20INFRA%20AND%20text%20~%20travis%20ORDER%
> 20BY%20updated%20DESC
>
> On Fri, Oct 14, 2016 at 5:41 PM, Attila Simon  wrote:
> > Denes I'm happy to help you in this endeavor of setting up jenkins job
> for
> > verifying pull requests.
> >
> >
> > *Attila Simon*
> > Software Engineer
> > Email:   s...@cloudera.com
> >
> > [image: Cloudera Inc.]
> >
> > On Fri, Oct 14, 2016 at 2:47 PM, Denes Arvay  wrote:
> >
> >> I'd also vote for Jenkins with github PRs.
> >> I just checked Mesos and the PRs are checked by Travis, or at least they
> >> experienced with it, there's a short discussion regarding to Travis at
> >> https://github.com/apache/mesos/pull/165
> >>
> >> As for the jenkins pull request job I'd be happy to set it up or help
> >> setting it up.
> >>
> >> Denes
> >>
> >> On Fri, Oct 14, 2016 at 2:15 PM Lior Zeno  wrote:
> >>
> >> Are we switching to PRs from patches + RB? In Apache Mesos, they have a
> >>
> >> review bot that can leave a comment on the patch, we could try and port
> it
> >>
> >> to Flume. I think they use Jenkins too.
> >>
> >>
> >>
> >> On Fri, Oct 14, 2016 at 3:11 PM, Balazs Donat Bessenyei <
> >> bes...@cloudera.com
> >>
> >> > wrote:
> >>
> >>
> >>
> >> > If the same function can be achieved with Jenkins and it's easy
> >>
> >> > (+quick) to set up, I'm totally happy with that.
> >>
> >> >
> >>
> >> > What do we have to do to enable Jenkins builds on PR-s?
> >>
> >> >
> >>
> >> > On Fri, Oct 14, 2016 at 2:05 PM, Lior Zeno 
> wrote:
> >>
> >> > > There are ways to do the same with Jenkins, for instance, see this
> SO
> >>
> >> > > thread
> >>
> >> > > http://stackoverflow.com/questions/37661602/how-to-set-
> >>
> >> > up-a-github-pull-request-build-in-a-jenkinsfile
> >>
> >> > >
> >>
> >> > > On Fri, Oct 14, 2016 at 11:09 AM, Balazs Donat Bessenyei <
> >>
> >> > > bes...@cloudera.com> wrote:
> >>
> >> > >
> >>
> >> > >> My primary reason for Travis (vs. Jenkins) was that I have
> experience
> >>
> >> > with
> >>
> >> > >> it.
> >>
> >> > >>
> >>
> >> > >> And it leaves these happy little checkmarks:
> >>
> >> > >> https://github.com/sebastianbergmann/phpunit/pull/1051/commits on
> the
> >>
> >> > >> commits and messages as seen on
> >>
> >> > >> https://github.com/apache/hive/pull/107 .
> >>
> >> > >>
> >>
> >> > >> Jenkins is probably configurable to achieve similar function.
> However,
> >>
> >> > >> I have no idea how to do such. (And could not find an example when
> I
> >>
> >> > >> did a quick search.)
> >>
> >> > >>
> >>
> >> > >> Are there any disadvantages of enabling Travis on Flume?
> >>
> >> > >>
> >>
> >> > >>
> >>
> >> > >> Thank you,
> >>
> >> > >>
> >>
> >> > >> Donat
> >>
> >> > >>
> >>
> >> > >> On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno 
> >> wrote:
> >>
> >> > >> > Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we
> >> will
> >>
> >> > be
> >>
> >> > >> > able to define the build step via Jenkinsfile which becomes very
> >>
> >> > similar
> >>
> >> > >> to
> >>
> >> > >> > Travis.
> >>
> >> > >> > Is there any reason to prefer Travis over Jenkins in our case?
> >>
> >> > >> >
> >>
> >> > >> > On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei <
> >>
> >> > >> bes...@cloudera.com
> >>
> >> > >> >> wrote:
> >>
> >> > >> >
> >>
> >> > >> >> Hi All,
> >>
> >> > >> >>
> >>
> >> > >> >> Having something that checks proposed patches (PR-s especially)
> >>
> >> > >> >> automatically would help a lot with the development on Flume.
> >>
> >> > >> >>
> >>
> >> > >> >> I think, Travis-CI could be an easy solution and (afaik) we'd
> only
> >>
> >> > have
> >>
> >> > >> to
> >>
> >> > >> >> ask infra to enable it for us.
> >>
> >> > >> >>
> >>
> >> > >> >> Please, let me know your thoughts.
> >>
> >> > >> >>
> >>
> >> > >> >> Thank you,
> >>
> >> > >> >>
> >>
> >> > >> >> Donat
> >>
> >> > >> >>
> >>
> >> > >>
> >>
> >> >
> >>
>

Re: Enabling Travis-CI on Flume

[Balazs Donat Bessenyei]

As I haven't received any objections to enabling Travis, I'm going to
ask INFRA to enable it for Flume soon.

This change would help submitting and reviewing pull requests.

If someone figures out how we could use Jenkins for this purpose, we
can always disable Travis.

PS. there are more projects using Travis:
https://issues.apache.org/jira/browse/INFRA-12757?jql=project%20%3D%20INFRA%20AND%20text%20~%20travis%20ORDER%20BY%20updated%20DESC

On Fri, Oct 14, 2016 at 5:41 PM, Attila Simon  wrote:
> Denes I'm happy to help you in this endeavor of setting up jenkins job for
> verifying pull requests.
>
>
> *Attila Simon*
> Software Engineer
> Email:   s...@cloudera.com
>
> [image: Cloudera Inc.]
>
> On Fri, Oct 14, 2016 at 2:47 PM, Denes Arvay  wrote:
>
>> I'd also vote for Jenkins with github PRs.
>> I just checked Mesos and the PRs are checked by Travis, or at least they
>> experienced with it, there's a short discussion regarding to Travis at
>> https://github.com/apache/mesos/pull/165
>>
>> As for the jenkins pull request job I'd be happy to set it up or help
>> setting it up.
>>
>> Denes
>>
>> On Fri, Oct 14, 2016 at 2:15 PM Lior Zeno  wrote:
>>
>> Are we switching to PRs from patches + RB? In Apache Mesos, they have a
>>
>> review bot that can leave a comment on the patch, we could try and port it
>>
>> to Flume. I think they use Jenkins too.
>>
>>
>>
>> On Fri, Oct 14, 2016 at 3:11 PM, Balazs Donat Bessenyei <
>> bes...@cloudera.com
>>
>> > wrote:
>>
>>
>>
>> > If the same function can be achieved with Jenkins and it's easy
>>
>> > (+quick) to set up, I'm totally happy with that.
>>
>> >
>>
>> > What do we have to do to enable Jenkins builds on PR-s?
>>
>> >
>>
>> > On Fri, Oct 14, 2016 at 2:05 PM, Lior Zeno  wrote:
>>
>> > > There are ways to do the same with Jenkins, for instance, see this SO
>>
>> > > thread
>>
>> > > http://stackoverflow.com/questions/37661602/how-to-set-
>>
>> > up-a-github-pull-request-build-in-a-jenkinsfile
>>
>> > >
>>
>> > > On Fri, Oct 14, 2016 at 11:09 AM, Balazs Donat Bessenyei <
>>
>> > > bes...@cloudera.com> wrote:
>>
>> > >
>>
>> > >> My primary reason for Travis (vs. Jenkins) was that I have experience
>>
>> > with
>>
>> > >> it.
>>
>> > >>
>>
>> > >> And it leaves these happy little checkmarks:
>>
>> > >> https://github.com/sebastianbergmann/phpunit/pull/1051/commits on the
>>
>> > >> commits and messages as seen on
>>
>> > >> https://github.com/apache/hive/pull/107 .
>>
>> > >>
>>
>> > >> Jenkins is probably configurable to achieve similar function. However,
>>
>> > >> I have no idea how to do such. (And could not find an example when I
>>
>> > >> did a quick search.)
>>
>> > >>
>>
>> > >> Are there any disadvantages of enabling Travis on Flume?
>>
>> > >>
>>
>> > >>
>>
>> > >> Thank you,
>>
>> > >>
>>
>> > >> Donat
>>
>> > >>
>>
>> > >> On Thu, Oct 13, 2016 at 6:06 PM, Lior Zeno 
>> wrote:
>>
>> > >> > Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we
>> will
>>
>> > be
>>
>> > >> > able to define the build step via Jenkinsfile which becomes very
>>
>> > similar
>>
>> > >> to
>>
>> > >> > Travis.
>>
>> > >> > Is there any reason to prefer Travis over Jenkins in our case?
>>
>> > >> >
>>
>> > >> > On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei <
>>
>> > >> bes...@cloudera.com
>>
>> > >> >> wrote:
>>
>> > >> >
>>
>> > >> >> Hi All,
>>
>> > >> >>
>>
>> > >> >> Having something that checks proposed patches (PR-s especially)
>>
>> > >> >> automatically would help a lot with the development on Flume.
>>
>> > >> >>
>>
>> > >> >> I think, Travis-CI could be an easy solution and (afaik) we'd only
>>
>> > have
>>
>> > >> to
>>
>> > >> >> ask infra to enable it for us.
>>
>> > >> >>
>>
>> > >> >> Please, let me know your thoughts.
>>
>> > >> >>
>>
>> > >> >> Thank you,
>>
>> > >> >>
>>
>> > >> >> Donat
>>
>> > >> >>
>>
>> > >>
>>
>> >
>>

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

> On Oct. 21, 2016, 1:21 p.m., Attila Simon wrote:
> > Ship It!

Discussed offline that there will be a follow up patch to use builder pattern 
for constructing BucketWriter. (So that further testing only instance mutators 
can be eliminated)


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153538
---


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153538
---


Ship it!




Ship It!

- Attila Simon


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 53056: FLUME-2945: Bump java target version to 1.8

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53056/#review153527
---



Hi Lior,
conf/flume-env.sh.template#22 is still referenceing to java-6-sun. Could you 
please include that as well.

- Attila Simon


On Oct. 20, 2016, 4:50 p.m., Lior Zeno wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53056/
> ---
> 
> (Updated Oct. 20, 2016, 4:50 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> We should move to Java 8 as a minimum requirement.
> 
> 1. Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html.
> 2. Many projects are adopting java 8 as a minimum requirement, for instance:
> 
> * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722
> * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624
> * elasticsearch 5: 
> https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html
> 
> 
> Diffs
> -
> 
>   README.md 9ebb2a3 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3 
>   flume-ng-sources/flume-taildir-source/pom.xml 96c2468 
>   pom.xml f62c99a 
> 
> Diff: https://reviews.apache.org/r/53056/diff/
> 
> 
> Testing
> ---
> 
> The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91".
> All unit tests, except for the known flaky ones, successfully passed.
> 
> 
> Thanks,
> 
> Lior Zeno
> 
>

Re: jira reference is missing from git commit messages

[Mike Percy]

Hi Attila,
Thanks for raising this concern of yours. Please see inline.

On Fri, Oct 21, 2016 at 12:17 PM, Attila Simon  wrote:

> The how to contribute page asks to have a jira first for each change. I
> guess with allowing pull requests we have to update the how to contribute
> page as well (which only describes attaching patch and using review board).
>

Yes, it appeared that we had consensus for allowing PR on a couple separate
dev@ threads a while back [1], [2].

The Flume contributor docs are currently a bit stale. I have recently
updated the How to Release cwiki page but if you want to volunteer to
update some of the other docs that would be very welcome! Please let me
know if you want to help and I can give you cwiki edit access if you send
me your account id. Side note: Donat has recently proposed moving those
docs from cwiki to the Git repo which I think would be a big improvement.

I would like to ask committers to reestablish the habit of having a jira
> for each commit and start the message with that jira.
>

Forcing people to file a JIRA when they are already doing a PR feels like
pointless extra paperwork to me. There is certainly a place for JIRA in a
software project, but I think that is to track unfixed bugs, ongoing tasks
/ projects, etc.


> If we would like to relax the have jira for each change (for pull requests)
> then I would suggest putting the request id as the first thing in the
> commit message.
>

Why?

If you click on this:
https://github.com/apache/flume/commit/87d4c2c13862144eb578b211bcf800b2206834ff

You will see that the text "Closes #70" creates a clickable hyperlink to
the PR. Is this not sufficient for tracking purposes?

Mike

[1] https://s.apache.org/k31f
[2] https://s.apache.org/Skm2

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

> On Oct. 18, 2016, 4:26 p.m., Attila Simon wrote:
> > Ahh now I see what is going on. BucketWriter instantiate a Clock class 
> > which is in turn used in the file name counter. The test first instantiates 
> > the BucketWriter - which initialize the filename counter -then overrides 
> > the Clock but since Clock is not used in BucketWriter directly - only via 
> > the file name counter at instantiation time so it won't be updated with the 
> > override - the test failes as it checks the file name.
> > So actually a single extra line into the `BucketWriter.setClock(Clock 
> > clock)` method would solve the issue:
> > ```
> >   void setClock(Clock clock) {
> > this.clock = clock;
> > this.fileExtensionCounter.set(clock.currentTimeMillis());
> >   }
> > ```
> > This way the test would update the relevant information - file name counter 
> > - which then in turn can be checked.
> > 
> > Could you please consider simplifying your change?
> 
> Balázs Donát Bessenyei wrote:
> Thank you for the review!
> 
> Even though the change you have suggested does fix the flakiness of the 
> tests (which is awesome!), I am not sure people would expect the caused 
> side-effect in void setClock(Clock clock) -> fileExtensionCounter.
> 
> The change in its current form does eliminate private Clock clock which 
> is "nice". However, it also eliminates void setClock(Clock clock) which helps 
> avoiding violation of the monotony of fileExtensionCounter (which is 
> assumed), thus it improves the integrity of BucketWriter (encapsulation).
> 
> Please, let me know your thoughts.

Clock is only used to initialize file extension counter (so there is 
initialization phase binding between these two). I just made sure this expected 
binding is maintained in the setter as well through the lifetime of a 
BucketWriter. I may suggest setClock should be also tagged with 
@VisibleForTesting (it is already scoped with package auto) to make it clear 
that it is only for tests (which is the case now by checking its usages).


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153104
---


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

jira reference is missing from git commit messages

[Attila Simon]

Hi,

Not too long ago all the messages in the git log started with the
corresponding jira. I found this habit very useful.
The how to contribute page asks to have a jira first for each change. I
guess with allowing pull requests we have to update the how to contribute
page as well (which only describes attaching patch and using review board).

I would like to ask committers to reestablish the habit of having a jira
for each commit and start the message with that jira.
If we would like to relax the have jira for each change (for pull requests)
then I would suggest putting the request id as the first thing in the
commit message.

instead of:
Author: Mike Percy 
Date:   Wed Oct 12 19:32:13 2016 +0200

Remove test dependencies from binary release artifact

This patch removes some test-specific dependencies from the binary
release artifact. These were introduced by the new
flume-shared-kafka-test module that is intended for sharing test code.
Please see the new comment in bin.xml for more information.

Reviewers: Bessenyei Balázs Donát

Closes #70


I would suggest:
Author: Mike Percy 
Date:   Wed Oct 12 19:32:13 2016 +0200

FLUME- Remove test dependencies from binary release artifact

This patch removes some test-specific dependencies from the binary
release artifact. These were introduced by the new
flume-shared-kafka-test module that is intended for sharing test code.
Please see the new comment in bin.xml for more information.

Reviewers: Bessenyei Balázs Donát

Closes #70

or the relaxed version if it is really needed to drop the jira requirement:
Author: Mike Percy 
Date:   Wed Oct 12 19:32:13 2016 +0200

FLUME-PR70 Remove test dependencies from binary release artifact

This patch removes some test-specific dependencies from the binary
release artifact. These were introduced by the new
flume-shared-kafka-test module that is intended for sharing test code.
Please see the new comment in bin.xml for more information.

Reviewers: Bessenyei Balázs Donát

Closes #70


Cheers,
Attila

[jira] [Commented] (FLUME-3012) Sending a term signal can not shutdown Flume agent when KafkaChannel starting has exceptions

[Ping Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15594537#comment-15594537
 ] 

Ping Wang commented on FLUME-3012:
--

I found flume-ng-node/src/main/java/org/apache/flume/node/Application.java 
already has Shutdown Hook to handle SIGTERM:

final Application appReference = application;
  Runtime.getRuntime().addShutdownHook(new Thread("agent-shutdown-hook") {
@Override
public void run() {
  appReference.stop();
}
  });

In the scenario,  addShutdownHook was called but stop operation was not done. 


> Sending a term signal can not shutdown Flume agent when KafkaChannel starting 
> has exceptions
> 
>
> Key: FLUME-3012
> URL: https://issues.apache.org/jira/browse/FLUME-3012
> Project: Flume
>  Issue Type: Bug
>  Components: Kafka Channel
>Affects Versions: v1.6.0
> Environment: Flume 1.6.0+Kafka 0.9 
>Reporter: Ping Wang
> Fix For: v1.8.0
>
> Attachments: threaddumps.log
>
>
> Use Kafka Channel in the agent configuration:
> tier1.sources = source1
> tier1.channels = channel1
> tier1.sinks = sink1
> tier1.sources.source1.type = exec
> tier1.sources.source1.command = /usr/bin/vmstat 1
> tier1.sources.source1.channels = channel1
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.kafka.bootstrap.servers = a.b.c.d:6667
> tier1.sinks.sink1.type = hdfs
> tier1.sinks.sink1.hdfs.path = /tmp/kafka/channel
> tier1.sinks.sink1.hdfs.rollInterval = 5
> tier1.sinks.sink1.hdfs.rollSize = 0
> tier1.sinks.sink1.hdfs.rollCount = 0
> tier1.sinks.sink1.hdfs.fileType = DataStream
> tier1.sinks.sink1.channel = channel1
> Accidentally kaka.bootstrap.servers is not correct,  errors will be thrown 
> out:
> ..
> )] Waiting for channel: channel1 to start. Sleeping for 500 ms
> 2016-10-21 01:15:50,739 (conf-file-poller-0) [INFO - 
> org.apache.flume.node.Application.startAllComponents(Application.java:161)] 
> Waiting for channel: channel1 to start. Sleeping for 500 ms
> 2016-10-21 01:15:51,240 (conf-file-poller-0) [INFO - 
> org.apache.flume.node.Application.startAllComponents(Application.java:161)] 
> Waiting for channel: channel1 to start. Sleeping for 500 ms
> 2016-10-21 01:15:51,735 (lifecycleSupervisor-1-6) [INFO - 
> org.apache.flume.channel.kafka.KafkaChannel.start(KafkaChannel.java:115)] 
> Starting Kafka Channel: channel1
> 2016-10-21 01:15:51,737 (lifecycleSupervisor-1-6) [INFO - 
> org.apache.kafka.common.config.AbstractConfig.logAll(AbstractConfig.java:165)]
>  ProducerConfig values: 
>   compression.type = none
>   metric.reporters = []
>   metadata.max.age.ms = 30
>   metadata.fetch.timeout.ms = 6
>   reconnect.backoff.ms = 50
>   sasl.kerberos.ticket.renew.window.factor = 0.8
>   bootstrap.servers = [a.b.c.d:6667]
>   retry.backoff.ms = 100
>   sasl.kerberos.kinit.cmd = /usr/bin/kinit
>   buffer.memory = 33554432
>   timeout.ms = 3
>   key.serializer = class 
> org.apache.kafka.common.serialization.StringSerializer
>   sasl.kerberos.service.name = null
>   sasl.kerberos.ticket.renew.jitter = 0.05
>   ssl.keystore.type = JKS
>   ssl.trustmanager.algorithm = PKIX
>   block.on.buffer.full = false
>   ssl.key.password = null
>   max.block.ms = 6
>   sasl.kerberos.min.time.before.relogin = 6
>   connections.max.idle.ms = 54
>   ssl.truststore.password = null
>   max.in.flight.requests.per.connection = 5
>   metrics.num.samples = 2
>   client.id = 
>   ssl.endpoint.identification.algorithm = null
>   ssl.protocol = TLS
>   request.timeout.ms = 3
>   ssl.provider = null
>   ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>   acks = all
>   batch.size = 16384
>   ssl.keystore.location = null
>   receive.buffer.bytes = 32768
>   ssl.cipher.suites = null
>   ssl.truststore.type = JKS
>   security.protocol = PLAINTEXT
>   retries = 0
>   max.request.size = 1048576
>   value.serializer = class 
> org.apache.kafka.common.serialization.ByteArraySerializer
>   ssl.truststore.location = null
>   ssl.keystore.password = null
>   ssl.keymanager.algorithm = SunX509
>   metrics.sample.window.ms = 3
>   partitioner.class = class 
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
>   send.buffer.bytes = 131072
>   linger.ms = 0
> 2016-10-21 01:15:51,742 (lifecycleSupervisor-1-6) [DEBUG - 
> org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added 
> sensor with name bufferpool-wait-time
> 2016-10-21 01:15:51,743 (lifecycleSupervisor-1-6) [DEBUG - 
> 

[jira] [Updated] (FLUME-3012) Sending a term signal can not shutdown Flume agent when KafkaChannel starting has exceptions

[Ping Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-3012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ping Wang updated FLUME-3012:
-
Attachment: threaddumps.log

Attach the the dump from jstack.

> Sending a term signal can not shutdown Flume agent when KafkaChannel starting 
> has exceptions
> 
>
> Key: FLUME-3012
> URL: https://issues.apache.org/jira/browse/FLUME-3012
> Project: Flume
>  Issue Type: Bug
>  Components: Kafka Channel
>Affects Versions: v1.6.0
> Environment: Flume 1.6.0+Kafka 0.9 
>Reporter: Ping Wang
> Fix For: v1.8.0
>
> Attachments: threaddumps.log
>
>
> Use Kafka Channel in the agent configuration:
> tier1.sources = source1
> tier1.channels = channel1
> tier1.sinks = sink1
> tier1.sources.source1.type = exec
> tier1.sources.source1.command = /usr/bin/vmstat 1
> tier1.sources.source1.channels = channel1
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.kafka.bootstrap.servers = a.b.c.d:6667
> tier1.sinks.sink1.type = hdfs
> tier1.sinks.sink1.hdfs.path = /tmp/kafka/channel
> tier1.sinks.sink1.hdfs.rollInterval = 5
> tier1.sinks.sink1.hdfs.rollSize = 0
> tier1.sinks.sink1.hdfs.rollCount = 0
> tier1.sinks.sink1.hdfs.fileType = DataStream
> tier1.sinks.sink1.channel = channel1
> Accidentally kaka.bootstrap.servers is not correct,  errors will be thrown 
> out:
> ..
> )] Waiting for channel: channel1 to start. Sleeping for 500 ms
> 2016-10-21 01:15:50,739 (conf-file-poller-0) [INFO - 
> org.apache.flume.node.Application.startAllComponents(Application.java:161)] 
> Waiting for channel: channel1 to start. Sleeping for 500 ms
> 2016-10-21 01:15:51,240 (conf-file-poller-0) [INFO - 
> org.apache.flume.node.Application.startAllComponents(Application.java:161)] 
> Waiting for channel: channel1 to start. Sleeping for 500 ms
> 2016-10-21 01:15:51,735 (lifecycleSupervisor-1-6) [INFO - 
> org.apache.flume.channel.kafka.KafkaChannel.start(KafkaChannel.java:115)] 
> Starting Kafka Channel: channel1
> 2016-10-21 01:15:51,737 (lifecycleSupervisor-1-6) [INFO - 
> org.apache.kafka.common.config.AbstractConfig.logAll(AbstractConfig.java:165)]
>  ProducerConfig values: 
>   compression.type = none
>   metric.reporters = []
>   metadata.max.age.ms = 30
>   metadata.fetch.timeout.ms = 6
>   reconnect.backoff.ms = 50
>   sasl.kerberos.ticket.renew.window.factor = 0.8
>   bootstrap.servers = [a.b.c.d:6667]
>   retry.backoff.ms = 100
>   sasl.kerberos.kinit.cmd = /usr/bin/kinit
>   buffer.memory = 33554432
>   timeout.ms = 3
>   key.serializer = class 
> org.apache.kafka.common.serialization.StringSerializer
>   sasl.kerberos.service.name = null
>   sasl.kerberos.ticket.renew.jitter = 0.05
>   ssl.keystore.type = JKS
>   ssl.trustmanager.algorithm = PKIX
>   block.on.buffer.full = false
>   ssl.key.password = null
>   max.block.ms = 6
>   sasl.kerberos.min.time.before.relogin = 6
>   connections.max.idle.ms = 54
>   ssl.truststore.password = null
>   max.in.flight.requests.per.connection = 5
>   metrics.num.samples = 2
>   client.id = 
>   ssl.endpoint.identification.algorithm = null
>   ssl.protocol = TLS
>   request.timeout.ms = 3
>   ssl.provider = null
>   ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>   acks = all
>   batch.size = 16384
>   ssl.keystore.location = null
>   receive.buffer.bytes = 32768
>   ssl.cipher.suites = null
>   ssl.truststore.type = JKS
>   security.protocol = PLAINTEXT
>   retries = 0
>   max.request.size = 1048576
>   value.serializer = class 
> org.apache.kafka.common.serialization.ByteArraySerializer
>   ssl.truststore.location = null
>   ssl.keystore.password = null
>   ssl.keymanager.algorithm = SunX509
>   metrics.sample.window.ms = 3
>   partitioner.class = class 
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
>   send.buffer.bytes = 131072
>   linger.ms = 0
> 2016-10-21 01:15:51,742 (lifecycleSupervisor-1-6) [DEBUG - 
> org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added 
> sensor with name bufferpool-wait-time
> 2016-10-21 01:15:51,743 (lifecycleSupervisor-1-6) [DEBUG - 
> org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added 
> sensor with name buffer-exhausted-records
> 2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [INFO - 
> org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:613)]
>  Closing the Kafka producer with timeoutMillis = 0 ms.
> 2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [DEBUG - 
> 

[jira] [Created] (FLUME-3012) Sending a term signal can not shutdown Flume agent when KafkaChannel starting has exceptions

[Ping Wang (JIRA)]

Ping Wang created FLUME-3012:


 Summary: Sending a term signal can not shutdown Flume agent when 
KafkaChannel starting has exceptions
 Key: FLUME-3012
 URL: https://issues.apache.org/jira/browse/FLUME-3012
 Project: Flume
  Issue Type: Bug
  Components: Kafka Channel
Affects Versions: v1.6.0
 Environment: Flume 1.6.0+Kafka 0.9 
Reporter: Ping Wang
 Fix For: v1.8.0


Use Kafka Channel in the agent configuration:
tier1.sources = source1
tier1.channels = channel1
tier1.sinks = sink1
tier1.sources.source1.type = exec
tier1.sources.source1.command = /usr/bin/vmstat 1
tier1.sources.source1.channels = channel1
tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
tier1.channels.channel1.kafka.bootstrap.servers = a.b.c.d:6667
tier1.sinks.sink1.type = hdfs
tier1.sinks.sink1.hdfs.path = /tmp/kafka/channel
tier1.sinks.sink1.hdfs.rollInterval = 5
tier1.sinks.sink1.hdfs.rollSize = 0
tier1.sinks.sink1.hdfs.rollCount = 0
tier1.sinks.sink1.hdfs.fileType = DataStream
tier1.sinks.sink1.channel = channel1

Accidentally kaka.bootstrap.servers is not correct,  errors will be thrown out:
..
)] Waiting for channel: channel1 to start. Sleeping for 500 ms
2016-10-21 01:15:50,739 (conf-file-poller-0) [INFO - 
org.apache.flume.node.Application.startAllComponents(Application.java:161)] 
Waiting for channel: channel1 to start. Sleeping for 500 ms
2016-10-21 01:15:51,240 (conf-file-poller-0) [INFO - 
org.apache.flume.node.Application.startAllComponents(Application.java:161)] 
Waiting for channel: channel1 to start. Sleeping for 500 ms
2016-10-21 01:15:51,735 (lifecycleSupervisor-1-6) [INFO - 
org.apache.flume.channel.kafka.KafkaChannel.start(KafkaChannel.java:115)] 
Starting Kafka Channel: channel1
2016-10-21 01:15:51,737 (lifecycleSupervisor-1-6) [INFO - 
org.apache.kafka.common.config.AbstractConfig.logAll(AbstractConfig.java:165)] 
ProducerConfig values: 
compression.type = none
metric.reporters = []
metadata.max.age.ms = 30
metadata.fetch.timeout.ms = 6
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
bootstrap.servers = [a.b.c.d:6667]
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
buffer.memory = 33554432
timeout.ms = 3
key.serializer = class 
org.apache.kafka.common.serialization.StringSerializer
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.keystore.type = JKS
ssl.trustmanager.algorithm = PKIX
block.on.buffer.full = false
ssl.key.password = null
max.block.ms = 6
sasl.kerberos.min.time.before.relogin = 6
connections.max.idle.ms = 54
ssl.truststore.password = null
max.in.flight.requests.per.connection = 5
metrics.num.samples = 2
client.id = 
ssl.endpoint.identification.algorithm = null
ssl.protocol = TLS
request.timeout.ms = 3
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
acks = all
batch.size = 16384
ssl.keystore.location = null
receive.buffer.bytes = 32768
ssl.cipher.suites = null
ssl.truststore.type = JKS
security.protocol = PLAINTEXT
retries = 0
max.request.size = 1048576
value.serializer = class 
org.apache.kafka.common.serialization.ByteArraySerializer
ssl.truststore.location = null
ssl.keystore.password = null
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 3
partitioner.class = class 
org.apache.kafka.clients.producer.internals.DefaultPartitioner
send.buffer.bytes = 131072
linger.ms = 0

2016-10-21 01:15:51,742 (lifecycleSupervisor-1-6) [DEBUG - 
org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added sensor 
with name bufferpool-wait-time
2016-10-21 01:15:51,743 (lifecycleSupervisor-1-6) [DEBUG - 
org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added sensor 
with name buffer-exhausted-records
2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [INFO - 
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:613)] 
Closing the Kafka producer with timeoutMillis = 0 ms.
2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [DEBUG - 
org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:654)] 
The Kafka producer has closed.
2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [ERROR - 
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)]
 Unable to start org.apache.flume.channel.kafka.KafkaChannel{name: channel1} - 
Exception follows.
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at 

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Balázs Donát Bessenyei]

> On Oct. 18, 2016, 4:26 p.m., Attila Simon wrote:
> > Ahh now I see what is going on. BucketWriter instantiate a Clock class 
> > which is in turn used in the file name counter. The test first instantiates 
> > the BucketWriter - which initialize the filename counter -then overrides 
> > the Clock but since Clock is not used in BucketWriter directly - only via 
> > the file name counter at instantiation time so it won't be updated with the 
> > override - the test failes as it checks the file name.
> > So actually a single extra line into the `BucketWriter.setClock(Clock 
> > clock)` method would solve the issue:
> > ```
> >   void setClock(Clock clock) {
> > this.clock = clock;
> > this.fileExtensionCounter.set(clock.currentTimeMillis());
> >   }
> > ```
> > This way the test would update the relevant information - file name counter 
> > - which then in turn can be checked.
> > 
> > Could you please consider simplifying your change?

Thank you for the review!

Even though the change you have suggested does fix the flakiness of the tests 
(which is awesome!), I am not sure people would expect the caused side-effect 
in void setClock(Clock clock) -> fileExtensionCounter.

The change in its current form does eliminate private Clock clock which is 
"nice". However, it also eliminates void setClock(Clock clock) which helps 
avoiding violation of the monotony of fileExtensionCounter (which is assumed), 
thus it improves the integrity of BucketWriter (encapsulation).

Please, let me know your thoughts.


- Balázs Donát


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153104
---


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

[jira] [Comment Edited] (FLUME-3011) metrics messages does not change when spooling directory source exit abnormaly

[huyanping (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593941#comment-15593941
 ] 

huyanping edited comment on FLUME-3011 at 10/21/16 6:01 AM:


I found that all the StopTime fields of every source doest not change when the 
source exit abnormally. Maybe it it better for users that add a new field or 
just change the value of the StopTime field when the sources exit abnormally.


was (Author: jenner):
I found that all the StopTime fields of every type of sources doest not change 
when the source exit abnormally. Maybe it it better for users that add a new 
field or just change the value of the StopTime field when the sources exit 
abnormally.

> metrics messages does not change when spooling directory source exit abnormaly
> --
>
> Key: FLUME-3011
> URL: https://issues.apache.org/jira/browse/FLUME-3011
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Affects Versions: v1.6.0, v1.5.2
>Reporter: huyanping
>
> I found this error in my flume logs. The source_xxx has stopped but the 
> StopTime field of metrics message is not changed. 
> 21 Oct 2016 11:26:02,460 ERROR [pool-88871-thread-1] 
> (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:256) 
>  - FATAL: Spool Directory source source_xxx: { spoolDir: /data/flume/logs/l 
> }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure 
> Flume to continue processing.
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at 
> org.apache.flume.serialization.ResettableFileInputStream.(ResettableFileInputStream.java:108)
> at 
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.openFile(ReliableSpoolingFileEventReader.java:491)
> at 
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:459)
> at 
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:229)
> at 
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:227)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)  
> The metrics message:
> "SOURCE.source_xxx":{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"20660","AppendBatchReceivedCount":"20660","EventAcceptedCount":"165337","AppendReceivedCount":"0","StopTime":"0","StartTime":"1475502164604","EventReceivedCount":"165337","AppendAcceptedCount":"0"}
>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)