Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153104
---



Ahh now I see what is going on. BucketWriter instantiate a Clock class which is 
in turn used in the file name counter. The test first instantiates the 
BucketWriter - which initialize the filename counter -then overrides the Clock 
but since Clock is not used in BucketWriter directly - only via the file name 
counter at instantiation time so it won't be updated with the override - the 
test failes as it checks the file name.
So actually a single extra line into the `BucketWriter.setClock(Clock clock)` 
method would solve the issue:
```
  void setClock(Clock clock) {
this.clock = clock;
this.fileExtensionCounter.set(clock.currentTimeMillis());
  }
```
This way the test would update the relevant information - file name counter - 
which then in turn can be checked.

Could you please consider simplifying your change?

- Attila Simon


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: Review Request 52955: FLUME-2698 - Upgrade Jetty Version

[Tristan Stevens]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52955/
---

(Updated Oct. 18, 2016, 4:03 p.m.)


Review request for Flume.


Changes
---

Added tests, JMX and changed reference to released stable version of Jetty 
9.2.19.v20160908. This diff should now be stable and passes tests and 
checkstyle.


Repository: flume-git


Description (updated)
---

This patch upgrades the HTTPSource and HTTP Metrics server to Jetty 9.2 (N.B. 
9.3/9.4 depend on JDK8). It also adds the ability to further configure the 
Jetty server and also run in multi-threaded mode. Finally, registered Jetty 
with the MBeans server to allow fine-grained performance tuning/monitoring if 
required.


Diffs (updated)
-

  flume-ng-core/pom.xml 2262d94 
  
flume-ng-core/src/main/java/org/apache/flume/instrumentation/http/HTTPMetricsServer.java
 921a1f7 
  flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSource.java 
38bdfda 
  
flume-ng-core/src/main/java/org/apache/flume/source/http/HTTPSourceConfigurationConstants.java
 86caf7d 
  flume-ng-core/src/main/java/org/apache/flume/tools/FlumeBeanConfigurator.java 
PRE-CREATION 
  
flume-ng-core/src/main/java/org/apache/flume/tools/HTTPServerConstraintUtil.java
 479cfc4 
  flume-ng-core/src/main/java/org/apache/flume/tools/ImmutablePair.java 
PRE-CREATION 
  
flume-ng-core/src/test/java/org/apache/flume/instrumentation/http/TestHTTPMetricsServer.java
 09d419f 
  
flume-ng-core/src/test/java/org/apache/flume/source/http/FlumeHttpServletRequestWrapper.java
 475d92f 
  flume-ng-core/src/test/java/org/apache/flume/source/http/TestBLOBHandler.java 
f770d51 
  flume-ng-core/src/test/java/org/apache/flume/source/http/TestHTTPSource.java 
3ad8282 
  flume-ng-doc/sphinx/FlumeUserGuide.rst cd76bb3 
  flume-ng-legacy-sources/flume-avro-source/pom.xml 7cecde2 
  flume-ng-legacy-sources/flume-thrift-source/pom.xml 1ceeb5b 
  flume-ng-sources/flume-scribe-source/pom.xml 73641df 
  pom.xml f62c99a 

Diff: https://reviews.apache.org/r/52955/diff/


Testing (updated)
---

Passes existing unit tests, added specific tests for configurability aspects 
and also for JMX monitoring.


Thanks,

Tristan Stevens

[GitHub] flume pull request #72: Fix startup order for ExecSource

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flume/pull/72


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Moving Developer Section of cwiki to the git repository

[Balazs Donat Bessenyei]

Hi All,

As it's kind of difficult to get permissions in the wiki to edit pages
like https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
, I suggest moving contents to the git repository into files like
CONTRIBUTING.md, etc.

I'd be happy to create the files and move the current texts.

Please, let me know your thoughts.


Thank you,

Donat

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Balazs Donat Bessenyei]

Thanks for the suggestions, Lior.

I have just tagged the release and marked the version as released in JIRA.


Donat

On Tue, Oct 18, 2016 at 2:40 PM, Lior Zeno  wrote:
> Congratulations everyone! A big thank you for all of those who made this
> possible.
>
> Donat, if you could please tag the final release in the repo that would be
> great. Also, mark the version as released on Jira.
>
> On Tue, Oct 18, 2016 at 3:32 PM, Balazs Donat Bessenyei > wrote:
>
>> And thank you, Mike Percy for the mentoring and tremendous amounts of
>> assistance with the release!
>>
>> On Tue, Oct 18, 2016 at 12:46 PM, Balazs Donat Bessenyei
>>  wrote:
>> > Thank you all very much who participated and helped with the release!
>> >
>> >
>> > Donat
>> >
>> >
>> > On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy  wrote:
>> >> Woot! Congrats everyone!
>> >>
>> >> Thanks Donat for working so hard to get this version of Flume out the
>> door!
>> >>
>> >> Best,
>> >> Mike
>> >>
>> >>
>> >> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát <
>> bes...@apache.org>
>> >> wrote:
>> >>>
>> >>> The Apache Flume team is pleased to announce the release of Flume
>> >>> version 1.7.0.
>> >>>
>> >>> Flume is a distributed, reliable, and available service for efficiently
>> >>> collecting, aggregating, and moving large amounts of log data.
>> >>>
>> >>> This release can be downloaded from the Flume download page at:
>> >>> http://flume.apache.org/download.html
>> >>>
>> >>> The change log and documentation are available on the 1.7.0 release
>> page:
>> >>> http://flume.apache.org/releases/1.7.0.html
>> >>>
>> >>> Your help and feedback is more than welcome. For more information on
>> how
>> >>> to report problems and to get involved, visit the project website at
>> >>> http://flume.apache.org/
>> >>>
>> >>> The Apache Flume Team
>> >>
>> >>
>>

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Lior Zeno]

Congratulations everyone! A big thank you for all of those who made this
possible.

Donat, if you could please tag the final release in the repo that would be
great. Also, mark the version as released on Jira.

On Tue, Oct 18, 2016 at 3:32 PM, Balazs Donat Bessenyei  wrote:

> And thank you, Mike Percy for the mentoring and tremendous amounts of
> assistance with the release!
>
> On Tue, Oct 18, 2016 at 12:46 PM, Balazs Donat Bessenyei
>  wrote:
> > Thank you all very much who participated and helped with the release!
> >
> >
> > Donat
> >
> >
> > On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy  wrote:
> >> Woot! Congrats everyone!
> >>
> >> Thanks Donat for working so hard to get this version of Flume out the
> door!
> >>
> >> Best,
> >> Mike
> >>
> >>
> >> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát <
> bes...@apache.org>
> >> wrote:
> >>>
> >>> The Apache Flume team is pleased to announce the release of Flume
> >>> version 1.7.0.
> >>>
> >>> Flume is a distributed, reliable, and available service for efficiently
> >>> collecting, aggregating, and moving large amounts of log data.
> >>>
> >>> This release can be downloaded from the Flume download page at:
> >>> http://flume.apache.org/download.html
> >>>
> >>> The change log and documentation are available on the 1.7.0 release
> page:
> >>> http://flume.apache.org/releases/1.7.0.html
> >>>
> >>> Your help and feedback is more than welcome. For more information on
> how
> >>> to report problems and to get involved, visit the project website at
> >>> http://flume.apache.org/
> >>>
> >>> The Apache Flume Team
> >>
> >>
>

Re: Review Request 52510: Some tests in TestBucketWriter are flaky

[Attila Simon]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52510/#review153076
---



Could you please give us some details about the idea of this change? Ie some 
high level description about what was the problem and how you solved it.

- Attila Simon


On Oct. 4, 2016, 11:11 a.m., Balázs Donát Bessenyei wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52510/
> ---
> 
> (Updated Oct. 4, 2016, 11:11 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-3002
> https://issues.apache.org/jira/browse/FLUME-3002
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> testFileSuffixNotGiven (and probably a few other tests) are flaky
> 
> 
> Diffs
> -
> 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
>  b096410 
>   
> c/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
>  742deb0 
> 
> Diff: https://reviews.apache.org/r/52510/diff/
> 
> 
> Testing
> ---
> 
> mvn clean install runs successfully in flume-ng-sinks
> 
> 
> Thanks,
> 
> Balázs Donát Bessenyei
> 
>

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Balazs Donat Bessenyei]

And thank you, Mike Percy for the mentoring and tremendous amounts of
assistance with the release!

On Tue, Oct 18, 2016 at 12:46 PM, Balazs Donat Bessenyei
 wrote:
> Thank you all very much who participated and helped with the release!
>
>
> Donat
>
>
> On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy  wrote:
>> Woot! Congrats everyone!
>>
>> Thanks Donat for working so hard to get this version of Flume out the door!
>>
>> Best,
>> Mike
>>
>>
>> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát 
>> wrote:
>>>
>>> The Apache Flume team is pleased to announce the release of Flume
>>> version 1.7.0.
>>>
>>> Flume is a distributed, reliable, and available service for efficiently
>>> collecting, aggregating, and moving large amounts of log data.
>>>
>>> This release can be downloaded from the Flume download page at:
>>> http://flume.apache.org/download.html
>>>
>>> The change log and documentation are available on the 1.7.0 release page:
>>> http://flume.apache.org/releases/1.7.0.html
>>>
>>> Your help and feedback is more than welcome. For more information on how
>>> to report problems and to get involved, visit the project website at
>>> http://flume.apache.org/
>>>
>>> The Apache Flume Team
>>
>>

[jira] [Commented] (FLUME-2989) Kafka Channel metrics missing eventTakeAttemptCount and eventPutAttemptCount

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLUME-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585341#comment-15585341
 ] 

Bessenyei Balázs Donát commented on FLUME-2989:
---

[~umesh9...@gmail.com]: thank you for the patch!

I have left some comments on the reviewboard request.
Please, let me know about your thoughts.

> Kafka Channel metrics missing eventTakeAttemptCount and eventPutAttemptCount
> 
>
> Key: FLUME-2989
> URL: https://issues.apache.org/jira/browse/FLUME-2989
> Project: Flume
>  Issue Type: Bug
>  Components: Kafka Channel
>Affects Versions: v1.5.0, v1.6.0, v1.5.1
>Reporter: Denes Arvay
>Assignee: Umesh Chaudhary
>Priority: Minor
> Attachments: FLUME-2989-0.patch, FLUME-2989-1.patch
>
>
> {{eventTakeAttemptCount}} and {{eventPutAttemptCount}} counters don't get 
> incremented in Kafka Channel



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Balazs Donat Bessenyei]

Thank you all very much who participated and helped with the release!


Donat


On Tue, Oct 18, 2016 at 12:37 PM, Mike Percy  wrote:
> Woot! Congrats everyone!
>
> Thanks Donat for working so hard to get this version of Flume out the door!
>
> Best,
> Mike
>
>
> On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát 
> wrote:
>>
>> The Apache Flume team is pleased to announce the release of Flume
>> version 1.7.0.
>>
>> Flume is a distributed, reliable, and available service for efficiently
>> collecting, aggregating, and moving large amounts of log data.
>>
>> This release can be downloaded from the Flume download page at:
>> http://flume.apache.org/download.html
>>
>> The change log and documentation are available on the 1.7.0 release page:
>> http://flume.apache.org/releases/1.7.0.html
>>
>> Your help and feedback is more than welcome. For more information on how
>> to report problems and to get involved, visit the project website at
>> http://flume.apache.org/
>>
>> The Apache Flume Team
>
>

[jira] [Commented] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel

[Attila Simon (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585137#comment-15585137
 ] 

Attila Simon commented on FLUME-2997:
-

I think I missed some description of this change so let me provide it now: 
Originally this test case aimed to verify behaviour about having multiple 
mocked sources putting events on the same SpillableMemoryChannel while multiple 
mocked sinks drains those events. Given the mock implementation didn't do retry 
if sources are faster than the sinks the channel will get full. Since full 
channel doesn't accept new events the mocked sources fail their task by raising 
a runtime exception. This edge case should be avoided thus the increased 
capacity (which makes sure that channel cannot be full). The "channel is full" 
edge case is already covered by another test. 

Those System.out.println-s were replaced by asserts to make the function to 
became a test. It wasn't testing anything before. It only ran to the end 
without checking anything. (The flakiness was caused by a 
ChannelFullException). I guess the original author used those 
System.out.println-s for manual testing (purely looking at the numbers). This 
change also made that manual step automatic.

> Fix flaky junit test in SpillableMemoryChannel
> --
>
> Key: FLUME-2997
> URL: https://issues.apache.org/jira/browse/FLUME-2997
> Project: Flume
>  Issue Type: Test
>Affects Versions: v1.7.0
>Reporter: Attila Simon
>Assignee: Attila Simon
> Fix For: v1.8.0
>
> Attachments: FLUME-2997-1.patch, FLUME-2997.patch
>
>
> testParallelSingleSourceAndSink sometimes trigger an edge case scenario if 
> sinks are slower than sources. In such situations the channel can get full 
> thus uncaught ChannelFullException breaks the test. Since 
> testCapacityWithOverflow was designed to cover such edge-case scenario 
> already we can safely fix the test by increasing the channel capacity to make 
> sure it won't get full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] Apache Flume 1.7.0 released

[Mike Percy]

Woot! Congrats everyone!

Thanks Donat for working so hard to get this version of Flume out the door!

Best,
Mike

On Tue, Oct 18, 2016 at 10:09 AM, Bessenyei Balázs Donát 
wrote:

> The Apache Flume team is pleased to announce the release of Flume
> version 1.7.0.
>
> Flume is a distributed, reliable, and available service for efficiently
> collecting, aggregating, and moving large amounts of log data.
>
> This release can be downloaded from the Flume download page at:
> http://flume.apache.org/download.html
>
> The change log and documentation are available on the 1.7.0 release page:
> http://flume.apache.org/releases/1.7.0.html
>
> Your help and feedback is more than welcome. For more information on how
> to report problems and to get involved, visit the project website at
> http://flume.apache.org/
>
> The Apache Flume Team
>

Migrate to Log4j 2

[Lior Zeno]

Hi All,

Log4j (v1) has been EOL'ed over a year now (
https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces)
and is no longer officially supported.

I propose we migrate to Log4j 2. We can begin with using the Log4j 1.x
bridge, and then incrementally move the whole codebase.

Since this is a breaking change, I think we should schedule this to Flume
2.0.0.

Thoughts?

Thanks

[jira] [Resolved] (FLUME-2924) Flume 1.7.0 release

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLUME-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bessenyei Balázs Donát resolved FLUME-2924.
---
Resolution: Fixed
  Assignee: Bessenyei Balázs Donát

> Flume 1.7.0 release
> ---
>
> Key: FLUME-2924
> URL: https://issues.apache.org/jira/browse/FLUME-2924
> Project: Flume
>  Issue Type: Umbrella
>Affects Versions: v1.7.0
>Reporter: Lior Zeno
>Assignee: Bessenyei Balázs Donát
> Fix For: v1.7.0
>
>
> Release 1.7.0 umbrella issue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLUME-3008) Announce the release for 1.7

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLUME-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bessenyei Balázs Donát resolved FLUME-3008.
---
Resolution: Fixed

> Announce the release for 1.7
> 
>
> Key: FLUME-3008
> URL: https://issues.apache.org/jira/browse/FLUME-3008
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Bessenyei Balázs Donát
>Assignee: Bessenyei Balázs Donát
> Fix For: v1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52550: FLUME-2989: Review Request

[Balázs Donát Bessenyei]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52550/#review153062
---




flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
 (line 769)


Do you think this Thread.sleep could be avoided?

If not, do you think you can change it in a way that it will be 
not-so-prone to flakiness?
(Maybe something like at https://reviews.apache.org/r/49025/diff/5#3 )


Thank you for the patch!

Can you please rebase it on trunk so that it applies cleanly?

Also, please reformat the code to pass checkstyle checks.

- Balázs Donát Bessenyei


On Oct. 10, 2016, 12:44 p.m., Umesh Chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52550/
> ---
> 
> (Updated Oct. 10, 2016, 12:44 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> Added appropriate function calls to capture eventTakeAttemptCount and 
> eventPutAttemptCount in KafkaChannel
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
>  66b553a 
>   
> flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
>  57c0b28 
> 
> Diff: https://reviews.apache.org/r/52550/diff/
> 
> 
> Testing
> ---
> 
> Testing Done.
> 
> 
> Thanks,
> 
> Umesh Chaudhary
> 
>

[ANNOUNCE] Apache Flume 1.7.0 released

[Bessenyei Balázs Donát]

The Apache Flume team is pleased to announce the release of Flume
version 1.7.0.

Flume is a distributed, reliable, and available service for efficiently
collecting, aggregating, and moving large amounts of log data.

This release can be downloaded from the Flume download page at:
http://flume.apache.org/download.html

The change log and documentation are available on the 1.7.0 release page:
http://flume.apache.org/releases/1.7.0.html

Your help and feedback is more than welcome. For more information on how
to report problems and to get involved, visit the project website at
http://flume.apache.org/

The Apache Flume Team