[jira] [Commented] (FLUME-2938) JDBC Source

2016-07-01 Thread Lior Zeno (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359206#comment-15359206
 ] 

Lior Zeno commented on FLUME-2938:
--

I'll start by stating that I'm not a Sqoop expert.

It's true that this source will have to pull for new events, unlike other 
sources that get new events by push. However, I'm still not sure how can Sqoop 
provide the functionality that Flume provides. Flume offers much more target 
options, simple transformations that do not require a MapReduce job, and so on.

I believe that Flume should provide this functionality. This ticket does not 
compete with Sqoop, it is not intended for batch computations (transformations) 
on data from relational databases, but a simple mechanism to transfer data from 
JDBC to any other source in small batches.



> JDBC Source
> ---
>
> Key: FLUME-2938
> URL: https://issues.apache.org/jira/browse/FLUME-2938
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Affects Versions: v1.8.0
>Reporter: Lior Zeno
> Fix For: v1.8.0
>
>
> The idea is to allow migrating data from SQL stores to NoSQL stores or HDFS 
> for archiving purposes.
> This source will get a statement to execute and a scheduling policy. It will 
> be able to fetch timestamped data by performing range queries on a 
> configurable field (this can fetch data with incremental id as well). For 
> fault-tolerance, the last fetched value can be checkpointed to a file.
> Dealing with large datasets can be done via the fetch_size parameter. (Ref: 
> https://docs.oracle.com/cd/A87860_01/doc/java.817/a83724/resltse5.htm)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2938) JDBC Source

2016-07-01 Thread Attila Simon (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359170#comment-15359170
 ] 

Attila Simon commented on FLUME-2938:
-

As it turned out I'm a big fan of the "do one thing well" design. My concern is 
duplicating work should be avoided so would be good to know what would be the 
additional functionality. 

On the other hand flume is for streaming data and the source you mentioned 
should have a scheduler. Is this really a functionality flume should provide? 
Or only a little tweak in Sqoop is required if there is any at all.

> JDBC Source
> ---
>
> Key: FLUME-2938
> URL: https://issues.apache.org/jira/browse/FLUME-2938
> Project: Flume
>  Issue Type: New Feature
>  Components: Sinks+Sources
>Affects Versions: v1.8.0
>Reporter: Lior Zeno
> Fix For: v1.8.0
>
>
> The idea is to allow migrating data from SQL stores to NoSQL stores or HDFS 
> for archiving purposes.
> This source will get a statement to execute and a scheduling policy. It will 
> be able to fetch timestamped data by performing range queries on a 
> configurable field (this can fetch data with incremental id as well). For 
> fault-tolerance, the last fetched value can be checkpointed to a file.
> Dealing with large datasets can be done via the fetch_size parameter. (Ref: 
> https://docs.oracle.com/cd/A87860_01/doc/java.817/a83724/resltse5.htm)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49458: FLUME-2921 Support Elasticsearch 2.0+

2016-07-01 Thread Attila Simon 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49458/#review140360
---




flume-ng-channels/flume-spillable-memory-channel/pom.xml (lines 81 - 85)


is this change related to elastic search?



flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml (line 13)


nit: changed indention (what is the guideline for pom.xml?)



flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java
 (lines 142 - 146)


serverAddresses[i] won't be initialized if host[i] was not found. wouldn't 
be better not keeping that entry from the returned array?



pom.xml (lines 874 - 879)


is this change related to elastic search?


After applying the this diff on trunk I got build failure
mvn clean install -DskipTests 
creates flume-ng-sinks/flume-ng-elasticsearch-sink/dependency-reduced-pom.xml 
which is then correctly caught by RAT check. Where is this file coming from? 
Should it be added to an ignore list?

mvn -pl flume-ng-sinks/flume-ng-elasticsearch-sink/  test  ===> passes

- Attila Simon


On July 1, 2016, 2:40 p.m., Lior Zeno wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49458/
> ---
> 
> (Updated July 1, 2016, 2:40 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2921
> https://issues.apache.org/jira/browse/FLUME-2921
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> This patch adds the support for Elasticsearch version 2.0+. The version I 
> used is 2.3.3, which is the latest stable release. This also required 
> upgrading guava.
> This patch does not fix any known issues with this sink, its only purpose is 
> to support current versions of elasticsearch.
> 
> 
> Diffs
> -
> 
>   flume-ng-channels/flume-spillable-memory-channel/pom.xml 60e49d2 
>   flume-ng-core/pom.xml 41586dd 
>   flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml c372c0b 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
>  83c3ffd 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java
>  3638368 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java
>  2cf365e 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
>  f9272fa 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchDynamicSerializer.java
>  d4e4654 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java
>  8022111 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchLogStashEventSerializer.java
>  ab9587d 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
>  a58f344 
>   pom.xml 85c0dc8 
> 
> Diff: https://reviews.apache.org/r/49458/diff/
> 
> 
> Testing
> ---
> 
> I made sure that all unit tests (due to guava upgrade) pass successfully. The 
> known flaky tests may not pass, though.
> 
> 
> Thanks,
> 
> Lior Zeno
> 
>

Re: Review Request 49458: FLUME-2921 Support Elasticsearch 2.0+

2016-07-01 Thread Mike Percy 


> On June 30, 2016, 6:48 p.m., Mike Percy wrote:
> > pom.xml, line 872
> > 
> >
> > Is this really the minimum Guava version required? Flume has to work 
> > with Hadoop and a lot of other components so this huge version jump scares 
> > me.
> 
> Lior Zeno wrote:
> The minimum Guava version requires is 18.0, due to 
> MoreExecutors.directExecutor. In the next elasticsearch release, Guava won't 
> be a dependency anymore.
> Also, I will submit a new patch guava 18 and a newer diff due to the 
> latest commits.
> 
> Lior Zeno wrote:
> I shaded guava in flume-ng-elasticsearch-sink. I agree that bumping 
> guava's version is problematic, we might need to consider removing this 
> dependency now that we are working with java 1.7.

Great, thanks for doing that. I won't be around for several days due to the 
July 4 holiday. Maybe another dev has time to take a pass at this review in the 
meantime.


- Mike


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49458/#review140215
---


On July 1, 2016, 2:40 p.m., Lior Zeno wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49458/
> ---
> 
> (Updated July 1, 2016, 2:40 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2921
> https://issues.apache.org/jira/browse/FLUME-2921
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> This patch adds the support for Elasticsearch version 2.0+. The version I 
> used is 2.3.3, which is the latest stable release. This also required 
> upgrading guava.
> This patch does not fix any known issues with this sink, its only purpose is 
> to support current versions of elasticsearch.
> 
> 
> Diffs
> -
> 
>   flume-ng-channels/flume-spillable-memory-channel/pom.xml 60e49d2 
>   flume-ng-core/pom.xml 41586dd 
>   flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml c372c0b 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
>  83c3ffd 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java
>  3638368 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java
>  2cf365e 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
>  f9272fa 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchDynamicSerializer.java
>  d4e4654 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java
>  8022111 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchLogStashEventSerializer.java
>  ab9587d 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
>  a58f344 
>   pom.xml 85c0dc8 
> 
> Diff: https://reviews.apache.org/r/49458/diff/
> 
> 
> Testing
> ---
> 
> I made sure that all unit tests (due to guava upgrade) pass successfully. The 
> known flaky tests may not pass, though.
> 
> 
> Thanks,
> 
> Lior Zeno
> 
>

Re: Review Request 49458: FLUME-2921 Support Elasticsearch 2.0+

2016-07-01 Thread Lior Zeno 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49458/
---

(Updated July 1, 2016, 2:40 p.m.)


Review request for Flume.


Bugs: FLUME-2921
https://issues.apache.org/jira/browse/FLUME-2921


Repository: flume-git


Description
---

This patch adds the support for Elasticsearch version 2.0+. The version I used 
is 2.3.3, which is the latest stable release. This also required upgrading 
guava.
This patch does not fix any known issues with this sink, its only purpose is to 
support current versions of elasticsearch.


Diffs (updated)
-

  flume-ng-channels/flume-spillable-memory-channel/pom.xml 60e49d2 
  flume-ng-core/pom.xml 41586dd 
  flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml c372c0b 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
 83c3ffd 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java
 3638368 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java
 2cf365e 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
 f9272fa 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchDynamicSerializer.java
 d4e4654 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java
 8022111 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchLogStashEventSerializer.java
 ab9587d 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
 a58f344 
  pom.xml 85c0dc8 

Diff: https://reviews.apache.org/r/49458/diff/


Testing
---

I made sure that all unit tests (due to guava upgrade) pass successfully. The 
known flaky tests may not pass, though.


Thanks,

Lior Zeno

Re: Review Request 49458: FLUME-2921 Support Elasticsearch 2.0+

2016-07-01 Thread Lior Zeno 


> On June 30, 2016, 6:48 p.m., Mike Percy wrote:
> > pom.xml, line 872
> > 
> >
> > Is this really the minimum Guava version required? Flume has to work 
> > with Hadoop and a lot of other components so this huge version jump scares 
> > me.
> 
> Lior Zeno wrote:
> The minimum Guava version requires is 18.0, due to 
> MoreExecutors.directExecutor. In the next elasticsearch release, Guava won't 
> be a dependency anymore.
> Also, I will submit a new patch guava 18 and a newer diff due to the 
> latest commits.

I shaded guava in flume-ng-elasticsearch-sink. I agree that bumping guava's 
version is problematic, we might need to consider removing this dependency now 
that we are working with java 1.7.


- Lior


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49458/#review140215
---


On June 30, 2016, 7:05 p.m., Lior Zeno wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49458/
> ---
> 
> (Updated June 30, 2016, 7:05 p.m.)
> 
> 
> Review request for Flume.
> 
> 
> Bugs: FLUME-2921
> https://issues.apache.org/jira/browse/FLUME-2921
> 
> 
> Repository: flume-git
> 
> 
> Description
> ---
> 
> This patch adds the support for Elasticsearch version 2.0+. The version I 
> used is 2.3.3, which is the latest stable release. This also required 
> upgrading guava.
> This patch does not fix any known issues with this sink, its only purpose is 
> to support current versions of elasticsearch.
> 
> 
> Diffs
> -
> 
>   flume-ng-channels/flume-spillable-memory-channel/pom.xml 60e49d2 
>   flume-ng-core/pom.xml 41586dd 
>   flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java 
> de727f6 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
>  83c3ffd 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java
>  3638368 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java
>  2cf365e 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
>  f9272fa 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchDynamicSerializer.java
>  d4e4654 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java
>  8022111 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchLogStashEventSerializer.java
>  ab9587d 
>   
> flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
>  a58f344 
>   pom.xml 85c0dc8 
> 
> Diff: https://reviews.apache.org/r/49458/diff/
> 
> 
> Testing
> ---
> 
> I made sure that all unit tests (due to guava upgrade) pass successfully. The 
> known flaky tests may not pass, though.
> 
> 
> Thanks,
> 
> Lior Zeno
> 
>

Re: Review Request 49458: FLUME-2921 Support Elasticsearch 2.0+

2016-07-01 Thread Lior Zeno 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49458/
---

(Updated July 1, 2016, 2:30 p.m.)


Review request for Flume.


Bugs: FLUME-2921
https://issues.apache.org/jira/browse/FLUME-2921


Repository: flume-git


Description
---

This patch adds the support for Elasticsearch version 2.0+. The version I used 
is 2.3.3, which is the latest stable release. This also required upgrading 
guava.
This patch does not fix any known issues with this sink, its only purpose is to 
support current versions of elasticsearch.


Diffs (updated)
-

  flume-ng-channels/flume-spillable-memory-channel/pom.xml 60e49d2 
  flume-ng-core/pom.xml 41586dd 
  flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml c372c0b 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
 83c3ffd 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java
 3638368 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java
 2cf365e 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
 f9272fa 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchDynamicSerializer.java
 d4e4654 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java
 8022111 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchLogStashEventSerializer.java
 ab9587d 
  
flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java
 a58f344 
  pom.xml 85c0dc8 

Diff: https://reviews.apache.org/r/49458/diff/


Testing
---

I made sure that all unit tests (due to guava upgrade) pass successfully. The 
known flaky tests may not pass, though.


Thanks,

Lior Zeno

[jira] [Commented] (FLUME-2939) Upgrade recursive SpoolDir to use Java7 features

2016-07-01 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/FLUME-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358901#comment-15358901
 ] 

Bessenyei Balázs Donát commented on FLUME-2939:
---

Patch submitted for review, please see https://reviews.apache.org/r/49506/

> Upgrade recursive SpoolDir to use Java7 features
> 
>
> Key: FLUME-2939
> URL: https://issues.apache.org/jira/browse/FLUME-2939
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Reporter: Bessenyei Balázs Donát
>Assignee: Bessenyei Balázs Donát
>Priority: Minor
> Fix For: v1.7.0
>
>
> FLUME-1899 was committed without the DirectoryStream API
> This ticket is for "upgrading" the patch provided to use Java 7 features as 
> the Java version for Flume has been recently bumped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2725) HDFS Sink does not use configured timezone for rounding

2016-07-01 Thread Denes Arvay (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358826#comment-15358826
 ] 

Denes Arvay commented on FLUME-2725:


I've attached a patch to use the configured time zone for rounding but I'm a 
bit concerned about this change as it might break stuff at users who already 
adapted to the current behavior. Shouldn't we put this behind a configuration 
parameter, wdyt?

> HDFS Sink does not use configured timezone for rounding
> ---
>
> Key: FLUME-2725
> URL: https://issues.apache.org/jira/browse/FLUME-2725
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Reporter: Eric Czech
>Assignee: Denes Arvay
>Priority: Minor
> Attachments: FLUME-2725.patch
>
>
> When a BucketPath used by an HDFS sink is configured to run with some 
> roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by 
> BucketPath does not actually round the date correctly.
> That function calls TimestampRoundDownUtil which creates a Calendar instance 
> using the *local* timezone to truncate a unix timestamp rather than the 
> TimeZone that the sink was configured to convert dates to paths with (and 
> that timezone is already available in the BucketPath class but it just isn't 
> passed to TimestampRoundDownUtil).
> The net effect of this is that if a flume jvm is running on a system with an 
> EST clock while trying to write, say, 6 hour directories in UTC time, the 
> directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, 
> 18 like you would expect.
> I found a workaround for this by passing 
> "-Duser.timezone=" as a system property, but I wanted to 
> create a ticket for this since it seems like it would be very minimal effort 
> to carry that configured timezone down into the rounding utility as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-2725) HDFS Sink does not use configured timezone for rounding

2016-07-01 Thread Denes Arvay (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denes Arvay updated FLUME-2725:
---
Attachment: FLUME-2725.patch

> HDFS Sink does not use configured timezone for rounding
> ---
>
> Key: FLUME-2725
> URL: https://issues.apache.org/jira/browse/FLUME-2725
> Project: Flume
>  Issue Type: Bug
>  Components: Sinks+Sources
>Reporter: Eric Czech
>Assignee: Denes Arvay
>Priority: Minor
> Attachments: FLUME-2725.patch
>
>
> When a BucketPath used by an HDFS sink is configured to run with some 
> roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by 
> BucketPath does not actually round the date correctly.
> That function calls TimestampRoundDownUtil which creates a Calendar instance 
> using the *local* timezone to truncate a unix timestamp rather than the 
> TimeZone that the sink was configured to convert dates to paths with (and 
> that timezone is already available in the BucketPath class but it just isn't 
> passed to TimestampRoundDownUtil).
> The net effect of this is that if a flume jvm is running on a system with an 
> EST clock while trying to write, say, 6 hour directories in UTC time, the 
> directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, 
> 18 like you would expect.
> I found a workaround for this by passing 
> "-Duser.timezone=" as a system property, but I wanted to 
> create a ticket for this since it seems like it would be very minimal effort 
> to carry that configured timezone down into the rounding utility as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 49453: Patch for FLUME-2725

2016-07-01 Thread Denes Arvay 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49453/
---

Review request for Flume, Balázs Donát Bessenyei and Attila Simon.


Summary (updated)
-

Patch for FLUME-2725


Bugs: FLUME-2725
https://issues.apache.org/jira/browse/FLUME-2725


Repository: flume-git


Description (updated)
---

Patch for FLUME-2725 - HDFS Sink does not use configured timezone for rounding


Diffs (updated)
-

  flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 
b2fe3f0 
  
flume-ng-core/src/main/java/org/apache/flume/tools/TimestampRoundDownUtil.java 
daa9606 
  
flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java
 21b972b 
  
flume-ng-core/src/test/java/org/apache/flume/tools/TestTimestampRoundDownUtil.java
 cc7eac0 

Diff: https://reviews.apache.org/r/49453/diff/


Testing (updated)
---

`org.apache.flume.formatter.output.TestBucketPath` and 
`org.apache.flume.tools.TestTimestampRoundDownUtil` were extended with new 
methods testing with `TimeZone`. Existing and new tests pass.


Thanks,

Denes Arvay