[jira] [Commented] (FLUME-2889) Fixes to DateTime computations

2016-02-25 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168397#comment-15168397
 ] 

Hudson commented on FLUME-2889:
---

FAILURE: Integrated in Flume-trunk-hbase-1 #151 (See 
[https://builds.apache.org/job/Flume-trunk-hbase-1/151/])
FLUME-2889. Fixes to DateTime computations (roshan: 
[http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ffb52b9e656df51e5e6881cfc8ed851a89cdc2f1])
* 
flume-ng-core/src/test/java/org/apache/flume/serialization/SyslogAvroEventSerializer.java
* flume-ng-core/src/main/java/org/apache/flume/source/SyslogParser.java


> Fixes to  DateTime computations
> ---
>
> Key: FLUME-2889
> URL: https://issues.apache.org/jira/browse/FLUME-2889
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.6.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Fix For: v1.7.0
>
> Attachments: FLUME-2889.patch
>
>
> date.withYear(year+1)  can lead to incorrect date calculations .. for example 
> if  the date is Feb 29th.   need to use date.plusYears(1) instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Flume-trunk-hbase-1 #151

2016-02-25 Thread Apache Jenkins Server 
See 

Changes:

[roshan] FLUME-2889. Fixes to DateTime computations

--
[...truncated 6320 lines...]
java.lang.Exception: Failed to locate tar-ball distribution. Please specify 
explicitly via system property: flume.dist.tarball
at 
org.apache.flume.test.util.StagedInstall.(StagedInstall.java:222)
at 
org.apache.flume.test.util.StagedInstall.getInstance(StagedInstall.java:78)
at 
org.apache.flume.test.agent.TestRpcClient.setUp(TestRpcClient.java:36)

testRpcClient(org.apache.flume.test.agent.TestRpcClient)  Time elapsed: 0.172 
sec  <<< ERROR!
java.lang.Exception: Failed to locate tar-ball distribution. Please specify 
explicitly via system property: flume.dist.tarball
at 
org.apache.flume.test.util.StagedInstall.(StagedInstall.java:222)
at 
org.apache.flume.test.util.StagedInstall.getInstance(StagedInstall.java:78)
at 
org.apache.flume.test.agent.TestRpcClient.tearDown(TestRpcClient.java:42)


Results :

Tests in error: 
  TestSpooldirSource.setup:59 »  Failed to locate tar-ball distribution. Please 
...
  TestSpooldirSource.teardown:111 »  Failed to locate tar-ball distribution. 
Ple...
  TestRpcClientCommunicationFailure.testFailure:61 »  Failed to locate tar-ball 
...
  TestFileChannel.testInOut:125 »  Failed to locate tar-ball distribution. 
Pleas...
  TestFileChannel.tearDown:103 »  Failed to locate tar-ball distribution. 
Please...
  TestSyslogSource.testKeepFields:72 »  Failed to locate tar-ball distribution. 
...
  TestSyslogSource.tearDown:63 » NullPointer
  TestSyslogSource.testRemoveFields:82 »  Failed to locate tar-ball 
distribution...
  TestSyslogSource.tearDown:63 » NullPointer
  TestSyslogSource.testKeepTimestampAndHostname:92 »  Failed to locate tar-ball 
...
  TestSyslogSource.tearDown:63 » NullPointer
  TestSyslogSource.testKeepFields:72 »  Failed to locate tar-ball distribution. 
...
  TestSyslogSource.tearDown:63 » NullPointer
  TestSyslogSource.testRemoveFields:82 »  Failed to locate tar-ball 
distribution...
  TestSyslogSource.tearDown:63 » NullPointer
  TestSyslogSource.testKeepTimestampAndHostname:92 »  Failed to locate tar-ball 
...
  TestSyslogSource.tearDown:63 » NullPointer
  TestRpcClient.setUp:36 »  Failed to locate tar-ball distribution. Please 
speci...
  TestRpcClient.tearDown:42 »  Failed to locate tar-ball distribution. Please 
sp...

Tests run: 19, Failures: 0, Errors: 19, Skipped: 0

[ERROR] There are test failures.

Please refer to 

 for the individual test results.
[JENKINS] Recording test results
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ flume-ng-tests ---
[INFO] Building jar: 

[INFO] 
[INFO] --- apache-rat-plugin:0.7:check (verify.rat) @ flume-ng-tests ---
[INFO] Exclude: **/.idea/
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/nb-configuration.xml
[INFO] Exclude: .git/
[INFO] Exclude: patchprocess/
[INFO] Exclude: .gitignore
[INFO] Exclude: .repository/
[INFO] Exclude: **/*.diff
[INFO] Exclude: **/*.patch
[INFO] Exclude: **/*.avsc
[INFO] Exclude: **/*.avro
[INFO] Exclude: **/docs/**
[INFO] Exclude: **/test/resources/**
[INFO] Exclude: **/.settings/*
[INFO] Exclude: **/.classpath
[INFO] Exclude: **/.project
[INFO] Exclude: **/target/**
[INFO] Exclude: **/derby.log
[INFO] Exclude: **/metastore_db/
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Flume .. SUCCESS [13.448s]
[INFO] Flume NG SDK .. SUCCESS [1:55.606s]
[INFO] Flume NG Configuration  SUCCESS [5.430s]
[INFO] Flume Auth  SUCCESS [40.500s]
[INFO] Flume NG Core . SUCCESS [7:44.833s]
[INFO] Flume NG Sinks  SUCCESS [0.211s]
[INFO] Flume NG HDFS Sink  SUCCESS [2:49.838s]
[INFO] Flume NG IRC Sink . SUCCESS [4.282s]
[INFO] Flume NG Channels . SUCCESS [0.200s]
[INFO] Flume NG JDBC channel . SUCCESS [2:06.476s]
[INFO] Flume NG file-based channel ... SUCCESS [20:26.104s]
[INFO] Flume NG Spillable Memory channel . SUCCESS [5:07.306s]
[INFO] Flume NG Node . SUCCESS [31.125s]
[INFO] Flume NG Embedded Agent ... SUCCESS [27.831s]
[INFO] Flume NG HBase Sink ... SUCCESS [5:36.099s]
[INFO] Flume NG ElasticSearch Sink ... SUCCESS [1:23.042s]
[INFO] Flume NG Morphline Solr Sink .. SUCCESS [2:01

[jira] [Updated] (FLUME-2889) Fixes to DateTime computations

2016-02-25 Thread Roshan Naik (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roshan Naik updated FLUME-2889:
---
Fix Version/s: v1.7.0

> Fixes to  DateTime computations
> ---
>
> Key: FLUME-2889
> URL: https://issues.apache.org/jira/browse/FLUME-2889
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.6.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Fix For: v1.7.0
>
> Attachments: FLUME-2889.patch
>
>
> date.withYear(year+1)  can lead to incorrect date calculations .. for example 
> if  the date is Feb 29th.   need to use date.plusYears(1) instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2889) Fixes to DateTime computations

2016-02-25 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168249#comment-15168249
 ] 

ASF subversion and git services commented on FLUME-2889:


Commit ffb52b9e656df51e5e6881cfc8ed851a89cdc2f1 in flume's branch 
refs/heads/trunk from [~roshan_naik]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ffb52b9 ]

FLUME-2889. Fixes to DateTime computations

(Roshan Naik via Hari Shreedharan)


> Fixes to  DateTime computations
> ---
>
> Key: FLUME-2889
> URL: https://issues.apache.org/jira/browse/FLUME-2889
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.6.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: FLUME-2889.patch
>
>
> date.withYear(year+1)  can lead to incorrect date calculations .. for example 
> if  the date is Feb 29th.   need to use date.plusYears(1) instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2889) Fixes to DateTime computations

2016-02-25 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168248#comment-15168248
 ] 

ASF subversion and git services commented on FLUME-2889:


Commit fe24607f42e2684128fbacdd73fa7489966aecc5 in flume's branch 
refs/heads/flume-1.7 from [~roshan_naik]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=fe24607 ]

FLUME-2889. Fixes to DateTime computations

(Roshan Naik via Hari Shreedharan)


> Fixes to  DateTime computations
> ---
>
> Key: FLUME-2889
> URL: https://issues.apache.org/jira/browse/FLUME-2889
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.6.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: FLUME-2889.patch
>
>
> date.withYear(year+1)  can lead to incorrect date calculations .. for example 
> if  the date is Feb 29th.   need to use date.plusYears(1) instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2889) Fixes to DateTime computations

2016-02-25 Thread Hari Shreedharan (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168114#comment-15168114
 ] 

Hari Shreedharan commented on FLUME-2889:
-

+1. LGTM. 

[~roshan_naik] Please go ahead and commit it, else I will commit it 
today/tomorrow

> Fixes to  DateTime computations
> ---
>
> Key: FLUME-2889
> URL: https://issues.apache.org/jira/browse/FLUME-2889
> Project: Flume
>  Issue Type: Bug
>Affects Versions: v1.6.0
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: FLUME-2889.patch
>
>
> date.withYear(year+1)  can lead to incorrect date calculations .. for example 
> if  the date is Feb 29th.   need to use date.plusYears(1) instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Leap year : Date parsing in SyslogParser used by MultiportSyslogTCP source

2016-02-25 Thread Hari Shreedharan 
I am looking at your patch now.

On Thu, Feb 25, 2016 at 1:15 PM, Roshan Naik  wrote:
> Recently found a DateTime computation in issue FLUME-2889 which might affect 
> dates when leap year is involved.
>
> Given that 2016 is a leap year, I am trying to assess the impact of this bug.
>
> Method SyslogParser.parseRfc3164Time()  appears to be actually adjusting the 
> year in the date that it is parsing.. It adds to subtracts 1 year to the 
> parsed date based on the current system datetime.
>
> Questions:
> 1 - Why is it trying to modify the year on the parsed date instead of just 
> using It as is ?
> 2 - On a flume agent that retains this bug... Intuitively it seems like this 
> will likely cause incorrect dates in data. Leading to messed up data. Would  
> that  be right ?
>
> -roshan

Leap year : Date parsing in SyslogParser used by MultiportSyslogTCP source

2016-02-25 Thread Roshan Naik 
Recently found a DateTime computation in issue FLUME-2889 which might affect 
dates when leap year is involved.

Given that 2016 is a leap year, I am trying to assess the impact of this bug.

Method SyslogParser.parseRfc3164Time()  appears to be actually adjusting the 
year in the date that it is parsing.. It adds to subtracts 1 year to the parsed 
date based on the current system datetime.

Questions:
1 - Why is it trying to modify the year on the parsed date instead of just 
using It as is ?
2 - On a flume agent that retains this bug... Intuitively it seems like this 
will likely cause incorrect dates in data. Leading to messed up data. Would  
that  be right ?

-roshan

[GitHub] flume pull request: FLUME-2459 Spooling Directory Source support f...

2016-02-25 Thread robin7m 
GitHub user robin7m opened a pull request:

https://github.com/apache/flume/pull/41

FLUME-2459 Spooling Directory Source support for gzip files

Flume has a fantastic source for spooling files, however, many systems 
store relevant files as compressed files.This change enables flume spooling 
directory Source to read gzip (.gz) compressed files just as if they were plain 
text (e.g. read line by line).


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robin7m/flume flume-2459

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/41.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #41


commit ed54a26f37dc9b5e2406493ed5c27bc1119837db
Author: robin7m 
Date:   2016-02-25T15:48:25Z

FLUME-2459 Spooling Directory Source support for gzip files




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLUME-2459) Spooling Directory Source support for compressed files

2016-02-25 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167324#comment-15167324
 ] 

ASF GitHub Bot commented on FLUME-2459:
---

GitHub user robin7m opened a pull request:

https://github.com/apache/flume/pull/41

FLUME-2459 Spooling Directory Source support for gzip files

Flume has a fantastic source for spooling files, however, many systems 
store relevant files as compressed files.This change enables flume spooling 
directory Source to read gzip (.gz) compressed files just as if they were plain 
text (e.g. read line by line).


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robin7m/flume flume-2459

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/41.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #41


commit ed54a26f37dc9b5e2406493ed5c27bc1119837db
Author: robin7m 
Date:   2016-02-25T15:48:25Z

FLUME-2459 Spooling Directory Source support for gzip files




> Spooling Directory Source support for compressed files
> --
>
> Key: FLUME-2459
> URL: https://issues.apache.org/jira/browse/FLUME-2459
> Project: Flume
>  Issue Type: Improvement
>  Components: Sinks+Sources
>Affects Versions: v1.5.0.1
>Reporter: Sverre Bakke
>Assignee: Johny Rufus
>
> Flume has a fantastic source for spooling files, however, many systems store 
> relevant files as compressed files. The Spooling Directory Source should 
> support GZip and BZip2 compressed files just as if they were plain text (e.g. 
> read line by line).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flume pull request: Spooling Directory Source support for gzip fil...

2016-02-25 Thread robin7m 
GitHub user robin7m opened a pull request:

https://github.com/apache/flume/pull/40

Spooling Directory Source support for gzip files

Flume has a fantastic source for spooling files, however, many systems 
store relevant files as compressed files. The Spooling Directory Source should 
support GZip compressed files just as if they were plain text (e.g. read line 
by line).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robin7m/flume FLUME-2459

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/40.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #40


commit ded3576f29a0e7f1fc67de056c27b063821dc12b
Author: robin m 
Date:   2016-02-24T18:10:24Z

Spooling Directory Source support for gzip files




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flume pull request: Spooling Directory Source support for gzip fil...

2016-02-25 Thread robin7m 
Github user robin7m closed the pull request at:

https://github.com/apache/flume/pull/40


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---