[jira] [Commented] (FLUME-3149) reduce cpu cost for file source transfer while still maintaining reliability

2017-08-22 Thread will zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137790#comment-16137790
 ] 

will zhang commented on FLUME-3149:
---

Hi [~bessbd],
Thanks for the reply. I'll patch it soon.

> reduce cpu cost for file source transfer while still maintaining reliability
> 
>
> Key: FLUME-3149
> URL: https://issues.apache.org/jira/browse/FLUME-3149
> Project: Flume
>  Issue Type: Improvement
>  Components: File Channel
>Reporter: will zhang
>
> File channel tracks transferred events and use transnational mechanism to 
> make transfer recoverable. However, it increases CPU cost due to frequent 
> system calls like write, read, etc. The Cpu cost could be very high if the 
> transfer rate is high. In contrast, Memory channel  has no such issue which 
> requires only about 10% of CPU cost  in the same environment but it's not 
> recovered if the system is down accidentally.
> For sources like taildir/spooldir, I propose we could track offsets of file 
> and store them locally to achieve reliability while still using memory 
> channel to reduce CPU cost. Actually, I have already implemented this feature 
> by storing the offsets in event headers and passing it to my own 
> "offsetMemoryChannel" and store theses offsets in local disk in our 
> production which reduces CPU cost by about 90 percent.
> Please let me know if it's worthwhile to have this feature in community 
> version. Thank you.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le

2017-08-22 Thread Attila Simon (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137204#comment-16137204
 ] 

Attila Simon edited comment on FLUME-3057 at 8/22/17 7:01 PM:
--

I have the same issue and updating snappy solved it. Unfortunately the patch 
doesn't apply any more. Since this ticket is unassigned at the moment: to move 
things forward I created a new patch (FLUME-3057-1.patch) which is ready to be 
committed on latest trunk (as of now). I cannot add it to your reviewboard 
review but please feel free to do that. Also you can look at the updated patch 
as a commit here (I'm happy to create a pull request if that helps resolving 
this issue): 
https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa
 


was (Author: sati):
I have the same issue and updating snappy solved it. Unfortunately the patch 
doesn't apply any more. Since this ticket is unassigned at the moment: to move 
things forward I created a new patch which is ready to be committed on latest 
trunk (as of now). I cannot add it to your reviewboard review but please feel 
free to do that. Also you can look at the updated patch as a commit here (I'm 
happy to create a pull request if that helps resolving this issue): 
https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa
 

> Build fails due to unsupported snappy-java version on ppc64le
> -
>
> Key: FLUME-3057
> URL: https://issues.apache.org/jira/browse/FLUME-3057
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.8.0
> Environment: $ uname -a
> Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC 
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Pravin Dsilva
>  Labels: powerpc, ppc64le
> Fix For: 1.8.0
>
> Attachments: FLUME-3057-1.patch, FLUME-3057.patch
>
>
> Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on 
> ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no 
> native library is found for os.name=Linux and os.arch=ppc64le" are seen
>  Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence 
> Flume needs to have this higher version of snappy-java.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le

2017-08-22 Thread Attila Simon (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137204#comment-16137204
 ] 

Attila Simon commented on FLUME-3057:
-

I have the same issue and updating snappy solved it. Unfortunately the patch 
doesn't apply any more. Since this ticket is unassigned at the moment: to move 
things forward I created a new patch which is ready to be committed on latest 
trunk (as of now). I cannot add it to your reviewboard review but please feel 
free to do that. Also you can look at the updated patch as a commit here (I'm 
happy to create a pull request if that helps resolving this issue): 
https://github.com/simonati/flume/commit/5dd54881be4a752ea458d54089dc9cc7816568fa
 

> Build fails due to unsupported snappy-java version on ppc64le
> -
>
> Key: FLUME-3057
> URL: https://issues.apache.org/jira/browse/FLUME-3057
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.8.0
> Environment: $ uname -a
> Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC 
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Pravin Dsilva
>  Labels: powerpc, ppc64le
> Fix For: 1.8.0
>
> Attachments: FLUME-3057-1.patch, FLUME-3057.patch
>
>
> Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on 
> ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no 
> native library is found for os.name=Linux and os.arch=ppc64le" are seen
>  Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence 
> Flume needs to have this higher version of snappy-java.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le

2017-08-22 Thread Attila Simon (JIRA) 

 [ 
https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Simon updated FLUME-3057:

Attachment: FLUME-3057-1.patch

> Build fails due to unsupported snappy-java version on ppc64le
> -
>
> Key: FLUME-3057
> URL: https://issues.apache.org/jira/browse/FLUME-3057
> Project: Flume
>  Issue Type: Bug
>  Components: Build
>Affects Versions: 1.8.0
> Environment: $ uname -a
> Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC 
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>Reporter: Pravin Dsilva
>  Labels: powerpc, ppc64le
> Fix For: 1.8.0
>
> Attachments: FLUME-3057-1.patch, FLUME-3057.patch
>
>
> Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on 
> ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no 
> native library is found for os.name=Linux and os.arch=ppc64le" are seen
>  Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence 
> Flume needs to have this higher version of snappy-java.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3149) reduce cpu cost for file source transfer while still maintaining reliability

2017-08-22 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/FLUME-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137008#comment-16137008
 ] 

Bessenyei Balázs Donát commented on FLUME-3149:
---

Hi [~zyfo2],

Thank you for the issue report and the information!

In case it is not too much of an effort, can you please share the patch you 
have made?
That would allow the community to review the impact of the change and see if we 
can merge it. The idea of improving performance does sound great to me.

If you need any help with creating a pull request or anything like that, please 
let me know.


Thank you,

Donat

> reduce cpu cost for file source transfer while still maintaining reliability
> 
>
> Key: FLUME-3149
> URL: https://issues.apache.org/jira/browse/FLUME-3149
> Project: Flume
>  Issue Type: Improvement
>  Components: File Channel
>Reporter: will zhang
>
> File channel tracks transferred events and use transnational mechanism to 
> make transfer recoverable. However, it increases CPU cost due to frequent 
> system calls like write, read, etc. The Cpu cost could be very high if the 
> transfer rate is high. In contrast, Memory channel  has no such issue which 
> requires only about 10% of CPU cost  in the same environment but it's not 
> recovered if the system is down accidentally.
> For sources like taildir/spooldir, I propose we could track offsets of file 
> and store them locally to achieve reliability while still using memory 
> channel to reduce CPU cost. Actually, I have already implemented this feature 
> by storing the offsets in event headers and passing it to my own 
> "offsetMemoryChannel" and store theses offsets in local disk in our 
> production which reduces CPU cost by about 90 percent.
> Please let me know if it's worthwhile to have this feature in community 
> version. Thank you.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3149) reduce cpu cost for file source transfer while still maintaining reliability

2017-08-22 Thread will zhang (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136600#comment-16136600
 ] 

will zhang commented on FLUME-3149:
---

Can anyone please help review this issue? Thanks.

> reduce cpu cost for file source transfer while still maintaining reliability
> 
>
> Key: FLUME-3149
> URL: https://issues.apache.org/jira/browse/FLUME-3149
> Project: Flume
>  Issue Type: Improvement
>  Components: File Channel
>Reporter: will zhang
>
> File channel tracks transferred events and use transnational mechanism to 
> make transfer recoverable. However, it increases CPU cost due to frequent 
> system calls like write, read, etc. The Cpu cost could be very high if the 
> transfer rate is high. In contrast, Memory channel  has no such issue which 
> requires only about 10% of CPU cost  in the same environment but it's not 
> recovered if the system is down accidentally.
> For sources like taildir/spooldir, I propose we could track offsets of file 
> and store them locally to achieve reliability while still using memory 
> channel to reduce CPU cost. Actually, I have already implemented this feature 
> by storing the offsets in event headers and passing it to my own 
> "offsetMemoryChannel" and store theses offsets in local disk in our 
> production which reduces CPU cost by about 90 percent.
> Please let me know if it's worthwhile to have this feature in community 
> version. Thank you.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLUME-3152) Add Flume Metric for Backup Checkpoint Errors

2017-08-22 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLUME-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136377#comment-16136377
 ] 

ASF GitHub Bot commented on FLUME-3152:
---

GitHub user szaboferee opened a pull request:

https://github.com/apache/flume/pull/156

FLUME-3152 Add Flume Metric for Backup Checkpoint Errors

Added a new metric to the file channel

new test case added
build with tests was successful

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/szaboferee/flume FLUME-3152

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #156


commit 9187cd7bbe158a46db424366e4ebcf43d679e253
Author: Ferenc Szabo 
Date:   2017-08-21T12:29:38Z

FLUME-3152 Add Flume Metric for Backup Checkpoint Errors

Change-Id: I525e4be2d0f95f50e4fa2307fc225a4faaab5aec




> Add Flume Metric for Backup Checkpoint Errors
> -
>
> Key: FLUME-3152
> URL: https://issues.apache.org/jira/browse/FLUME-3152
> Project: Flume
>  Issue Type: Bug
>  Components: File Channel
>Affects Versions: 1.7.0
>Reporter: Ferenc Szabo
>Assignee: Ferenc Szabo
> Fix For: 1.8.0
>
>
> File channel metrics should contain a counter for checkpoint backup errors



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] flume pull request #156: FLUME-3152 Add Flume Metric for Backup Checkpoint E...

2017-08-22 Thread szaboferee 
GitHub user szaboferee opened a pull request:

https://github.com/apache/flume/pull/156

FLUME-3152 Add Flume Metric for Backup Checkpoint Errors

Added a new metric to the file channel

new test case added
build with tests was successful

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/szaboferee/flume FLUME-3152

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/156.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #156


commit 9187cd7bbe158a46db424366e4ebcf43d679e253
Author: Ferenc Szabo 
Date:   2017-08-21T12:29:38Z

FLUME-3152 Add Flume Metric for Backup Checkpoint Errors

Change-Id: I525e4be2d0f95f50e4fa2307fc225a4faaab5aec




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---