Re: Review Request 16415: flume-ng accumulo sink

2014-01-09 Thread Hari Shreedharan
is empty, not if we could not get a complete the batch flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloSink.java https://reviews.apache.org/r/16415/#comment59963 txn.rollback() should also be in try catch. See the current HBaseSink code. - Hari

Re: Review Request 16415: flume-ng accumulo sink

2014-01-09 Thread Hari Shreedharan
On Dec. 31, 2013, 6:43 a.m., Bruno MAHE wrote: flume-ng-sinks/flume-ng-accumulo-sink/src/main/java/org/apache/flume/sink/accumulo/AccumuloSink.java, lines 181-182 https://reviews.apache.org/r/16415/diff/1/?file=401611#file401611line181 Naive question: If your serializer

[jira] [Updated] (FLUME-2181) Optionally disable File Channel fsyncs

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-2181: Attachment: FLUME-2181-1.patch Patch that addresses the failure cases we discussed earlier

[jira] [Updated] (FLUME-2181) Optionally disable File Channel fsyncs

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-2181: Attachment: FLUME-2181-1.patch Previous patch had a couple bugs in tests. This should fix

[jira] [Commented] (FLUME-1618) Make Flume NG build and tests work with Hadoop 2.0 Hbase 0.96

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13867505#comment-13867505 ] Hari Shreedharan commented on FLUME-1618: - I have a patch for this issue, which

[jira] [Commented] (FLUME-1618) Make Flume NG build and tests work with Hadoop 2.0 Hbase 0.96

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13867506#comment-13867506 ] Hari Shreedharan commented on FLUME-1618: - [~roshan_naik] - Please let me know

[jira] [Updated] (FLUME-1618) Make Flume NG build and tests work with Hadoop 2.0 Hbase 0.96

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-1618: Attachment: FLUME-1618-4.patch Patch that allows Flume to build against HBase 96. Make

[jira] [Updated] (FLUME-1618) Make Flume NG build and tests work with Hadoop 2.0 Hbase 0.96

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-1618: Attachment: (was: FLUME-1618-4.patch) Make Flume NG build and tests work with Hadoop

[jira] [Updated] (FLUME-1618) Make Flume NG build and tests work with Hadoop 2.0 Hbase 0.96

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-1618: Attachment: FLUME-1618-4.patch Minor fix to avoid a Hadoop test failure in the new profile

[jira] [Updated] (FLUME-1618) Make Flume NG build and tests work with Hadoop 2.0 Hbase 0.96

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-1618: Attachment: FLUME-1618-4.patch Make Flume NG build and tests work with Hadoop 2.0 Hbase

[jira] [Updated] (FLUME-1618) Make Flume NG build and tests work with Hadoop 2.0 Hbase 0.96

2014-01-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-1618: Attachment: (was: FLUME-1618-4.patch) Make Flume NG build and tests work with Hadoop

[jira] [Commented] (FLUME-2216) HDFS file rotations get disabled once maxUnderReplicationCount is reached

2014-01-07 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1386#comment-1386 ] Hari Shreedharan commented on FLUME-2216: - This patch seems to have caused

[jira] [Created] (FLUME-2291) Compressed Sequence files should not use default extension for codecs that are not splittable

2014-01-07 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created FLUME-2291: --- Summary: Compressed Sequence files should not use default extension for codecs that are not splittable Key: FLUME-2291 URL: https://issues.apache.org/jira/browse/FLUME-2291

[jira] [Created] (FLUME-2292) Upgrade mapdb to 0.9.8

2014-01-07 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created FLUME-2292: --- Summary: Upgrade mapdb to 0.9.8 Key: FLUME-2292 URL: https://issues.apache.org/jira/browse/FLUME-2292 Project: Flume Issue Type: Bug

[jira] [Commented] (FLUME-2292) Upgrade mapdb to 0.9.8

2014-01-07 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865074#comment-13865074 ] Hari Shreedharan commented on FLUME-2292: - Please go ahead. Upgrade mapdb

[jira] [Commented] (FLUME-2275) Improve scalability of MorphlineInterceptor under contention

2014-01-02 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13860909#comment-13860909 ] Hari Shreedharan commented on FLUME-2275: - This line causes a compiler warning

[jira] [Commented] (FLUME-2275) Improve scalability of MorphlineInterceptor under contention

2014-01-02 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861037#comment-13861037 ] Hari Shreedharan commented on FLUME-2275: - Please go ahead and submit the patch

[jira] [Created] (FLUME-2287) TestMultiportSyslogSource#testMultiplePorts fails consistently

2014-01-02 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created FLUME-2287: --- Summary: TestMultiportSyslogSource#testMultiplePorts fails consistently Key: FLUME-2287 URL: https://issues.apache.org/jira/browse/FLUME-2287 Project: Flume

[jira] [Commented] (FLUME-2275) Improve scalability of MorphlineInterceptor under contention

2014-01-02 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861116#comment-13861116 ] Hari Shreedharan commented on FLUME-2275: - +1. Committing this. Improve

[jira] [Resolved] (FLUME-2275) Improve scalability of MorphlineInterceptor under contention

2014-01-02 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan resolved FLUME-2275. - Resolution: Fixed Committed, rev: c3a9c80ab431f6ba670142c7ce6813692422f764. Thanks

[jira] [Assigned] (FLUME-2282) File Channel throws IllegalStateException causing some sources to not be to recover from a reconfig

2013-12-24 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan reassigned FLUME-2282: --- Assignee: Hari Shreedharan File Channel throws IllegalStateException causing some

[jira] [Assigned] (FLUME-2282) File Channel throws IllegalStateException causing some sources to not be to recover from a reconfig

2013-12-24 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan reassigned FLUME-2282: --- Assignee: (was: Hari Shreedharan) File Channel throws IllegalStateException

[jira] [Commented] (FLUME-2282) File Channel throws IllegalStateException causing some sources to not be to recover from a reconfig

2013-12-23 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855850#comment-13855850 ] Hari Shreedharan commented on FLUME-2282: - We could add a check for the interrupt

[jira] [Created] (FLUME-2283) Spool Dir source must check interrupt flag before writing to channel

2013-12-23 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created FLUME-2283: --- Summary: Spool Dir source must check interrupt flag before writing to channel Key: FLUME-2283 URL: https://issues.apache.org/jira/browse/FLUME-2283 Project

[jira] [Updated] (FLUME-2283) Spool Dir source must check interrupt flag before writing to channel

2013-12-23 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-2283: Description: FLUME-2282 was likely caused by the Spool Dir Source continuing to attempt

[jira] [Updated] (FLUME-2283) Spool Dir source must check interrupt flag before writing to channel

2013-12-23 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-2283: Attachment: FLUME-2283.patch Patch that handles this. Spool Dir source must check

[jira] [Commented] (FLUME-2127) JMX shutdown command for Flume

2013-12-22 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855299#comment-13855299 ] Hari Shreedharan commented on FLUME-2127: - [~roshan_naik] - You can connect

[jira] [Commented] (FLUME-2282) File Channel throws IllegalStateException causing some sources to not be to recover from a reconfig

2013-12-22 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855300#comment-13855300 ] Hari Shreedharan commented on FLUME-2282: - When the channel throws

[jira] [Commented] (FLUME-2282) File Channel throws IllegalStateException causing some sources to not be to recover from a reconfig

2013-12-22 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855336#comment-13855336 ] Hari Shreedharan commented on FLUME-2282: - That seems to be indicative of some

[jira] [Commented] (FLUME-2282) File Channel throws IllegalStateException causing some sources to not be to recover from a reconfig

2013-12-22 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855444#comment-13855444 ] Hari Shreedharan commented on FLUME-2282: - That is possible - in which case

Re: Review Request 16416: FLUME-2265. Closed bucket writers should be removed from sfwriters map

2013-12-20 Thread Hari Shreedharan
://reviews.apache.org/r/16416/diff/ Testing --- Thanks, Hari Shreedharan

Review Request 16416: FLUME-2265. Closed bucket writers should be removed from sfwriters map

2013-12-20 Thread Hari Shreedharan
--- Thanks, Hari Shreedharan

Re: Review Request 16416: FLUME-2265. Closed bucket writers should be removed from sfwriters map

2013-12-20 Thread Hari Shreedharan
/TestHDFSEventSink.java 5b7cec9 Diff: https://reviews.apache.org/r/16416/diff/ Testing --- Thanks, Hari Shreedharan

Re: Phoenix- Hbase Sink

2013-12-19 Thread Hari Shreedharan
, Dec 15, 2013 at 7:25 AM, Ravi Kiran maghamraviki...@gmail.com (mailto:maghamraviki...@gmail.com) wrote: Hi Hari, Its maghamravikiran Thanks Ravi. On Sat, Dec 14, 2013 at 7:03 AM, Hari Shreedharan hshreedha...@cloudera.com (mailto:hshreedha...@cloudera.com

[jira] [Commented] (FLUME-2265) Closed bucket writers should be removed from sfwriters map

2013-12-18 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13852091#comment-13852091 ] Hari Shreedharan commented on FLUME-2265: - On idle or roll time based close

[jira] [Commented] (FLUME-2270) Twitter Source Documentation Does not load properly

2013-12-17 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851054#comment-13851054 ] Hari Shreedharan commented on FLUME-2270: - +1. Committing now Twitter Source

[jira] [Commented] (FLUME-2181) Optionally disable File Channel fsyncs

2013-12-16 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849633#comment-13849633 ] Hari Shreedharan commented on FLUME-2181: - [~brocknoland] - What do you think

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

2013-12-16 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849852#comment-13849852 ] Hari Shreedharan commented on FLUME-1227: - Hey [~roshan_naik] - Any updates here

[jira] [Commented] (FLUME-2181) Optionally disable File Channel fsyncs

2013-12-16 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849963#comment-13849963 ] Hari Shreedharan commented on FLUME-2181: - If during replay we hit a bad event

[jira] [Commented] (FLUME-2181) Optionally disable File Channel fsyncs

2013-12-16 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850211#comment-13850211 ] Hari Shreedharan commented on FLUME-2181: - I am planning to start doing the new

[jira] [Commented] (FLUME-1946) Tolerate disk failure in FileChannel

2013-12-15 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848659#comment-13848659 ] Hari Shreedharan commented on FLUME-1946: - I am going to implement the mirroring

[jira] [Commented] (FLUME-1501) Flume Scribe Source needs unit tests.

2013-12-15 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848867#comment-13848867 ] Hari Shreedharan commented on FLUME-1501: - Actually you need to just do git diff

[jira] [Commented] (FLUME-1501) Flume Scribe Source needs unit tests.

2013-12-15 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848868#comment-13848868 ] Hari Shreedharan commented on FLUME-1501: - This should work. I see the required

[jira] [Commented] (FLUME-2216) HDFS file rotations get disabled once maxUnderReplicationCount is reached

2013-12-15 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848871#comment-13848871 ] Hari Shreedharan commented on FLUME-2216: - [~gopinathan.av] - Please go ahead

Re: Flume-1.4.0 doesn't suport HBaseSink/AsyncHBaseSink?

2013-12-15 Thread Hari Shreedharan
You should be able to. I have not seen any issue of HBase sink/AsyncHBaseSink with hbase 0.94. Asynchbase sink may not work with HBase 0.96 though. Thanks, Hari On Sunday, December 15, 2013 at 7:30 PM, Bing Li wrote: Hi, All I'm using Flume-1.4.0 with HBaseSink, but can't load data into

Re: Measuring Flume perf

2013-12-15 Thread Hari Shreedharan
Are all of your HDFS sinks writing to the same directory with the same file prefix? Looks like it is - and they are stepping on each other’s toes. Thanks, Hari On Sunday, December 15, 2013 at 7:30 PM, Roshan Naik wrote: Increasing the batch size to 10k didnt make any diff: 24.6k event/sec

[jira] [Commented] (FLUME-2186) Log4jAppender throws a NPE when Flume is turned off

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847243#comment-13847243 ] Hari Shreedharan commented on FLUME-2186: - Looks like the test is a bit flakey

[jira] [Commented] (FLUME-1491) Dynamic configuration from Zookeeper watcher

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847824#comment-13847824 ] Hari Shreedharan commented on FLUME-1491: - You'd also need to pass in a command

[jira] [Commented] (FLUME-2186) Log4jAppender throws a NPE when Flume is turned off

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847856#comment-13847856 ] Hari Shreedharan commented on FLUME-2186: - OK..so theoretically, Msg 4 could have

[jira] [Commented] (FLUME-2155) Improve replay time

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847879#comment-13847879 ] Hari Shreedharan commented on FLUME-2155: - +1. The latest patch looks good. I am

[jira] [Resolved] (FLUME-2155) Improve replay time

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan resolved FLUME-2155. - Resolution: Fixed Committed, rev: 6373032a620bdc687b6d03b12726713d08c71a10. Thanks Brock

[jira] [Assigned] (FLUME-1679) Add dependency on Guava to flume-ng-elasticsearch-sink POM

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan reassigned FLUME-1679: --- Assignee: Andrew Purtell (was: Ashish Paliwal) Assigning to Andrew since he

[jira] [Commented] (FLUME-1679) Add dependency on Guava to flume-ng-elasticsearch-sink POM

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847916#comment-13847916 ] Hari Shreedharan commented on FLUME-1679: - +1. Looks good. Committing. Add

[jira] [Commented] (FLUME-2264) Log4j Appender + Avro Reflection on string results in an invalid avro schema

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847976#comment-13847976 ] Hari Shreedharan commented on FLUME-2264: - +1. Committing. Log4j Appender + Avro

[jira] [Commented] (FLUME-2130) Handle larger payloads via SyslogUDPSource

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847984#comment-13847984 ] Hari Shreedharan commented on FLUME-2130: - The patch does not seem to apply

[jira] [Commented] (FLUME-1501) Flume Scribe Source needs unit tests.

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847990#comment-13847990 ] Hari Shreedharan commented on FLUME-1501: - git apply ~/work/patches-commited

[jira] [Commented] (FLUME-2243) AvroSource to use TransceiverThreadFactory for Thread naming while initializing NioServerSocketChannelFactory

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848010#comment-13848010 ] Hari Shreedharan commented on FLUME-2243: - Thanks Ashish. Looks good in general

[jira] [Commented] (FLUME-2239) Clarify File Channel's dataDirs setting in User Guide

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848020#comment-13848020 ] Hari Shreedharan commented on FLUME-2239: - +1. Clarify File Channel's dataDirs

[DISCUSS] Feature bloat and contrib module

2013-12-13 Thread Hari Shreedharan
Hi all Over the past few weeks, there were some discussions about including additional features in core flume itself. This led to a discussion about adding a contrib module. I thought I’d start an official discussion regarding this. The argument for contrib module was that there are too many

Re: Phoenix- Hbase Sink

2013-12-13 Thread Hari Shreedharan
at 10:40 AM, Hari Shreedharan hshreedha...@cloudera.com (mailto:hshreedha...@cloudera.com) wrote: Hi Ravi, Thanks for the information. You could post a link to this on the wiki here: https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Plugins for users to be able to find

Re: Measuring Flume perf

2013-12-13 Thread Hari Shreedharan
Much of that can be attributed to the fact that the File Channel does actually write to only one file per data directory and all writes are blocking. There are probably a lot more improvements to make, especially when it comes to how we handle transactions (fsync etc). I’d try working with

[jira] [Commented] (FLUME-2271) Log4j appender source can cause NullPointerException even in Unsafe mode

2013-12-13 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848227#comment-13848227 ] Hari Shreedharan commented on FLUME-2271: - Can you check the patch on FLUME-2186

Re: Generic JDBC Sink

2013-12-12 Thread Hari Shreedharan
the discussion on contributed sources / sinks? -- Jeremy On Thu, Nov 28, 2013 at 11:04 AM, Hari Shreedharan hshreedha...@cloudera.com (mailto:hshreedha...@cloudera.com) wrote: I think we could add this to flume as a contrib module (rather than in core flume itself). At this time

[jira] [Commented] (FLUME-1860) Remove transaction capacity from Memory and File channels

2013-12-12 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846935#comment-13846935 ] Hari Shreedharan commented on FLUME-1860: - I think we should close this one out

[jira] [Commented] (FLUME-1860) Remove transaction capacity from Memory and File channels

2013-12-12 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846950#comment-13846950 ] Hari Shreedharan commented on FLUME-1860: - Don't mind - go ahead. Remove

Re: Review Request 16216: FLUME-2259: transaction closure not happening for all the scenario in hbasesink

2013-12-12 Thread Hari Shreedharan
write a dummy serializer which throws an exception the first time and then behaves normally (you could extend the Simple serializer and check if it is the first time it is being called, throw else just call super.getIncrements etc). - Hari Shreedharan On Dec. 12, 2013, 4:09 p.m., Gopinathan

[jira] [Commented] (FLUME-2186) Log4jAppender throws a NPE when Flume is turned off

2013-12-12 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847012#comment-13847012 ] Hari Shreedharan commented on FLUME-2186: - {code} // this alone may not suffice

[jira] [Commented] (FLUME-2186) Log4jAppender throws a NPE when Flume is turned off

2013-12-12 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847142#comment-13847142 ] Hari Shreedharan commented on FLUME-2186: - The idea is to write a test that would

[jira] [Commented] (FLUME-2155) Improve replay time

2013-12-12 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847182#comment-13847182 ] Hari Shreedharan commented on FLUME-2155: - Adding queueset to Log.EXCLUDES

Re: Review Request 15779: Flume-2245 HDFS Sink BucketWriter failing to close after datanode issues

2013-12-12 Thread Hari Shreedharan
On Nov. 25, 2013, 8:07 p.m., Hari Shreedharan wrote: Juhani, I don't mind the idea of this patch, but I am not sure this works. If the flush fails, we ignore the failure and we set isOpen to false, we are not going to try to close the file again (barring retries which is possible

[jira] [Commented] (FLUME-1086) duplicate lib in Flume's lib dir

2013-12-11 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845635#comment-13845635 ] Hari Shreedharan commented on FLUME-1086: - Yes, please do. duplicate lib

[jira] [Commented] (FLUME-2265) Closed bucket writers should be removed from sfwriters map

2013-12-11 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845648#comment-13845648 ] Hari Shreedharan commented on FLUME-2265: - Idle close does remove the reference

[jira] [Updated] (FLUME-2265) Closed bucket writers should be removed from sfwriters map

2013-12-11 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated FLUME-2265: Attachment: FLUME-2265.patch Patch that handles this. It changes the way we handle

[jira] [Commented] (FLUME-2186) Log4jAppender throws a NPE when Flume is turned off

2013-12-11 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845684#comment-13845684 ] Hari Shreedharan commented on FLUME-2186: - Well, the only thing you can really do

[jira] [Commented] (FLUME-2259) transaction closure not happening for all the scenario in hbasesink

2013-12-10 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13844464#comment-13844464 ] Hari Shreedharan commented on FLUME-2259: - [~gopinathan.av] - This looks good

Re: Non Heap Memory Leak (HDFS sink?)

2013-12-10 Thread Hari Shreedharan
The reason for this is the direct memory allocations by HDFS codecs. Reduce your maxOpenFiles and idleTimeout to have the bucket writers garbage collected regularly. Thanks, Hari On Tuesday, December 10, 2013 at 11:19 AM, Roshan Naik wrote: Flume version: 1.4 (compiled with hadoop 2)

[jira] [Created] (FLUME-2265) Closed bucket writers should be removed from sfwriters map

2013-12-10 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created FLUME-2265: --- Summary: Closed bucket writers should be removed from sfwriters map Key: FLUME-2265 URL: https://issues.apache.org/jira/browse/FLUME-2265 Project: Flume

[jira] [Assigned] (FLUME-2265) Closed bucket writers should be removed from sfwriters map

2013-12-10 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan reassigned FLUME-2265: --- Assignee: Hari Shreedharan Closed bucket writers should be removed from sfwriters

Re: Non Heap Memory Leak (HDFS sink?)

2013-12-10 Thread Hari Shreedharan
Flume config - these are parameters for the HDFS sink. Thanks, Hari On Tuesday, December 10, 2013 at 11:54 AM, Steve Morin wrote: Hari Hadoop config or in the flume config On Dec 10, 2013, at 11:30, Hari Shreedharan hshreedha...@cloudera.com (mailto:hshreedha...@cloudera.com) wrote

Re: Non Heap Memory Leak (HDFS sink?)

2013-12-10 Thread Hari Shreedharan
Roshan, This parameter is hdfs.idleTimeout and hdfs.maxOpenFiles (you need to do agent.hdfsSink.hdfs.idleTimeout), thanks to some historical configuration formatting. Thanks, Hari On Tuesday, December 10, 2013 at 11:58 AM, Hari Shreedharan wrote: Flume config - these are parameters

Re: Review Request 16107: FLUME-2155: Improve replay time

2013-12-10 Thread Hari Shreedharan
) - Hari Shreedharan On Dec. 10, 2013, 2:58 p.m., Brock Noland wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16107/ --- (Updated

Re: Review Request 16107: FLUME-2155: Improve replay time

2013-12-10 Thread Hari Shreedharan
On Dec. 10, 2013, 9:10 p.m., Hari Shreedharan wrote: Brock: Looks like this patch is causing test failures: Failed tests: testRestartWhenMetaDataExistsButCheckpointDoesNotWithBackup(org.apache.flume.channel.file.TestFileChannelRestart

[jira] [Commented] (FLUME-2117) HBaseSink flume.SinkRunner - Infinite loop

2013-12-10 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13844700#comment-13844700 ] Hari Shreedharan commented on FLUME-2117: - Yes, it should. HBaseSink

[jira] [Commented] (FLUME-2228) HDFS namenode failover cause flume error

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843388#comment-13843388 ] Hari Shreedharan commented on FLUME-2228: - Typically, the only thing you can do

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843403#comment-13843403 ] Hari Shreedharan commented on FLUME-1227: - Hi Roshan, In the takePrimary

[jira] [Commented] (FLUME-2261) Flume HDFS sinks can't overcome NN HA switching

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843405#comment-13843405 ] Hari Shreedharan commented on FLUME-2261: - Does this not recover after your HDFS

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843421#comment-13843421 ] Hari Shreedharan commented on FLUME-1227: - Also, there are several lines 80

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843442#comment-13843442 ] Hari Shreedharan commented on FLUME-1227: - The patch seems to be failing tests

[jira] [Commented] (FLUME-2262) Log4j Appender should use timeStamp field not getTimestamp

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843616#comment-13843616 ] Hari Shreedharan commented on FLUME-2262: - +1. Looks good. Running tests

[jira] [Commented] (FLUME-2238) Provide option to configure worker threads in NettyAvroRpcClient

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843702#comment-13843702 ] Hari Shreedharan commented on FLUME-2238: - +1. I am running the tests now

Re: Review Request 15683: FLUME-2238 - Provide option to configure worker threads in NettyAvroRpcClient

2013-12-09 Thread Hari Shreedharan
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15683/#review30060 --- Ship it! Ship It! - Hari Shreedharan On Nov. 21, 2013, 8:13 p.m

[jira] [Commented] (FLUME-2243) AvroSource to use TransceiverThreadFactory for Thread naming while initializing NioServerSocketChannelFactory

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843770#comment-13843770 ] Hari Shreedharan commented on FLUME-2243: - This one got committed with FLUME-2238

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843789#comment-13843789 ] Hari Shreedharan commented on FLUME-1227: - Yes, it was v7. Introduce some sort

[jira] [Commented] (FLUME-2130) Handle larger payloads via SyslogUDPSource

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843896#comment-13843896 ] Hari Shreedharan commented on FLUME-2130: - Hi Ashish, Thanks for the patch. I

Re: Review Request 16031: Patch for FLUME-2209

2013-12-09 Thread Hari Shreedharan
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16031/#review30072 --- Ship it! Ship It! - Hari Shreedharan On Dec. 5, 2013, 1:57 a.m

[jira] [Created] (FLUME-2263) Bump Hadoop 2 version to 2.2.0

2013-12-09 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created FLUME-2263: --- Summary: Bump Hadoop 2 version to 2.2.0 Key: FLUME-2263 URL: https://issues.apache.org/jira/browse/FLUME-2263 Project: Flume Issue Type: Bug

[jira] [Commented] (FLUME-2209) AsyncHBaseSink will never recover if the column family does not exists for the first start

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843915#comment-13843915 ] Hari Shreedharan commented on FLUME-2209: - +1. Looks good. Committing

[jira] [Commented] (FLUME-2243) AvroSource to use TransceiverThreadFactory for Thread naming while initializing NioServerSocketChannelFactory

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843920#comment-13843920 ] Hari Shreedharan commented on FLUME-2243: - [~ashishpaliwal] - Please confirm

[jira] [Commented] (FLUME-1501) Flume Scribe Source needs unit tests.

2013-12-09 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/FLUME-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843924#comment-13843924 ] Hari Shreedharan commented on FLUME-1501: - This patch does not seem to apply. Can

Re: Review Request 16107: FLUME-2155: Improve replay time

2013-12-08 Thread Hari Shreedharan
) at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1331) at org.apache.flume.channel.file.FlumeEventQueue.init(FlumeEventQueue.java:87) at org.apache.flume.channel.file.TestCheckpoint.testSerialization(TestCheckpoint.java:54) - Hari Shreedharan On Dec. 8, 2013, 5:11 p.m., Brock Noland wrote

<    5   6   7   8   9   10   11   12   13   14   >