[jira] [Updated] (FLUME-2945) Bump java target version to 1.8
[ https://issues.apache.org/jira/browse/FLUME-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2945: - Description: We should move to Java 8 as a minimum requirement. # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. # Many projects are adopting java 8 as a minimum requirement, for instance: * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 * elasticsearch 5: https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91". All unit tests, except for the known flaky ones, successfully passed. was: We should move to Java 8 as a minimum requirement. # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. # Many projects are adopting java 8 as a minimum requirement, for instance: * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 * elasticsearch 5: https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html > Bump java target version to 1.8 > --- > > Key: FLUME-2945 > URL: https://issues.apache.org/jira/browse/FLUME-2945 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.8.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.8.0 > > Attachments: FLUME-2945-0.patch > > > We should move to Java 8 as a minimum requirement. > # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. > # Many projects are adopting java 8 as a minimum requirement, for instance: > * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 > * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 > * elasticsearch 5: > https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html > The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91". > All unit tests, except for the known flaky ones, successfully passed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2945) Bump java target version to 1.8
[ https://issues.apache.org/jira/browse/FLUME-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2945: - Attachment: FLUME-2945-0.patch > Bump java target version to 1.8 > --- > > Key: FLUME-2945 > URL: https://issues.apache.org/jira/browse/FLUME-2945 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.8.0 >Reporter: Lior Zeno > Fix For: v1.8.0 > > Attachments: FLUME-2945-0.patch > > > We should move to Java 8 as a minimum requirement. > # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. > # Many projects are adopting java 8 as a minimum requirement, for instance: > * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 > * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 > * elasticsearch 5: > https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2945) Bump java target version to 1.8
[ https://issues.apache.org/jira/browse/FLUME-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno reassigned FLUME-2945: Assignee: Lior Zeno > Bump java target version to 1.8 > --- > > Key: FLUME-2945 > URL: https://issues.apache.org/jira/browse/FLUME-2945 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.8.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.8.0 > > Attachments: FLUME-2945-0.patch > > > We should move to Java 8 as a minimum requirement. > # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. > # Many projects are adopting java 8 as a minimum requirement, for instance: > * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 > * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 > * elasticsearch 5: > https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-3009) Update the website for 1.7 release
[ https://issues.apache.org/jira/browse/FLUME-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572050#comment-15572050 ] Lior Zeno commented on FLUME-3009: -- We already have a ticket for this - FLUME-2932. > Update the website for 1.7 release > -- > > Key: FLUME-3009 > URL: https://issues.apache.org/jira/browse/FLUME-3009 > Project: Flume > Issue Type: Sub-task >Reporter: Bessenyei Balázs Donát > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2994) flume-taildir-source: support for windows
[
https://issues.apache.org/jira/browse/FLUME-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549280#comment-15549280
]
Lior Zeno commented on FLUME-2994:
--
I tested the patch on Windows 10 64bit and got a test failure:
{code}
Running org.apache.flume.source.taildir.TestTaildirEventReader
Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.022 sec <<<
FAILURE!
testLoadPositionFile(org.apache.flume.source.taildir.TestTaildirEventReader)
Time elapsed: 38 sec <<< FAILURE!
java.lang.AssertionError: expected:<22> but was:<0>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.flume.source.taildir.TestTaildirEventReader.testLoadPositionFile(TestTaildirEventReader.java:425)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{code}
I also got a compilation warning:
{code}
[WARNING]
\Development\flume\flume-ng-sources\flume-taildir-source\src\main\java\org\apache\flume\source\taildir\ReliableTaildirEventReader.java:[36,17]
FileKey is internal proprietary API and may be removed in a future release
[WARNING]
\Development\flume\flume-ng-sources\flume-taildir-source\src\main\java\org\apache\flume\source\taildir\ReliableTaildirEventReader.java:[285,8]
FileKey is internal proprietary API and may be removed in a future release
[WARNING]
\Development\flume\flume-ng-sources\flume-taildir-source\src\main\java\org\apache\flume\source\taildir\ReliableTaildirEventReader.java:[285,26]
FileKey is internal proprietary API and may be removed in a future release
{code}
> flume-taildir-source: support for windows
> -
>
> Key: FLUME-2994
> URL: https://issues.apache.org/jira/browse/FLUME-2994
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources, Windows
>Affects Versions: v1.7.0
>Reporter: Jason Kushmaul
>Assignee: Jason Kushmaul
>Priority: Trivial
> Fix For: v1.7.0
>
> Attachments: FLUME-2994-2.patch, taildir-mac.conf,
[jira] [Commented] (FLUME-3000) Update morphline solr sink to use solr-6
[ https://issues.apache.org/jira/browse/FLUME-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544894#comment-15544894 ] Lior Zeno commented on FLUME-3000: -- This is a duplication of FLUME-2919, which is already scheduled to 1.8.0. > Update morphline solr sink to use solr-6 > > > Key: FLUME-3000 > URL: https://issues.apache.org/jira/browse/FLUME-3000 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.2 >Reporter: wolfgang hoschek >Assignee: wolfgang hoschek >Priority: Minor > Fix For: v1.8.0 > > > Move flume from solr-4 to solr-6. This involves changing flume to depend on > the upcoming kite-1.2 release, which in turn used the solr-6 API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2811) Taildir source doesn't call stop() on graceful shutdown
[
https://issues.apache.org/jira/browse/FLUME-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518721#comment-15518721
]
Lior Zeno commented on FLUME-2811:
--
Thanks [~umesh9...@gmail.com], go ahead and assign this issue to yourself.
> Taildir source doesn't call stop() on graceful shutdown
> ---
>
> Key: FLUME-2811
> URL: https://issues.apache.org/jira/browse/FLUME-2811
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Jun Seok Hong
>Priority: Critical
> Labels: newbie
> Fix For: v1.7.0
>
>
> Taildir source doesn't call stop() on graceful shutdown.
> Test configuration.
> source - taildir
> channel - PseudoTxnMemoryChannel / flume-kafka-channel
> sink - none
> I found that flume sometimes doesn't terminate with Taildir source.
> I had to kill the process to terminate it.
> tailFileProcess() function in TaildirSource.java has a infinite loop.
> When the process interrupted, ChannelException will happen, but it can't
> breaks the infinite loop.
> I think that's the reason why Taildir can't call stop() function.
> {code:title=TaildirSource.java|borderStyle=solid}
> private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
> throws IOException, InterruptedException {
> while (true) {
> reader.setCurrentFile(tf);
> List events = reader.readEvents(batchSize, backoffWithoutNL);
> if (events.isEmpty()) {
> break;
> }
> sourceCounter.addToEventReceivedCount(events.size());
> sourceCounter.incrementAppendBatchReceivedCount();
> try {
> getChannelProcessor().processEventBatch(events);
> reader.commit();
> } catch (ChannelException ex) {
> logger.warn("The channel is full or unexpected failure. " +
> "The source will try again after " + retryInterval + " ms");
> TimeUnit.MILLISECONDS.sleep(retryInterval);
> retryInterval = retryInterval << 1;
> retryInterval = Math.min(retryInterval, maxRetryInterval);
> continue;
> }
> retryInterval = 1000;
> sourceCounter.addToEventAcceptedCount(events.size());
> sourceCounter.incrementAppendBatchAcceptedCount();
> if (events.size() < batchSize) {
> break;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2977) fix compile warnings from SAXParser
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458409#comment-15458409
]
Lior Zeno commented on FLUME-2977:
--
Looks like this issue is fixed by a recent pull request
https://github.com/apache/flume/pull/63
> fix compile warnings from SAXParser
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2977) fix compile warnings from SAXParser
[
https://issues.apache.org/jira/browse/FLUME-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438887#comment-15438887
]
Lior Zeno commented on FLUME-2977:
--
+1. Thanks for the patch, these warning messages are really annoying.
> fix compile warnings from SAXParser
> ---
>
> Key: FLUME-2977
> URL: https://issues.apache.org/jira/browse/FLUME-2977
> Project: Flume
> Issue Type: Improvement
> Components: Build
>Reporter: Attila Simon
>Assignee: Attila Simon
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2977.patch
>
>
> Before RAT check mvn install prints out the following warnings:
> {noformat}
> Warning: org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> Compiler warnings:
> WARNING: 'org.apache.xerces.jaxp.SAXParserImpl: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.'
> Warning: org.apache.xerces.parsers.SAXParser: Feature
> 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
> Warning: org.apache.xerces.parsers.SAXParser: Property
> 'http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
> recognized.
> [INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0
> approved: 9 licence.
> {noformat}
> It doesn't break the build but seems misleading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2484) NullPointerException in Kafka Sink test
[ https://issues.apache.org/jira/browse/FLUME-2484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430734#comment-15430734 ] Lior Zeno commented on FLUME-2484: -- Sure, go ahead. > NullPointerException in Kafka Sink test > --- > > Key: FLUME-2484 > URL: https://issues.apache.org/jira/browse/FLUME-2484 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Santiago M. Mola >Priority: Blocker > Fix For: v1.7.0 > > > Kafka Sink test fails on Travis with NullPointerException: > https://travis-ci.org/Stratio/flume/jobs/36814710#L6560 > Running org.apache.flume.sink.kafka.TestKafkaSink > Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 17.061 sec > <<< FAILURE! > testStaticTopic(org.apache.flume.sink.kafka.TestKafkaSink) Time elapsed: > 1823 sec <<< ERROR! > java.lang.NullPointerException > at > org.apache.flume.sink.kafka.TestKafkaSink.testStaticTopic(TestKafkaSink.java:113) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) > Results : > Tests in error: > testStaticTopic(org.apache.flume.sink.kafka.TestKafkaSink) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2959) Fix issues with flume-checkstyle module
[ https://issues.apache.org/jira/browse/FLUME-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396261#comment-15396261 ] Lior Zeno commented on FLUME-2959: -- [~mpercy], thanks for committing this. Do you understand why the build has failed? I must say that I think something is wrong with the jenkins server, we have had multiple weird build failures recently, each time with a unique unexplained error. > Fix issues with flume-checkstyle module > --- > > Key: FLUME-2959 > URL: https://issues.apache.org/jira/browse/FLUME-2959 > Project: Flume > Issue Type: Improvement >Reporter: Lior Zeno >Assignee: Lior Zeno >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2959-0.patch, FLUME-2959-1.patch > > > In FLUME-2937 we added the flume-checkstyle module to Flume. > There are three minor issues with it: > # It does not follow our modules naming convention. It should be > flume-ng-checkstyle. > # It does not declare a parent module, resulting in build warnings (the > source encoding is not explicitly set to UTF-8) > # Resource files don't include the ASF licensing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2959) flume-checkstyle module should be consistent with flume-ng prefix convention
[
https://issues.apache.org/jira/browse/FLUME-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395973#comment-15395973
]
Lior Zeno commented on FLUME-2959:
--
{quote}
I don't think the -ng stuff is really helpful and I don't think we need to
change it. We do already have flume-tools that is missing the -ng suffix so
it's not unprecedented.
{quote}
I'm not a big fan of the flume-ng prefix either, but I do think we should have
a consistent module naming convention.
{quote}
The POM should not have flume-parent as a parent because then it would have
itself as a dependency which is strange. Maybe we should just specify UTF-8 as
the encoding if that is the problem.
{quote}
I've submitted a new patch.
Thanks.
> flume-checkstyle module should be consistent with flume-ng prefix convention
>
>
> Key: FLUME-2959
> URL: https://issues.apache.org/jira/browse/FLUME-2959
> Project: Flume
> Issue Type: Improvement
>Reporter: Lior Zeno
>Assignee: Lior Zeno
>Priority: Minor
> Fix For: v1.7.0
>
> Attachments: FLUME-2959-0.patch, FLUME-2959-1.patch
>
>
> In FLUME-2937 we added the flume-checkstyle module to Flume.
> There are three minor issues with it:
> # It does not follow our modules naming convention. It should be
> flume-ng-checkstyle.
> # It does not declare a parent module, resulting in build warnings (the
> source encoding is not explicitly set to UTF-8)
> # Resource files don't include the ASF licensing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2959) flume-checkstyle module should be consistent with flume-ng prefix convention
[ https://issues.apache.org/jira/browse/FLUME-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2959: - Attachment: FLUME-2959-1.patch > flume-checkstyle module should be consistent with flume-ng prefix convention > > > Key: FLUME-2959 > URL: https://issues.apache.org/jira/browse/FLUME-2959 > Project: Flume > Issue Type: Improvement >Reporter: Lior Zeno >Assignee: Lior Zeno >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2959-0.patch, FLUME-2959-1.patch > > > In FLUME-2937 we added the flume-checkstyle module to Flume. > There are three minor issues with it: > # It does not follow our modules naming convention. It should be > flume-ng-checkstyle. > # It does not declare a parent module, resulting in build warnings (the > source encoding is not explicitly set to UTF-8) > # Resource files don't include the ASF licensing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2173) Exactly once semantics for Flume
[ https://issues.apache.org/jira/browse/FLUME-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390755#comment-15390755 ] Lior Zeno commented on FLUME-2173: -- Kafka's offset management alone is not enough. Flume may fail between writing to the sink and committing an offset. > Exactly once semantics for Flume > > > Key: FLUME-2173 > URL: https://issues.apache.org/jira/browse/FLUME-2173 > Project: Flume > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Hari Shreedharan > Fix For: v2.0.0 > > > Currently Flume guarantees only at least once semantics. This jira is meant > to track exactly once semantics for Flume. My initial idea is to include uuid > event ids on events at the original source (use a config to mark a source an > original source) and identify destination sinks. At the destination sinks, > use a unique ZK Znode to track the events. If once seen (and configured), > pull the duplicate out. > This might need some refactoring, but my belief is we can do this in a > backward compatible way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2959) flume-checkstyle module should be consistent with flume-ng prefix convention
[ https://issues.apache.org/jira/browse/FLUME-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2959: - Description: In FLUME-2937 we added the flume-checkstyle module to Flume. There are three minor issues with it: # It does not follow our modules naming convention. It should be flume-ng-checkstyle. # It does not declare a parent module, resulting in build warnings (the source encoding is not explicitly set to UTF-8) # Resource files don't include the ASF licensing. was: In FLUME-2937 we added the flume-checkstyle module to Flume. There are two minor issues with it: # It does not follow our modules naming convention. It should be flume-ng-checkstyle. # It does not declare a parent module, resulting in build warnings (the source encoding is not explicitly set to UTF-8) # Resource files don't include the ASF licensing. > flume-checkstyle module should be consistent with flume-ng prefix convention > > > Key: FLUME-2959 > URL: https://issues.apache.org/jira/browse/FLUME-2959 > Project: Flume > Issue Type: Improvement >Reporter: Lior Zeno >Assignee: Lior Zeno >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2959-0.patch > > > In FLUME-2937 we added the flume-checkstyle module to Flume. > There are three minor issues with it: > # It does not follow our modules naming convention. It should be > flume-ng-checkstyle. > # It does not declare a parent module, resulting in build warnings (the > source encoding is not explicitly set to UTF-8) > # Resource files don't include the ASF licensing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2959) flume-checkstyle module should be consistent with flume-ng prefix convention
[ https://issues.apache.org/jira/browse/FLUME-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2959: - Description: In FLUME-2937 we added the flume-checkstyle module to Flume. There are two minor issues with it: # It does not follow our modules naming convention. It should be flume-ng-checkstyle. # It does not declare a parent module, resulting in build warnings (the source encoding is not explicitly set to UTF-8) # Resource files don't include the ASF licensing. was: In FLUME-2937 we added the flume-checkstyle module to Flume. There are two minor issues with it: # It does not follow our modules naming convention. It should be flume-ng-checkstyle. # It does not declare a parent module, resulting in build warnings (the source encoding is not explicitly set to UTF-8) > flume-checkstyle module should be consistent with flume-ng prefix convention > > > Key: FLUME-2959 > URL: https://issues.apache.org/jira/browse/FLUME-2959 > Project: Flume > Issue Type: Improvement >Reporter: Lior Zeno >Priority: Minor > Fix For: v1.7.0 > > > In FLUME-2937 we added the flume-checkstyle module to Flume. > There are two minor issues with it: > # It does not follow our modules naming convention. It should be > flume-ng-checkstyle. > # It does not declare a parent module, resulting in build warnings (the > source encoding is not explicitly set to UTF-8) > # Resource files don't include the ASF licensing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2959) flume-checkstyle module should be consistent with flume-ng prefix convention
Lior Zeno created FLUME-2959: Summary: flume-checkstyle module should be consistent with flume-ng prefix convention Key: FLUME-2959 URL: https://issues.apache.org/jira/browse/FLUME-2959 Project: Flume Issue Type: Improvement Reporter: Lior Zeno Priority: Minor Fix For: v1.7.0 In FLUME-2937 we added the flume-checkstyle module to Flume. There are two minor issues with it: # It does not follow our modules naming convention. It should be flume-ng-checkstyle. # It does not declare a parent module, resulting in build warnings (the source encoding is not explicitly set to UTF-8) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2958) Add ignorePattern for TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2958: - Fix Version/s: v1.7.0 > Add ignorePattern for TaildirSource > --- > > Key: FLUME-2958 > URL: https://issues.apache.org/jira/browse/FLUME-2958 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Hu Liu, >Assignee: Hu Liu, > Fix For: v1.7.0 > > > we have tried the TaildirSource and found that it's lack of ignorePattern > specifying which files to ignore. I'm glad to work on it if anyone could > assign it to me -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2890) Typo in Twitter source warning
[ https://issues.apache.org/jira/browse/FLUME-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2890: - Fix Version/s: v1.7.0 LGTM +1. > Typo in Twitter source warning > -- > > Key: FLUME-2890 > URL: https://issues.apache.org/jira/browse/FLUME-2890 > Project: Flume > Issue Type: Documentation >Affects Versions: v1.6.0 >Reporter: Daniel Templeton >Priority: Trivial > Fix For: v1.7.0 > > Attachments: FLUME-2890.001.patch > > > "This source is hightly experimental" should be "This source is highly > experimental" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2641) Drop Java 6 support post Flume 1.6
[ https://issues.apache.org/jira/browse/FLUME-2641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2641. -- Resolution: Implemented Implemented by FLUME-2935. > Drop Java 6 support post Flume 1.6 > -- > > Key: FLUME-2641 > URL: https://issues.apache.org/jira/browse/FLUME-2641 > Project: Flume > Issue Type: Bug >Reporter: Hari Shreedharan > > We should also test Java 7 + Hadoop 1. We will not target Java 6 at all, so > we might have to drop Hadoop 1 support if Hadoop 1 jars won't run on Java 7 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2436) Make hadoop-2 the default build profile
[ https://issues.apache.org/jira/browse/FLUME-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2436: - Fix Version/s: v1.8.0 > Make hadoop-2 the default build profile > --- > > Key: FLUME-2436 > URL: https://issues.apache.org/jira/browse/FLUME-2436 > Project: Flume > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Johny Rufus > Labels: build > Fix For: v1.8.0 > > Attachments: FLUME-2436.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2437) S3 Source
[ https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2437: - Fix Version/s: v2.0.0 > S3 Source > - > > Key: FLUME-2437 > URL: https://issues.apache.org/jira/browse/FLUME-2437 > Project: Flume > Issue Type: New Feature >Reporter: Jonathan Natkins >Assignee: Johny Rufus > Fix For: v2.0.0 > > Attachments: FLUME-2437-2.patch, FLUME-2437.patch > > > There have been multiple requests on the mailing list for an S3 source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2399) Flume 1.5.0.1 release
[ https://issues.apache.org/jira/browse/FLUME-2399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2399. -- Resolution: Fixed > Flume 1.5.0.1 release > - > > Key: FLUME-2399 > URL: https://issues.apache.org/jira/browse/FLUME-2399 > Project: Flume > Issue Type: Bug >Reporter: Hari Shreedharan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2333) HTTP source handler doesn't allow for responses
[ https://issues.apache.org/jira/browse/FLUME-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2333: - Fix Version/s: v1.8.0 > HTTP source handler doesn't allow for responses > --- > > Key: FLUME-2333 > URL: https://issues.apache.org/jira/browse/FLUME-2333 > Project: Flume > Issue Type: Improvement >Reporter: Jeremy Karlson >Assignee: Jeremy Karlson > Fix For: v1.8.0 > > Attachments: FLUME-2333-2.diff, FLUME-2333-3.diff, FLUME-2333-4.diff, > FLUME-2333-CUMULATIVE.diff, FLUME-2333.diff > > > Existing HTTP source handlers recieve events via a HTTPServletRequest. This > works, but because the handler doesn't have access to the > HTTPServletResponse, there is no ability to return a response. This makes it > unsuitable for some sort of protocol that relies on bidirectional > communication. > My solution: In addition to the existing HTTPSource interface, I've added a > BidirectionalHTTPSource interface that is provided the servlet response as a > parameter. I've made some changes in the HTTP source allow for both types to > co-exist, and my changes shouldn't affect anyone who is already using the > existing interface. > Also includes minor documentation updates to reflect this. > Review: https://reviews.apache.org/r/18555/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2298) Replication Channel
[ https://issues.apache.org/jira/browse/FLUME-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2298. -- Resolution: Implemented Implemented (using Kafka) by FLUME-2500. > Replication Channel > --- > > Key: FLUME-2298 > URL: https://issues.apache.org/jira/browse/FLUME-2298 > Project: Flume > Issue Type: New Feature > Components: Channel >Reporter: Ted Malaska >Assignee: Ted Malaska > Attachments: Flume Replication Channel Design.0.3.pdf, > FlumeDistributedChannelDesign.0.1.pdf, > FlumeDistributedChannelDesign.0.2.1.pdf, FlumeDistributedChannelDesign.0.2.pdf > > > This channel will allow for events to be persisted with a plugable method on > more then one agent or node. > The goal is to gain the following benefits: > 1. Events will continue to flow to sinks with out loss or with out large > delay even in the case of node failure. > 2. Protect against loss in the case of a complete single node failure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2173) Exactly once semantics for Flume
[ https://issues.apache.org/jira/browse/FLUME-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2173: - Fix Version/s: v2.0.0 > Exactly once semantics for Flume > > > Key: FLUME-2173 > URL: https://issues.apache.org/jira/browse/FLUME-2173 > Project: Flume > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Hari Shreedharan > Fix For: v2.0.0 > > > Currently Flume guarantees only at least once semantics. This jira is meant > to track exactly once semantics for Flume. My initial idea is to include uuid > event ids on events at the original source (use a config to mark a source an > original source) and identify destination sinks. At the destination sinks, > use a unique ZK Znode to track the events. If once seen (and configured), > pull the duplicate out. > This might need some refactoring, but my belief is we can do this in a > backward compatible way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2101) Flume 1.4.0 Umbrella JIRA
[ https://issues.apache.org/jira/browse/FLUME-2101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2101. -- Resolution: Fixed > Flume 1.4.0 Umbrella JIRA > - > > Key: FLUME-2101 > URL: https://issues.apache.org/jira/browse/FLUME-2101 > Project: Flume > Issue Type: Umbrella >Reporter: Mike Percy >Assignee: Mike Percy > > Flume 1.4.0 Umbrella JIRA -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2811) Taildir source doesn't call stop() on graceful shutdown
[
https://issues.apache.org/jira/browse/FLUME-2811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2811:
-
Fix Version/s: v1.7.0
I'm scheduling this to 1.7.0, since it's the first release to include this
source. Still, it requires further investigation.
> Taildir source doesn't call stop() on graceful shutdown
> ---
>
> Key: FLUME-2811
> URL: https://issues.apache.org/jira/browse/FLUME-2811
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Jun Seok Hong
>Priority: Critical
> Labels: newbie
> Fix For: v1.7.0
>
>
> Taildir source doesn't call stop() on graceful shutdown.
> Test configuration.
> source - taildir
> channel - PseudoTxnMemoryChannel / flume-kafka-channel
> sink - none
> I found that flume sometimes doesn't terminate with Taildir source.
> I had to kill the process to terminate it.
> tailFileProcess() function in TaildirSource.java has a infinite loop.
> When the process interrupted, ChannelException will happen, but it can't
> breaks the infinite loop.
> I think that's the reason why Taildir can't call stop() function.
> {code:title=TaildirSource.java|borderStyle=solid}
> private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
> throws IOException, InterruptedException {
> while (true) {
> reader.setCurrentFile(tf);
> List events = reader.readEvents(batchSize, backoffWithoutNL);
> if (events.isEmpty()) {
> break;
> }
> sourceCounter.addToEventReceivedCount(events.size());
> sourceCounter.incrementAppendBatchReceivedCount();
> try {
> getChannelProcessor().processEventBatch(events);
> reader.commit();
> } catch (ChannelException ex) {
> logger.warn("The channel is full or unexpected failure. " +
> "The source will try again after " + retryInterval + " ms");
> TimeUnit.MILLISECONDS.sleep(retryInterval);
> retryInterval = retryInterval << 1;
> retryInterval = Math.min(retryInterval, maxRetryInterval);
> continue;
> }
> retryInterval = 1000;
> sourceCounter.addToEventAcceptedCount(events.size());
> sourceCounter.incrementAppendBatchAcceptedCount();
> if (events.size() < batchSize) {
> break;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLUME-2776) 1.6 Download link not actually Flume
[ https://issues.apache.org/jira/browse/FLUME-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2776. -- Resolution: Not A Problem > 1.6 Download link not actually Flume > > > Key: FLUME-2776 > URL: https://issues.apache.org/jira/browse/FLUME-2776 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: v1.6.0 > Environment: Cygwin running on Windows 7 >Reporter: Bryan Glazer >Priority: Critical > Original Estimate: 24h > Remaining Estimate: 24h > > This link on the Flume download page: > http://www.apache.org/dyn/closer.cgi/flume/1.6.0/apache-flume-1.6.0-bin.tar.gz > downloads a 4.7kb file that doesn't match the md5sum given and obviously is > not a valid Flume binary. > I'm unsure if this is a simple mistake or a malicious file. > Going to the main distribution page (http://www.apache.org/dist/flume/1.6.0/) > gives a valid version of flume. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-710) ExecNIOSource blocks indefinitely if an exception is encountered
[ https://issues.apache.org/jira/browse/FLUME-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-710. - Resolution: Invalid ExecNIOSource was a part of Flume-OG and does not exist anymore. > ExecNIOSource blocks indefinitely if an exception is encountered > > > Key: FLUME-710 > URL: https://issues.apache.org/jira/browse/FLUME-710 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v0.9.5 >Reporter: E. Sammer >Priority: Blocker > > In the ExecNIOSource, any exception that occurs in one of the two reader > threads causes the source to stop transferring data[1] but continue to run. > This, in turn, leaves the source in a state that requires human intervention > (i.e. the configuration needs to be refreshed or the node restarted). > [1] This is because the reader thread for one of stderr or stdout will exit > but the next() call which is reading from a blocking queue won't know a > thread died. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-1687) ApacheSolrSink
[ https://issues.apache.org/jira/browse/FLUME-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-1687. -- Resolution: Implemented Implemented by FLUME-2070. > ApacheSolrSink > -- > > Key: FLUME-1687 > URL: https://issues.apache.org/jira/browse/FLUME-1687 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: v1.2.0, v1.4.0 >Reporter: wolfgang hoschek >Assignee: Israel Ekpo > Attachments: flume-new-feature-dependencies.zip, > flume-new-features-1.3.1-sources.jar, flume-new-features-1.3.1.jar > > > Some use cases need near real time full text indexing of data through Flume > into Solr, where a Flume sink can write directly to a Solr search server. > This is a scalable way to provide low latency querying and data acquisition. > It complements (rather than replaces) use cases based on Map Reduce batch > analysis of HDFS data. > Apache Solr has a client API that uses REST to add documents to a Solr > server, which in turn is based on Lucene. A Solr Sink can extract documents > from flume events and forward them to Solr. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-2001) TwitterSource - An EventDrivenSource for Streaming In Data from Twitter
[ https://issues.apache.org/jira/browse/FLUME-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390621#comment-15390621 ] Lior Zeno edited comment on FLUME-2001 at 7/23/16 8:28 AM: --- Implemented by FLUME-2190. was (Author: liorze): Implemented in FLUME-2190. > TwitterSource - An EventDrivenSource for Streaming In Data from Twitter > --- > > Key: FLUME-2001 > URL: https://issues.apache.org/jira/browse/FLUME-2001 > Project: Flume > Issue Type: New Feature >Affects Versions: v1.4.0 >Reporter: Israel Ekpo >Assignee: Israel Ekpo > > This event-driven source streams in events from Twitter. > We will start with searching tweets and then go from there > https://dev.twitter.com/docs/api/1.1/get/search/tweets -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2031) Removal of Unused Imports from Project
[ https://issues.apache.org/jira/browse/FLUME-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2031. -- Resolution: Not A Problem > Removal of Unused Imports from Project > -- > > Key: FLUME-2031 > URL: https://issues.apache.org/jira/browse/FLUME-2031 > Project: Flume > Issue Type: Improvement > Components: Easy, Technical Debt >Affects Versions: v1.4.0 >Reporter: Israel Ekpo >Assignee: Israel Ekpo >Priority: Minor > Attachments: FLUME-2031.patch > > > Clean up effort to get rid of unused imports throughout the project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2031) Removal of Unused Imports from Project
[ https://issues.apache.org/jira/browse/FLUME-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390620#comment-15390620 ] Lior Zeno commented on FLUME-2031: -- +1. We will optimize imports when we commit changes to a file as part of another issue. I don't think we should be actively working on this. Closing. > Removal of Unused Imports from Project > -- > > Key: FLUME-2031 > URL: https://issues.apache.org/jira/browse/FLUME-2031 > Project: Flume > Issue Type: Improvement > Components: Easy, Technical Debt >Affects Versions: v1.4.0 >Reporter: Israel Ekpo >Assignee: Israel Ekpo >Priority: Minor > Attachments: FLUME-2031.patch > > > Clean up effort to get rid of unused imports throughout the project. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2876) ElasticSearchSink: indexnameBuilderContext.putAll bug fixes
[ https://issues.apache.org/jira/browse/FLUME-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390616#comment-15390616 ] Lior Zeno commented on FLUME-2876: -- [~yh123555], thank you for the configuration example. Maybe I was not clear enough, but I meant adding another test in your patch that catches this issue, i.e. the test fails without the extra code in your patch, and succeeds with it. Now that I'm looking at the jira history, maybe you don't need to do that because this issue has MANY instances, and a patch with a unit test has already been contributed. For example: FLUME-2576 (includes a unit test in the patch), FLUME-2884, FLUME-2391, FLUME-2435, FLUME-2424. > ElasticSearchSink: indexnameBuilderContext.putAll bug fixes > --- > > Key: FLUME-2876 > URL: https://issues.apache.org/jira/browse/FLUME-2876 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.0, v1.6.0, v1.7.0 >Reporter: yh123555 > Labels: unit-test-missing > Fix For: v1.7.0 > > Attachments: > 0001-ElasticSearchSink-indexnameBuilderContext.putAll-bug.patch > > > ElasticSearchSink: indexnameBuilderContext.putAll bug fixes > org.apache.flume.sink.elasticsearch.ElasticSearchSink > indexnameBuilderContext.putAll wrong serializerContext.putAll -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2717) Add commons-io dependency into hadoop-2 profile to enable Flume 1.5 to support Hadoop 2.7
[ https://issues.apache.org/jira/browse/FLUME-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387374#comment-15387374 ] Lior Zeno commented on FLUME-2717: -- Are we planning to support hadoop-2.7.0? Currently hadoop2.version is set to 2.4.0. I wonder why we don't have this issue with 2.4.0, as it depends on commons-io 2.4, and the Charsets class was added to commons-io at 2.3 (https://commons.apache.org/proper/commons-io/javadocs/api-2.4/index.html?org/apache/commons/io/Charsets.html). Is it because hadoop does not use this class in 2.4.0? > Add commons-io dependency into hadoop-2 profile to enable Flume 1.5 to > support Hadoop 2.7 > - > > Key: FLUME-2717 > URL: https://issues.apache.org/jira/browse/FLUME-2717 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.5.0 >Reporter: sam liu >Assignee: Xiang Li > Fix For: v1.7.0 > > Attachments: FLUME-2717-001.patch > > > By default, for branch origin/flume-1.5, the hadoop2.version is 2.4.0. > However, if we want to use hadoop-2.7.0 in flume-1.5, some hadoop regarding > tests will fail with exception 'java.lang.NoClassDefFoundError: > org/apache/commons/io/Charsets'. > The missed class Charsets is a new class in commons-io 2.4 jar file which is > invoked by hadoop-2.7.0, however flume-1.5 depends on commons-io 2.1 which > does not include the class 'Charsets'. > Therefore the solution to enable flume-1.5 to support hadoop-2.7.0 is to add > commons-io 2.4 as a dependency into the hadoop-2 profile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2514) Some TestFileChannelRestart tests are extremely slow
[ https://issues.apache.org/jira/browse/FLUME-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387340#comment-15387340 ] Lior Zeno commented on FLUME-2514: -- [~mpercy], do you manually resolve issues after commit or is there an automatic background process that does it? > Some TestFileChannelRestart tests are extremely slow > > > Key: FLUME-2514 > URL: https://issues.apache.org/jira/browse/FLUME-2514 > Project: Flume > Issue Type: Bug > Components: File Channel >Reporter: Santiago M. Mola >Assignee: Santiago M. Mola > Fix For: v1.7.0 > > Attachments: FLUME-2215-0.patch, FLUME-2215-1.patch > > > TestFileChannelRestart tests are really slow. For example, > testToggleCheckpointCompressionFromFalseToTrue and > testToggleCheckpointCompressionFromTrueToFalse take ~4 minutes each. > Some of them could be made faster by using channels with lower capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2919) Upgrade the Solr version to 6.0.1
[ https://issues.apache.org/jira/browse/FLUME-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2919: - Fix Version/s: (was: v1.7.0) v1.8.0 > Upgrade the Solr version to 6.0.1 > - > > Key: FLUME-2919 > URL: https://issues.apache.org/jira/browse/FLUME-2919 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Minoru Osuka > Fix For: v1.8.0 > > Attachments: FLUME-2919-1.patch, FLUME-2919-2.patch, FLUME-2919.patch > > > Flume morphline-solr-sink is using Solr 4.3.0. Recently, Solr 6.0.1 has been > released. I propose to upgrade to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387332#comment-15387332 ] Lior Zeno commented on FLUME-2797: -- +1 for Hari's suggestion. In our documentation we also mention that (the Multiport Syslog Source) "This is a newer, faster, multi-port capable version of the Syslog TCP source." > SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source > --- > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2340) Refactor to make room for Morphlines Elasticsearch Sink
[ https://issues.apache.org/jira/browse/FLUME-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2340: - Fix Version/s: (was: v1.8.0) v2.0.0 > Refactor to make room for Morphlines Elasticsearch Sink > --- > > Key: FLUME-2340 > URL: https://issues.apache.org/jira/browse/FLUME-2340 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Otis Gospodnetic > Fix For: v2.0.0 > > > Right now there are some non-Solr-specific classes in > org.apache.flume.sink.solr.morphline and everything assumes data will get > loaded into Solr. This should be refactored to make it possible to use > Morphlines and send data to Elasticsearch, too, for example. > See > http://search-hadoop.com/m/Jrb3G1tSCQK1=Re+Questions+about+Morphline+Solr+Sink+structure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2927) Update flume docs and release notes to 1.8-SNAPSHOT in trunk
[ https://issues.apache.org/jira/browse/FLUME-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2927: - Description: Updated due to the new branching model as on the mailing list. Summary: Update flume docs and release notes to 1.8-SNAPSHOT in trunk (was: Update flume docs and release notes to 1.8-SNAPSHOT in trunk and 1.8 branch) > Update flume docs and release notes to 1.8-SNAPSHOT in trunk > > > Key: FLUME-2927 > URL: https://issues.apache.org/jira/browse/FLUME-2927 > Project: Flume > Issue Type: Sub-task >Affects Versions: v1.7.0 >Reporter: Lior Zeno > Fix For: v1.7.0 > > > Updated due to the new branching model as on the mailing list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2926) Update pom.xml version in trunk
[ https://issues.apache.org/jira/browse/FLUME-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2926: - Summary: Update pom.xml version in trunk (was: Update pom.xml version in trunk and 1.8 branch) Updated due to the new branching model as on the mailing list. > Update pom.xml version in trunk > --- > > Key: FLUME-2926 > URL: https://issues.apache.org/jira/browse/FLUME-2926 > Project: Flume > Issue Type: Sub-task >Affects Versions: v1.7.0 >Reporter: Lior Zeno > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2340) Refactor to make room for Morphlines Elasticsearch Sink
[ https://issues.apache.org/jira/browse/FLUME-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2340: - Fix Version/s: (was: v1.7.0) v1.8.0 > Refactor to make room for Morphlines Elasticsearch Sink > --- > > Key: FLUME-2340 > URL: https://issues.apache.org/jira/browse/FLUME-2340 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Otis Gospodnetic > Fix For: v1.8.0 > > > Right now there are some non-Solr-specific classes in > org.apache.flume.sink.solr.morphline and everything assumes data will get > loaded into Solr. This should be refactored to make it possible to use > Morphlines and send data to Elasticsearch, too, for example. > See > http://search-hadoop.com/m/Jrb3G1tSCQK1=Re+Questions+about+Morphline+Solr+Sink+structure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-2340) Refactor to make room for Morphlines Elasticsearch Sink
[ https://issues.apache.org/jira/browse/FLUME-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385518#comment-15385518 ] Lior Zeno edited comment on FLUME-2340 at 7/20/16 8:01 AM: --- Recent elasticsearch versions require Guava 18.0, blocked by FLUME-2957. In addition, we need to figure our if we want to implement this sink via Morphlines or via FLUME-2921. was (Author: liorze): Recent elasticsearch versions require Guava 18.0. > Refactor to make room for Morphlines Elasticsearch Sink > --- > > Key: FLUME-2340 > URL: https://issues.apache.org/jira/browse/FLUME-2340 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Otis Gospodnetic > Fix For: v1.8.0 > > > Right now there are some non-Solr-specific classes in > org.apache.flume.sink.solr.morphline and everything assumes data will get > loaded into Solr. This should be refactored to make it possible to use > Morphlines and send data to Elasticsearch, too, for example. > See > http://search-hadoop.com/m/Jrb3G1tSCQK1=Re+Questions+about+Morphline+Solr+Sink+structure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2340) Refactor to make room for Morphlines Elasticsearch Sink
[ https://issues.apache.org/jira/browse/FLUME-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385518#comment-15385518 ] Lior Zeno commented on FLUME-2340: -- Recent elasticsearch versions require Guava 18.0. > Refactor to make room for Morphlines Elasticsearch Sink > --- > > Key: FLUME-2340 > URL: https://issues.apache.org/jira/browse/FLUME-2340 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Otis Gospodnetic > Fix For: v1.7.0 > > > Right now there are some non-Solr-specific classes in > org.apache.flume.sink.solr.morphline and everything assumes data will get > loaded into Solr. This should be refactored to make it possible to use > Morphlines and send data to Elasticsearch, too, for example. > See > http://search-hadoop.com/m/Jrb3G1tSCQK1=Re+Questions+about+Morphline+Solr+Sink+structure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-2919) Upgrade the Solr version to 6.0.1
[ https://issues.apache.org/jira/browse/FLUME-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385501#comment-15385501 ] Lior Zeno edited comment on FLUME-2919 at 7/20/16 7:59 AM: --- Yes, we a have a ticket for that FLUME-2945. We can either include this in 1.7.0 and require java 8 for this feature, or simply reschedule to 1.8.0/2.0.0. was (Author: liorze): Yes, we a have a ticket for that FLUME-2945. > Upgrade the Solr version to 6.0.1 > - > > Key: FLUME-2919 > URL: https://issues.apache.org/jira/browse/FLUME-2919 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Minoru Osuka > Fix For: v1.7.0 > > Attachments: FLUME-2919-1.patch, FLUME-2919-2.patch, FLUME-2919.patch > > > Flume morphline-solr-sink is using Solr 4.3.0. Recently, Solr 6.0.1 has been > released. I propose to upgrade to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2921) Support Elasticsearch 2.0+
[ https://issues.apache.org/jira/browse/FLUME-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2921: - Fix Version/s: (was: v1.7.0) v1.8.0 > Support Elasticsearch 2.0+ > -- > > Key: FLUME-2921 > URL: https://issues.apache.org/jira/browse/FLUME-2921 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.8.0 > > Attachments: FLUME-2921-0.patch, FLUME-2921-1.patch > > > Elasticsearch sink supports an ancient version of ES. We should make the sink > work with newer versions of Elasticsearch. > I attached a patch for that. Please note that this involves upgrading netty > and guava. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2957) Remove Guava from our public API
Lior Zeno created FLUME-2957: Summary: Remove Guava from our public API Key: FLUME-2957 URL: https://issues.apache.org/jira/browse/FLUME-2957 Project: Flume Issue Type: Task Affects Versions: v1.8.0 Reporter: Lior Zeno Fix For: v1.8.0 Context.getParameters (flume-ng-configuration module) returns com.google.common.collect.ImmutableMap (Guava). We should clean our API and return either a native java interface or Flume's. In addition to the current state being a bad practice, this also means that we are unable to shade Guava in Flume. Note: Since this breaks our public API, I'll reschedule this issue to 2.0 once we have this version managed in jira. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2921) Support Elasticsearch 2.0+
[ https://issues.apache.org/jira/browse/FLUME-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385506#comment-15385506 ] Lior Zeno commented on FLUME-2921: -- It could work, but I highly discourage this solution. We don't want to change elasticsearch dependency from optional to bundled and then back to optional. I think this can wait to a release after 1.7. In the mean time users can choose one of the following workarounds: * Use a third party sink for elasticsearch2 (these can be found on GitHub). * Use the rest client which supports any version of ES. > Support Elasticsearch 2.0+ > -- > > Key: FLUME-2921 > URL: https://issues.apache.org/jira/browse/FLUME-2921 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2921-0.patch, FLUME-2921-1.patch > > > Elasticsearch sink supports an ancient version of ES. We should make the sink > work with newer versions of Elasticsearch. > I attached a patch for that. Please note that this involves upgrading netty > and guava. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2919) Upgrade the Solr version to 6.0.1
[ https://issues.apache.org/jira/browse/FLUME-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385501#comment-15385501 ] Lior Zeno commented on FLUME-2919: -- Yes, we a have a ticket for that FLUME-2945. > Upgrade the Solr version to 6.0.1 > - > > Key: FLUME-2919 > URL: https://issues.apache.org/jira/browse/FLUME-2919 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Minoru Osuka > Fix For: v1.7.0 > > Attachments: FLUME-2919-1.patch, FLUME-2919-2.patch, FLUME-2919.patch > > > Flume morphline-solr-sink is using Solr 4.3.0. Recently, Solr 6.0.1 has been > released. I propose to upgrade to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2876) ElasticSearchSink: indexnameBuilderContext.putAll bug fixes
[ https://issues.apache.org/jira/browse/FLUME-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385496#comment-15385496 ] Lior Zeno commented on FLUME-2876: -- Could you please add a test for this case? > ElasticSearchSink: indexnameBuilderContext.putAll bug fixes > --- > > Key: FLUME-2876 > URL: https://issues.apache.org/jira/browse/FLUME-2876 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.0, v1.6.0, v1.7.0 >Reporter: yh123555 > Fix For: v1.7.0 > > Attachments: > 0001-ElasticSearchSink-indexnameBuilderContext.putAll-bug.patch > > > ElasticSearchSink: indexnameBuilderContext.putAll bug fixes > org.apache.flume.sink.elasticsearch.ElasticSearchSink > indexnameBuilderContext.putAll wrong serializerContext.putAll -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2921) Support Elasticsearch 2.0+
[
https://issues.apache.org/jira/browse/FLUME-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375442#comment-15375442
]
Lior Zeno commented on FLUME-2921:
--
{quote}
I don't think that is possible. The client JARs for those systems would have to
shade the dependency and ship it. Once there is a com.google.guava import in a
compiled JAR (i.e. the HDFS or ES client JAR), we have to classload classes
that match that package name. If one requires Guava 11, and the other requires
Guava 18, then it simply isn't possible to run them at the same time in the
same classloader. However, the workaround is pretty simple: Run multiple agents.
{quote}
The idea is that the user will not use the original elasticsearch.jar, for
instance. But, instead, he will create a shaded jar that shades guava. This
way, we will only have to load one version of Guava and avoid collisions.
{quote}
I agree with you. It's a very sticky problem at this point.
{quote}
It's a bit disappointing, but it is what it is. We will revisit in 2.0.
> Support Elasticsearch 2.0+
> --
>
> Key: FLUME-2921
> URL: https://issues.apache.org/jira/browse/FLUME-2921
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Reporter: Lior Zeno
>Assignee: Lior Zeno
> Fix For: v1.7.0
>
> Attachments: FLUME-2921-0.patch, FLUME-2921-1.patch
>
>
> Elasticsearch sink supports an ancient version of ES. We should make the sink
> work with newer versions of Elasticsearch.
> I attached a patch for that. Please note that this involves upgrading netty
> and guava.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2921) Support Elasticsearch 2.0+
[ https://issues.apache.org/jira/browse/FLUME-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374402#comment-15374402 ] Lior Zeno commented on FLUME-2921: -- I think we should avoid a custom classloader, and simply shade/remove dependencies as necessary. If a user wants to run both ES and HDFS sinks in the same agent, he would have to shade one of them. This is perfectly acceptable. I think that the best thing to do right now is to block this issue until we clean our public API. In the meantime, users may use the REST client with 1.x and up clusters. Since this is a popular sink, there are other projects on GitHub that implements elasticsearch 2.x support on Flume (obviously, by breaking backwards-compatibility), e.g. https://github.com/arberzal/flume-ng-elasticsearch2-sink. > Support Elasticsearch 2.0+ > -- > > Key: FLUME-2921 > URL: https://issues.apache.org/jira/browse/FLUME-2921 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2921-0.patch, FLUME-2921-1.patch > > > Elasticsearch sink supports an ancient version of ES. We should make the sink > work with newer versions of Elasticsearch. > I attached a patch for that. Please note that this involves upgrading netty > and guava. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2921) Support Elasticsearch 2.0+
[ https://issues.apache.org/jira/browse/FLUME-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370147#comment-15370147 ] Lior Zeno commented on FLUME-2921: -- Flume ships all of its deps, that is true. Elasticsearch is an optional dependency,therefore it is not shipped together and we don't have conflicting guava versions in the same classpath. In addition, we had the same problem with the previous release, users that wanted to use this sink had to copy elasticsearch along with its deps to the library dir. This caused collisions of both Guava and Jackson. In this case, if the user wants to use elasticsearch alongside another sink with colliding versions, he has to deal with it via shading. I find it completely reasonable, as we cannot ship shaded versions for each dependency we use. > Support Elasticsearch 2.0+ > -- > > Key: FLUME-2921 > URL: https://issues.apache.org/jira/browse/FLUME-2921 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2921-0.patch, FLUME-2921-1.patch > > > Elasticsearch sink supports an ancient version of ES. We should make the sink > work with newer versions of Elasticsearch. > I attached a patch for that. Please note that this involves upgrading netty > and guava. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2950) ReliableSpoolingFileEventReader.rollCurrentFile is broken
[ https://issues.apache.org/jira/browse/FLUME-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2950: - Attachment: FLUME-2950-0.patch > ReliableSpoolingFileEventReader.rollCurrentFile is broken > - > > Key: FLUME-2950 > URL: https://issues.apache.org/jira/browse/FLUME-2950 > Project: Flume > Issue Type: Bug >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2950-0.patch > > > FLUME-2939 introduced Java 7 API (nio.file) to > ReliableSpoolingFileEventReader. However, the committed patch breaks > rollCurrentFile on Windows machines. > File rename might be not atomic, in such cases, and under the assumption that > the rolled file and the current file store the same data, Flume only logs a > warning and does not assume that the user broke the contract that the > spooling directory source demands. > In order to determine whether the files store the same data, we used the > com.google.common.io.Files.equal method. However, FLUME-2939 uses > java.nio.file.Files.isSameFile which has different semantics. > Consequently, this relaxation does not hold anymore. In addition, > org.apache.flume.client.avro.TestSpoolingFileLineReader.testDestinationExistsAndSameFileWindows > fails. > I attach a patch that brings back the old check for Files.equal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2950) ReliableSpoolingFileEventReader.rollCurrentFile is broken
[ https://issues.apache.org/jira/browse/FLUME-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno reassigned FLUME-2950: Assignee: Lior Zeno > ReliableSpoolingFileEventReader.rollCurrentFile is broken > - > > Key: FLUME-2950 > URL: https://issues.apache.org/jira/browse/FLUME-2950 > Project: Flume > Issue Type: Bug >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > > FLUME-2939 introduced Java 7 API (nio.file) to > ReliableSpoolingFileEventReader. However, the committed patch breaks > rollCurrentFile on Windows machines. > File rename might be not atomic, in such cases, and under the assumption that > the rolled file and the current file store the same data, Flume only logs a > warning and does not assume that the user broke the contract that the > spooling directory source demands. > In order to determine whether the files store the same data, we used the > com.google.common.io.Files.equal method. However, FLUME-2939 uses > java.nio.file.Files.isSameFile which has different semantics. > Consequently, this relaxation does not hold anymore. In addition, > org.apache.flume.client.avro.TestSpoolingFileLineReader.testDestinationExistsAndSameFileWindows > fails. > I attach a patch that brings back the old check for Files.equal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2950) ReliableSpoolingFileEventReader.rollCurrentFile is broken
Lior Zeno created FLUME-2950: Summary: ReliableSpoolingFileEventReader.rollCurrentFile is broken Key: FLUME-2950 URL: https://issues.apache.org/jira/browse/FLUME-2950 Project: Flume Issue Type: Bug Affects Versions: v1.7.0 Reporter: Lior Zeno Fix For: v1.7.0 FLUME-2939 introduced Java 7 API (nio.file) to ReliableSpoolingFileEventReader. However, the committed patch breaks rollCurrentFile on Windows machines. File rename might be not atomic, in such cases, and under the assumption that the rolled file and the current file store the same data, Flume only logs a warning and does not assume that the user broke the contract that the spooling directory source demands. In order to determine whether the files store the same data, we used the com.google.common.io.Files.equal method. However, FLUME-2939 uses java.nio.file.Files.isSameFile which has different semantics. Consequently, this relaxation does not hold anymore. In addition, org.apache.flume.client.avro.TestSpoolingFileLineReader.testDestinationExistsAndSameFileWindows fails. I attach a patch that brings back the old check for Files.equal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2949) Flume fails to build on Windows
[ https://issues.apache.org/jira/browse/FLUME-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2949: - Description: Now that style violations fail Flume build process, Flume fails to build on Windows. The reason for that is that file paths in checkstyle-suppressions.xml are written with non-portable separators. Switching to "[/\\]" as a path separator, should fix this issue. References: * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter * http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html was: Now that style violations fail Flume build process, Flume fails to build on Windows. The reason for that is that file paths in checkstyle-suppresions.xml are written with non-portable separators. Switching to "[/\\]" as a path separator, should fix this issue. References: * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter * http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html > Flume fails to build on Windows > --- > > Key: FLUME-2949 > URL: https://issues.apache.org/jira/browse/FLUME-2949 > Project: Flume > Issue Type: Bug >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2949-0.patch > > > Now that style violations fail Flume build process, Flume fails to build on > Windows. > The reason for that is that file paths in checkstyle-suppressions.xml are > written with non-portable separators. Switching to "[/\\]" as a path > separator, should fix this issue. > References: > * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter > * > http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2949) Flume fails to build on Windows
[ https://issues.apache.org/jira/browse/FLUME-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno reassigned FLUME-2949: Assignee: Lior Zeno > Flume fails to build on Windows > --- > > Key: FLUME-2949 > URL: https://issues.apache.org/jira/browse/FLUME-2949 > Project: Flume > Issue Type: Bug >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2949-0.patch > > > Now that style violations fail Flume build process, Flume fails to build on > Windows. > The reason for that is that file paths in checkstyle-suppresions.xml are > written with non-portable separators. Switching to "[/\\]" as a path > separator, should fix this issue. > References: > * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter > * > http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2949) Flume fails to build on Windows
[ https://issues.apache.org/jira/browse/FLUME-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2949: - Attachment: FLUME-2949-0.patch > Flume fails to build on Windows > --- > > Key: FLUME-2949 > URL: https://issues.apache.org/jira/browse/FLUME-2949 > Project: Flume > Issue Type: Bug >Affects Versions: v1.7.0 >Reporter: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2949-0.patch > > > Now that style violations fail Flume build process, Flume fails to build on > Windows. > The reason for that is that file paths in checkstyle-suppresions.xml are > written with non-portable separators. Switching to "[/\\]" as a path > separator, should fix this issue. > References: > * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter > * > http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2949) Flume fails to build on Windows
[ https://issues.apache.org/jira/browse/FLUME-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2949: - Description: Now that style violations fail Flume build process, Flume fails to build on Windows. The reason for that is that file paths in checkstyle-suppresions.xml are written with non-portable separators. Switching to "[/\\]" as a path separator, should fix this issue. References: * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter * http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html was: Now that style violations fail Flume build process, Flume fails to build in Windows. The reason for that is that file paths in checkstyle-suppresions.xml are written with non-portable separators. Switching to "[/\\]" as a path separator, should fix this issue. References: * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter * http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html Summary: Flume fails to build on Windows (was: Flume fails to build in Windows) > Flume fails to build on Windows > --- > > Key: FLUME-2949 > URL: https://issues.apache.org/jira/browse/FLUME-2949 > Project: Flume > Issue Type: Bug >Affects Versions: v1.7.0 >Reporter: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2949-0.patch > > > Now that style violations fail Flume build process, Flume fails to build on > Windows. > The reason for that is that file paths in checkstyle-suppresions.xml are > written with non-portable separators. Switching to "[/\\]" as a path > separator, should fix this issue. > References: > * http://checkstyle.sourceforge.net/config_filters.html#SuppressionFilter > * > http://checkstyle.2069334.n4.nabble.com/suppression-filter-regex-td2070198.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2853) Allow for YAML configuration files
[
https://issues.apache.org/jira/browse/FLUME-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2853:
-
Affects Version/s: v1.8.0
Fix Version/s: v1.8.0
> Allow for YAML configuration files
> --
>
> Key: FLUME-2853
> URL: https://issues.apache.org/jira/browse/FLUME-2853
> Project: Flume
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: v1.8.0
>Reporter: Christopher White
>Priority: Minor
> Fix For: v1.8.0
>
>
> Allow for YAML formatted configuration files
> (http://www.yaml.org/spec/1.2/spec.html).
> This provides:
> * A more condensed format than properties files
> * Less 'typo' prone for repetitive common prefixes
> * Ability to define a value once and reuse via references (see [spec -
> Structures - Example 2.10|http://www.yaml.org/spec/1.2/spec.html#id2760395]
> For example compare the following properties file and potential YAML
> equivalent:
> {code:title=agent.properties}
> host1.sources = source1
> host1.channels = channel1
> host1.sinks = sink1
> host1.sources.source1.type = seq
> host1.sources.source1.channels = channel1
> host1.channels.channel1.type = memory
> host1.channels.channel1.capacity = 1
> host1.sinks.sink1.type = null
> host1.sinks.sink1.channel = channel1
> {code}
> {code:title=agent.yaml}
> host1:
> sources:
> _: source1
> source1:
> type: seq
> channels: channel1
>
> channels:
> _: channel1
> channel1:
> type: memory
> capacity: 1
>
> sinks:
> _: sink1
> sink1:
> type: null
> channel: channel1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-2945) Bump java target version to 1.8
Lior Zeno created FLUME-2945: Summary: Bump java target version to 1.8 Key: FLUME-2945 URL: https://issues.apache.org/jira/browse/FLUME-2945 Project: Flume Issue Type: Improvement Affects Versions: v1.8.0 Reporter: Lior Zeno Fix For: v1.8.0 We should move to Java 8 as a minimum requirement. # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. # Many projects are adopting java 8 as a minimum requirement, for instance: * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 * elasticsearch 5: https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2912) thrift Sources/Sinks can only authenticate with kerberos principal in format with hostname
[ https://issues.apache.org/jira/browse/FLUME-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360125#comment-15360125 ] Lior Zeno commented on FLUME-2912: -- [~jrufus], what's your status on this? > thrift Sources/Sinks can only authenticate with kerberos principal in format > with hostname > --- > > Key: FLUME-2912 > URL: https://issues.apache.org/jira/browse/FLUME-2912 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Ping Wang >Assignee: Johny Rufus > Fix For: v1.7.0 > > > Using Thrift Sources/Sinks in Kerberos environment, the Flume agents > only work with principle in format "name/_h...@your-realm.com". > If using other valid principle in the format "n...@your-realm.com" it will > hit ERROR of "GSS initiate failed". > Here's the configuration file: > g1.sources.source1.type = spooldir > g1.sources.source1.spoolDir = /test > g1.sources.source1.fileHeader = false > g1.sinks.sink1.type = thrift > g1.sinks.sink1.hostname = localhost > g1.sinks.sink1.port = 5 > g1.channels.channel1.type = memory > g1.channels.channel1.capacity = 1000 > g1.channels.channel1.transactionCapacity = 100 > g1.sources.source1.channels = channel1 > g1.sinks.sink1.channel = channel1 > g2.sources = source2 > g2.sinks = sink2 > g2.channels = channel2 > g2.sources.source2.type = thrift > g2.sources.source2.bind = localhost > g2.sources.source2.port = 5 > g2.sinks.sink2.type = hdfs > g2.sinks.sink2.hdfs.path = /tmp > g2.sinks.sink2.hdfs.filePrefix = thriftData > g2.sinks.sink2.hdfs.writeFormat = Text > g2.sinks.sink2.hdfs.fileType = DataStream > g2.channels.channel2.type = memory > g2.channels.channel2.capacity = 1000 > g2.channels.channel2.transactionCapacity = 100 > g2.sources.source2.channels = channel2 > g2.sinks.sink2.channel = channel2 > g1.sinks.sink1.kerberos = true > g1.sinks.sink1.client-principal = flume/hostn...@xxx.com > g1.sinks.sink1.client-keytab > = /etc/security/keytabs/flume-1563.server.keytab > g1.sinks.sink1.server-principal = flume/hostn...@xxx.com > g2.sources.source2.kerberos = true > g2.sources.source2.agent-principal = flume/hostn...@xxx.com > g2.sources.source2.agent-keytab > = /etc/security/keytabs/flume-1563.server.keytab > If using other valid principle like "t...@ibm.com" as below, will hit error: > g1.sinks.sink1.kerberos = true > g1.sinks.sink1.client-principal = t...@ibm.com > g1.sinks.sink1.client-keytab = /home/test/test.keytab > g1.sinks.sink1.server-principal = t...@ibm.com > g2.sources.source2.kerberos = true > g2.sources.source2.agent-principal = t...@ibm.com > g2.sources.source2.agent-keytab = /home/test/test.keytab > Agent g1: > ERROR server.TThreadPoolServer: Error occurred during processing of > message. > java.lang.RuntimeException: > org.apache.thrift.transport.TTransportException: Peer indicated failure: > GSS initiate failed > at org.apache.thrift.transport.TSaslServerTransport > $Factory.getTransport(TSaslServerTransport.java:219) > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run > (TThreadPoolServer.java:189) > at java.util.concurrent.ThreadPoolExecutor.runWorker > (ThreadPoolExecutor.java:1142) > Agent g2: > ERROR transport.TSaslTransport: SASL negotiation failure > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Server not > found in Kerberos database (7) - UNKNOWN_SERVER)] > at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge > (GssKrb5Client.java:211) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2944) Remove guava as dependency
Lior Zeno created FLUME-2944: Summary: Remove guava as dependency Key: FLUME-2944 URL: https://issues.apache.org/jira/browse/FLUME-2944 Project: Flume Issue Type: Improvement Affects Versions: v1.8.0 Reporter: Lior Zeno Fix For: v1.8.0 Guava is a very popular dependency, which often causes version collisions. Especially, due to lack of backwards compatibility. Adding new dependencies that rely on guava requires us to shade these dependencies. This is hard to maintain and bloats our distribution. Therefore, we should omit guava as a dependency and delegate these problems to our users. If a user would like to use X-Source with Y-Sink, where X and Y rely on colliding versions of guava, the user will have to shade one of them. This task should be easier, now that we have moved to Java 1.7. If we move to 1.8 in this release, then most of it becomes virtually trivial. Flume relies on the following guava imports: * com.google.common.base.Preconditions * com.google.common.collect.ArrayListMultimap * com.google.common.collect.ListMultimap * com.google.common.collect.Lists * com.google.common.collect.Maps * com.google.common.annotations.VisibleForTesting * com.google.common.util.concurrent.ThreadFactoryBuilder * com.google.common.base.Charsets * com.google.common.base.Strings * com.google.common.base.Throwables * com.google.common.eventbus.EventBus * com.google.common.eventbus.Subscribe * com.google.common.primitives.UnsignedBytes * com.google.common.cache.CacheBuilder * com.google.common.cache.CacheLoader * com.google.common.cache.LoadingCache * com.google.common.util.concurrent.UncheckedExecutionException * com.google.common.collect.HashMultimap * com.google.common.collect.SetMultimap * com.google.common.collect.ImmutableMap * com.google.common.base.Joiner * com.google.common.io.Files * com.google.common.io.Resources * com.google.common.collect.ImmutableSortedSet * com.google.common.base.Splitter * com.google.common.collect.Iterables * com.google.common.base.Optional * com.google.common.io.ByteStreams * com.google.common.collect.HashBasedTable * com.google.common.primitives.Longs * com.google.common.collect.Sets * com.google.common.collect.ImmutableListMultimap * com.google.common.collect.ListMultimap * import com.google.common.collect.Table * com.google.common.base.Function -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2938) JDBC Source
[ https://issues.apache.org/jira/browse/FLUME-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359206#comment-15359206 ] Lior Zeno commented on FLUME-2938: -- I'll start by stating that I'm not a Sqoop expert. It's true that this source will have to pull for new events, unlike other sources that get new events by push. However, I'm still not sure how can Sqoop provide the functionality that Flume provides. Flume offers much more target options, simple transformations that do not require a MapReduce job, and so on. I believe that Flume should provide this functionality. This ticket does not compete with Sqoop, it is not intended for batch computations (transformations) on data from relational databases, but a simple mechanism to transfer data from JDBC to any other source in small batches. > JDBC Source > --- > > Key: FLUME-2938 > URL: https://issues.apache.org/jira/browse/FLUME-2938 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: v1.8.0 >Reporter: Lior Zeno > Fix For: v1.8.0 > > > The idea is to allow migrating data from SQL stores to NoSQL stores or HDFS > for archiving purposes. > This source will get a statement to execute and a scheduling policy. It will > be able to fetch timestamped data by performing range queries on a > configurable field (this can fetch data with incremental id as well). For > fault-tolerance, the last fetched value can be checkpointed to a file. > Dealing with large datasets can be done via the fetch_size parameter. (Ref: > https://docs.oracle.com/cd/A87860_01/doc/java.817/a83724/resltse5.htm) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2943) Integrate checkstyle - second pass
[ https://issues.apache.org/jira/browse/FLUME-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2943: - Attachment: google_checks.xml > Integrate checkstyle - second pass > -- > > Key: FLUME-2943 > URL: https://issues.apache.org/jira/browse/FLUME-2943 > Project: Flume > Issue Type: Improvement >Affects Versions: v1.8.0 >Reporter: Lior Zeno >Priority: Minor > Fix For: v1.8.0 > > Attachments: google_checks.xml > > > In the first phase of this task we used a weakened style rules. This issue > proposes to add the following rules: > * Naming conventions > * Java docs (relaxed version) > * Disallowing static star imports > * Disallowing abbreviations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2943) Integrate checkstyle - second pass
Lior Zeno created FLUME-2943: Summary: Integrate checkstyle - second pass Key: FLUME-2943 URL: https://issues.apache.org/jira/browse/FLUME-2943 Project: Flume Issue Type: Improvement Affects Versions: v1.8.0 Reporter: Lior Zeno Priority: Minor Fix For: v1.8.0 In the first phase of this task we used a weakened style rules. This issue proposes to add the following rules: * Naming conventions * Java docs (relaxed version) * Disallowing static star imports * Disallowing abbreviations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2938) JDBC Source
[ https://issues.apache.org/jira/browse/FLUME-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357104#comment-15357104 ] Lior Zeno commented on FLUME-2938: -- Right, my bad. However, my point is that it is much less versatile than Flume. > JDBC Source > --- > > Key: FLUME-2938 > URL: https://issues.apache.org/jira/browse/FLUME-2938 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: v1.8.0 >Reporter: Lior Zeno > Fix For: v1.8.0 > > > The idea is to allow migrating data from SQL stores to NoSQL stores or HDFS > for archiving purposes. > This source will get a statement to execute and a scheduling policy. It will > be able to fetch timestamped data by performing range queries on a > configurable field (this can fetch data with incremental id as well). For > fault-tolerance, the last fetched value can be checkpointed to a file. > Dealing with large datasets can be done via the fetch_size parameter. (Ref: > https://docs.oracle.com/cd/A87860_01/doc/java.817/a83724/resltse5.htm) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2938) JDBC Source
[ https://issues.apache.org/jira/browse/FLUME-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15355567#comment-15355567 ] Lior Zeno commented on FLUME-2938: -- The semantics are the same. However, Sqoop can't export from JDBC to Hbase, for example. So there is a clear motivation here. > JDBC Source > --- > > Key: FLUME-2938 > URL: https://issues.apache.org/jira/browse/FLUME-2938 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: v1.8.0 >Reporter: Lior Zeno > Fix For: v1.8.0 > > > The idea is to allow migrating data from SQL stores to NoSQL stores or HDFS > for archiving purposes. > This source will get a statement to execute and a scheduling policy. It will > be able to fetch timestamped data by performing range queries on a > configurable field (this can fetch data with incremental id as well). For > fault-tolerance, the last fetched value can be checkpointed to a file. > Dealing with large datasets can be done via the fetch_size parameter. (Ref: > https://docs.oracle.com/cd/A87860_01/doc/java.817/a83724/resltse5.htm) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2937) Integrate checkstyle
[ https://issues.apache.org/jira/browse/FLUME-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2937: - Assignee: Mike Percy Affects Version/s: (was: v1.8.0) Fix Version/s: (was: v1.8.0) v1.7.0 > Integrate checkstyle > > > Key: FLUME-2937 > URL: https://issues.apache.org/jira/browse/FLUME-2937 > Project: Flume > Issue Type: Improvement >Reporter: Lior Zeno >Assignee: Mike Percy >Priority: Minor > Fix For: v1.7.0 > > > We should add the maven-checkstyle-plugin to the build process. This plugin > can fail a build if the code does not honor the style of our project. This > way we can make sure that we have one common style in the code. In addition, > reviewers can focus on design, correctness, performance and other important > coding aspects other than style issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2448) Building flume from trunk failing with dependency error
[ https://issues.apache.org/jira/browse/FLUME-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2448: - Fix Version/s: (was: 1.7.0) v1.7.0 > Building flume from trunk failing with dependency error > --- > > Key: FLUME-2448 > URL: https://issues.apache.org/jira/browse/FLUME-2448 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: v1.5.0.1 >Reporter: Ankul Rastogi > Fix For: v1.7.0 > > > I am getting a error during building Flume from source > As per documentation I just cloned from git > (https://git-wip-us.apache.org/repos/asf/flume.git) and ran mvn clean install > -DskipTests > Failing module - Flume NG Morphline Solr Sink > [ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink: Could > not resolve dependencies for project > org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.6.0-SNAPSHOT: > Failed to collect dependencies at > org.kitesdk:kite-morphlines-solr-core:jar:tests:0.15.0 -> > org.apache.solr:solr-core:jar:4.6.1 -> > org.apache.hadoop:hadoop-common:jar:1.0.1: Failed to read artifact descriptor > for org.apache.hadoop:hadoop-common:jar:1.0.1: Could not transfer artifact > org.apache.hadoop:hadoop-common:pom:1.0.1 from/to cdh.repo > (https://repository.cloudera.com/artifactory/cloudera-repos): > repository.cloudera.com: Unknown host repository.cloudera.com -> [Help 1] > Seems that the hadoop-common entries are missing from the cloudera repo. I > see that there is a upgrade of kite from 0.12 -> 0.15 > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=49db6f59173bb9e9ae7ce671034d6562551cedf0 > Is their any help that I can get on this. > PS - When I reverted the kite SDK version back to 0.12 the build completed > successfully -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2935) Bump java target version to 1.7
[ https://issues.apache.org/jira/browse/FLUME-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351552#comment-15351552 ] Lior Zeno commented on FLUME-2935: -- Thanks! It seems like we have a few issues with the tests: # TestThriftLegacySource did not run for some reason (I can't reproduce this on my local machine). # TestKafkaSource did not run - we have a jira for that. (FLUME-2936) # The IT failed since flume-ng-distribution was skipped and the tarball was not generated. We should run mvn package on the dist project before running the IT tests. > Bump java target version to 1.7 > --- > > Key: FLUME-2935 > URL: https://issues.apache.org/jira/browse/FLUME-2935 > Project: Flume > Issue Type: Task >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2395-0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2935) Bump java target version to 1.7
[ https://issues.apache.org/jira/browse/FLUME-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350415#comment-15350415 ] Lior Zeno commented on FLUME-2935: -- Thanks Mike! There are a few issues: # Something went wrong with the patch. For instance, the README.textile file seems incorrect. Can you please take a look at it? Maybe there are other files that were merged incorrectly. # The build fails due to rat check. Can you please upload rat's report? I don't really understand what went wrong here. > Bump java target version to 1.7 > --- > > Key: FLUME-2935 > URL: https://issues.apache.org/jira/browse/FLUME-2935 > Project: Flume > Issue Type: Task >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2395-0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2132) Exception while syncing from Flume to HDFS
[
https://issues.apache.org/jira/browse/FLUME-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno resolved FLUME-2132.
--
Resolution: Incomplete
Closing since this issue does not provide enough information to reproduce.
> Exception while syncing from Flume to HDFS
> --
>
> Key: FLUME-2132
> URL: https://issues.apache.org/jira/browse/FLUME-2132
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.3.0
> Environment: Flume 1.3.0, Hadoop 1.2.0, 8GB RAM, Intel Pentium core 2
> duo
>Reporter: Divya R
> Labels: flume, hadoop
> Fix For: v1.7.0
>
>
> I'm running hadoop 1.2.0 and flume 1.3.0. Every thing works fine if its
> independently run. When I start my tomcat I get the below exception after
> some time.
> {quote}2013-07-17 12:40:35,640 (ResponseProcessor for block
> blk_5249456272858461891_436734) [WARN -
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:3015)]
> DFSOutputStream ResponseProcessor exception for block
> blk_5249456272858461891_436734java.net.SocketTimeoutException: 63000 millis
> timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/127.0.0.1:24433
> remote=/127.0.0.1:50010]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2967){quote}
> {quote}2013-07-17 12:40:35,800 (hdfs-hdfs-write-roll-timer-0) [WARN -
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:277)]
> failed to close() HDFSWriter for file
> (hdfs://localhost:9000/flume/Broadsoft_App2/20130717/jboss/Broadsoft_App2.1374044838498.tmp).
> Exception follows.
> java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3096)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2589)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2793){quote}
> Java snippet for Configuraion
> {quote}configuration.set("fs.default.name", "hdfs://localhost:9000");
> configuration.set("mapred.job.tracker", "hdfs://localhost:9000");{quote}
> I'm using a single datanode to read the files that where written to hdfs by
> flume, my java program just reads the files from hdfs to show it on the
> screen nothing much.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLUME-2143) Flume build occasionally fails with OutOfMemoryError on Windows.
[ https://issues.apache.org/jira/browse/FLUME-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2143. -- Resolution: Not A Problem Setting MAVEN_OPTS solves this issue. > Flume build occasionally fails with OutOfMemoryError on Windows. > > > Key: FLUME-2143 > URL: https://issues.apache.org/jira/browse/FLUME-2143 > Project: Flume > Issue Type: Bug > Components: Build, Windows >Reporter: Roshan Naik >Assignee: Roshan Naik > Labels: windows > Fix For: v1.7.0 > > Attachments: FLUME-2143.patch > > > The build sporadically fails in Flume with the following exception: > [exec] The system is out of resources. > [exec] Consult the following stack trace for details. > [exec] java.lang.OutOfMemoryError: PermGen space > [exec] at java.lang.ClassLoader.defineClass1(Native Method) > [exec] at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > [exec] at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > [exec] at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > [exec] at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > [exec] at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > [exec] at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > [exec] at java.security.AccessController.doPrivileged(Native Method) > [exec] at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > [exec] at > org.codehaus.plexus.compiler.javac.IsolatedClassLoader.loadClass(IsolatedClassLoader.java:56) > [exec] at com.sun.tools.javac.comp.Lower.mapForEnum(Lower.java:319) > [exec] at com.sun.tools.javac.comp.Lower.visitEnumSwitch(Lower.java:3075) > [exec] at com.sun.tools.javac.comp.Lower.visitSwitch(Lower.java:3066) > [exec] at > com.sun.tools.javac.tree.JCTree$JCSwitch.accept(JCTree.java:896) > [exec] at > com.sun.tools.javac.tree.TreeTranslator.translate(TreeTranslator.java:44) > [exec] at com.sun.tools.javac.comp.Lower.translate(Lower.java:1899) > [exec] at > com.sun.tools.javac.tree.TreeTranslator.translate(TreeTranslator.java:56) > [exec] at > com.sun.tools.javac.tree.TreeTranslator.visitBlock(TreeTranslator.java:146) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-1668) Hdfs Sink File Rollover
[ https://issues.apache.org/jira/browse/FLUME-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-1668. -- Resolution: Cannot Reproduce I failed to reproduce this problem on flume 1.6.0 and hadoop 1.2.1. > Hdfs Sink File Rollover > --- > > Key: FLUME-1668 > URL: https://issues.apache.org/jira/browse/FLUME-1668 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources > Environment: Centos >Reporter: Sachin Gupta >Priority: Critical > Fix For: v1.7.0 > > > With HDFS Sink with following config , files are rolling after every 510 > bytes > Hadoop Environment : 1.0.0 > Flume-Ng - 1.2.0 > Config below > yum.sinks.ad-error-sink.type=hdfs > yum.sinks.ad-erorr-sink.hdfs.timeZone=UTC > yum.sinks.ad-error-sink.hdfs.path=hdfs://10.16.1.156:9000/home/data/flume-data/ad-error > yum.sinks.ad-error-sink.hdfs.filePrefix=aderror > yum.sinks.ad-error-sink.hdfs.fileType=DataStream > yum.sinks.ad-error-sink.hdfs.writeFormat=Text > yum.sinks.ad-error.sink.hdfs.rollInterval=0 > yum.sinks.ad-error.sink.hdfs.rollSize=134217728 > yum.sinks.ad-error.sink.hdfs.rollCount=0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2567) Remove unneeded repository declarations in pom.xml
[ https://issues.apache.org/jira/browse/FLUME-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2567: - Fix Version/s: v1.7.0 > Remove unneeded repository declarations in pom.xml > -- > > Key: FLUME-2567 > URL: https://issues.apache.org/jira/browse/FLUME-2567 > Project: Flume > Issue Type: Bug >Reporter: Santiago M. Mola >Assignee: Santiago M. Mola >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2567-0.patch > > > pom.xml contains multiple repository declarations that do not seem to be > needed (jboss, maven central, cloudera). Actually, it seems to me that none > of these is needed. > Can we remove all repository declarations and let it use just the default > (Maven Central)? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2256) Generic JDBC Sink
[
https://issues.apache.org/jira/browse/FLUME-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349503#comment-15349503
]
Lior Zeno commented on FLUME-2256:
--
I have one comment here, I don't think that the user should pass a SQL command
(insert into), but should specify the target table and the headers to column
mapping. The sink will use insert into when batch size is one, however, when
the batch size is larger than one it may use COPY IN, for example.
[~karlj000], are you still available for this?
> Generic JDBC Sink
> -
>
> Key: FLUME-2256
> URL: https://issues.apache.org/jira/browse/FLUME-2256
> Project: Flume
> Issue Type: New Feature
>Reporter: Jeremy Karlson
>Assignee: Jeremy Karlson
> Fix For: v1.8.0
>
> Attachments: FLUME-2256.diff
>
>
> I've been working on a generic JDBC sink. It needs a bit more testing, but I
> think it's ready for review and feedback. I have not yet updated the Flume
> documentation, but I can / will if people are happy with this.
> Since the config file is how you’d interact with it, here’s a working example
> from my source tree:
> {code}
> a.sinks.k.type=jdbc
> a.sinks.k.channel=c
> a.sinks.k.driver=com.mysql.jdbc.Driver
> a.sinks.k.url=jdbc:mysql://localhost:8889/flume
> a.sinks.k.user=username
> a.sinks.k.password=password
> a.sinks.k.batchSize=100
> a.sinks.k.sql=insert into twitter (body, timestamp) values (${body:string},
> ${header.timestamp:long})
> {code}
> The interesting part is the SQL statement. You can put anything you want in
> there - it will get converted to a prepared statement on execution. The
> Ant-ish tokens get parsed and replaced with parameters at startup.
> The tokens are three part. For example, in:
> {code}
> ${body:string(UTF-8)}
> {code}
> The first is a place in the event to get the value from (“body”,
> “header.foo”, or “custom”). The second part ("string") is a type identifier
> that converts into an appropriate JDBC parameter. The third part (“UTF-8")
> is a configuration string for that type, if needed. As for types, so far
> I’ve defined:
> body: string (with optional charset encoding), bytearray
> header: string, long, int, float, double, date (with mandatory date format
> and optional timezone)
> Additionally, if none of those make you happy you can define you own
> parameter converters:
> {code}
> ${custom:com.company.foo.MyConverter(optionaltextconfig)}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-2921) Support Elasticsearch 2.0+
[ https://issues.apache.org/jira/browse/FLUME-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno reassigned FLUME-2921: Assignee: Lior Zeno > Support Elasticsearch 2.0+ > -- > > Key: FLUME-2921 > URL: https://issues.apache.org/jira/browse/FLUME-2921 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2921-0.patch, FLUME-2921-1.patch > > > Elasticsearch sink supports an ancient version of ES. We should make the sink > work with newer versions of Elasticsearch. > I attached a patch for that. Please note that this involves upgrading netty > and guava. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2344) New source for tailing files
[ https://issues.apache.org/jira/browse/FLUME-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2344. -- Resolution: Duplicate > New source for tailing files > > > Key: FLUME-2344 > URL: https://issues.apache.org/jira/browse/FLUME-2344 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.4.0 > Environment: Centos 6.4, Java 1.6.0_34 >Reporter: Guillermo Ortiz Fernández, Pragsis. > Attachments: FLUME-2344-0.patch > > > New source to be able to tail a file. There's a extra file when it saves the > last offset until it has been read. So, it Flume is down, it could read the > data it lost. It has been implemented control about rotated files. > The possible variables to configure this source are: > -BufferSize gives us the possibility to send little to little. > -Separator to cut the lines wherever we want to to generate our events. > -WatchedFile to indicate what file we want to watch. > -RotatedFile to indicate where the wacthed file is going to rotate. > -Type of events to indicate if we generate one event per line or we want to > group many lines and just emit one event. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2362) Memory mapping channel
[ https://issues.apache.org/jira/browse/FLUME-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2362: - Fix Version/s: v1.8.0 > Memory mapping channel > -- > > Key: FLUME-2362 > URL: https://issues.apache.org/jira/browse/FLUME-2362 > Project: Flume > Issue Type: Improvement > Components: Channel >Affects Versions: v1.5.0 >Reporter: Lining Sun > Fix For: v1.8.0 > > Attachments: FLUME-2362.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I've implemented memory mapping channel that has the same performance as > memory channel and the same reliability as file channel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2514) Some TestFileChannelRestart tests are extremely slow
[ https://issues.apache.org/jira/browse/FLUME-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2514: - Fix Version/s: v1.7.0 > Some TestFileChannelRestart tests are extremely slow > > > Key: FLUME-2514 > URL: https://issues.apache.org/jira/browse/FLUME-2514 > Project: Flume > Issue Type: Bug > Components: File Channel >Reporter: Santiago M. Mola >Assignee: Santiago M. Mola > Fix For: v1.7.0 > > Attachments: FLUME-2215-0.patch > > > TestFileChannelRestart tests are really slow. For example, > testToggleCheckpointCompressionFromFalseToTrue and > testToggleCheckpointCompressionFromTrueToFalse take ~4 minutes each. > Some of them could be made faster by using channels with lower capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2524) Adding an HTTP Sink
[ https://issues.apache.org/jira/browse/FLUME-2524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2524: - Fix Version/s: v1.8.0 > Adding an HTTP Sink > --- > > Key: FLUME-2524 > URL: https://issues.apache.org/jira/browse/FLUME-2524 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Jeff Guilmard > Fix For: v1.8.0 > > Attachments: FLUME-2524-0.patch, FLUME-2524-1.patch > > Original Estimate: 72h > Remaining Estimate: 72h > > Flume whould have an HTTP Sink, with following capacities: > - Using up to date performant Http Client > - Capacity to load balance on multiple target servers (simple round robin) > - Handle HTTP Authentication > - use HTTP POST > - Capacity to send binary data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2530) Resource leaks found by Coverity tool
[ https://issues.apache.org/jira/browse/FLUME-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2530: - Fix Version/s: v1.8.0 > Resource leaks found by Coverity tool > - > > Key: FLUME-2530 > URL: https://issues.apache.org/jira/browse/FLUME-2530 > Project: Flume > Issue Type: Bug >Reporter: Roshan Naik >Assignee: Roshan Naik > Fix For: v1.8.0 > > Attachments: FLUME-2530.patch, coverity.patch > > > A recent run of coverity on the Flume code base found some issues in various > components. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2616) Add Cassandra sink
[ https://issues.apache.org/jira/browse/FLUME-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2616: - Fix Version/s: v1.8.0 > Add Cassandra sink > -- > > Key: FLUME-2616 > URL: https://issues.apache.org/jira/browse/FLUME-2616 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Reporter: Santiago M. Mola >Assignee: Santiago M. Mola > Fix For: v1.8.0 > > Attachments: FLUME-2616-0.patch > > > A Cassandra sink would be a useful addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2855) Add kafka 0.9 support for Flume
[ https://issues.apache.org/jira/browse/FLUME-2855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno resolved FLUME-2855. -- Resolution: Duplicate > Add kafka 0.9 support for Flume > --- > > Key: FLUME-2855 > URL: https://issues.apache.org/jira/browse/FLUME-2855 > Project: Flume > Issue Type: Improvement > Components: Channel, Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Grigoriy Rozhkov > Attachments: FLUME-2821.patch > > > Kafka 0.9 already released and it introduces new clients API that not > compatible with old one. > Add support for Apache Flume to use new kafka consumer and producer API. Old > classes will not be removed to save backward compatibility with old Kafka > Source, Sink and Channel implementations. User will not need to change agent > configuration if wants to use old Kafka. Kafka API version will able to be > configured in agent configuration file. > 1. Add new Kafka Source implementation which uses new Kafka consumer API. > 2. Add new Kafka Sink implementation which uses new Kafka producer API. > 3. Add new Kafka Channel implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2935) Bump java target version to 1.7
[ https://issues.apache.org/jira/browse/FLUME-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349096#comment-15349096 ] Lior Zeno commented on FLUME-2935: -- I'll be much more verbose regarding the changes I've made in this patch. All the changes I've made are related to the build process and to the documentation. # Renamed README to README.textile in order to allow github to properly render the headings, i.e. instead of rendering h1 it will actually render a heading. In addition, I changed the requirement to JDK 1.7 instead of JDK 1.6. # FlumeUserGuide.rst: I changed the system requirements to Java 1.7 instead of 1.6 with a recommendation on 1.7. I did not add a recommendation for 1.8, since I do not see a reason for that. More over, I removed the note on the TailDirSource saying it requires 1.7, since it is the minimum required for the project. # flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml: This pom file had two identical dependencies, therefore I removed one of them (httpclient). This was generating a warning in the beginning of the build. # KafkaLocal.java: The tests threw a MethodNotFoundException, since there isn't a constructor for KafkaConfig with a java.util.properties parameter. Instead, I used the static method KafkaConfig.fromProps. # Main pom.xml: I changed the sourceJavaVersion and targetJavaVersion to 1.7. In addition, I removed the test dependency hadoop-minicluster since it is already required as a "normal" dependency. Maven was complaining about that as well. Let me know if you want to split these changes into multiple small issues, or we can deal with them in this issue since it's about making Flume build properly under 1.7. > Bump java target version to 1.7 > --- > > Key: FLUME-2935 > URL: https://issues.apache.org/jira/browse/FLUME-2935 > Project: Flume > Issue Type: Task >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > Attachments: FLUME-2395-0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2937) Integrate checkstyle
Lior Zeno created FLUME-2937: Summary: Integrate checkstyle Key: FLUME-2937 URL: https://issues.apache.org/jira/browse/FLUME-2937 Project: Flume Issue Type: Improvement Affects Versions: v1.8.0 Reporter: Lior Zeno Priority: Minor Fix For: v1.8.0 We should add the maven-checkstyle-plugin to the build process. This plugin can fail a build if the code does not honor the style of our project. This way we can make sure that we have one common style in the code. In addition, reviewers can focus on design, correctness, performance and other important coding aspects other than style issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2935) Bump java target version to 1.7
[ https://issues.apache.org/jira/browse/FLUME-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348294#comment-15348294 ] Lior Zeno commented on FLUME-2935: -- I attached a patch. We still need to modify Jenkins and the wiki page, e.g. the Development Environment page in the Developer section. https://cwiki.apache.org/confluence/display/FLUME/Development+Environment All tests passed, except for kafka source, elasticsearch and solr, but we have separate jira issues for these: FLUME-2936, FLUME-2921 and FLUME-2919 respectively. I tested on openjdk 1.7.0_95 64bit. Finally, I took the opportunity in fixed a few warnings that came up during the build by maven. > Bump java target version to 1.7 > --- > > Key: FLUME-2935 > URL: https://issues.apache.org/jira/browse/FLUME-2935 > Project: Flume > Issue Type: Task >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2936) KafkaSource tests arbitrarily fail
[
https://issues.apache.org/jira/browse/FLUME-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno reassigned FLUME-2936:
Assignee: Lior Zeno
> KafkaSource tests arbitrarily fail
> ---
>
> Key: FLUME-2936
> URL: https://issues.apache.org/jira/browse/FLUME-2936
> Project: Flume
> Issue Type: Bug
> Components: Test
>Affects Versions: v1.7.0
>Reporter: Lior Zeno
>Assignee: Lior Zeno
> Fix For: v1.7.0
>
> Attachments: FLUME-2936-0.patch
>
>
> KafkaSource tests are flaky, and tend to fail. I investigated this issue and
> discovered the following issues:
> * KafkaSource hangs in doStart at line 413:
> {code:title=KafkaSourceEmbeddedKafka.java|borderStyle=solid}
> it = consumer.poll(1000).iterator();
> {code}
> I fixed that by simply changing HOST in KafkaSourceEmbeddedKafka to localhost.
> * Out of memory exception. I fixed that by properly configuring MAVEN_OPTS,
> as stated in the documentation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2936) KafkaSource tests arbitrarily fail
[
https://issues.apache.org/jira/browse/FLUME-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2936:
-
Attachment: FLUME-2936-0.patch
> KafkaSource tests arbitrarily fail
> ---
>
> Key: FLUME-2936
> URL: https://issues.apache.org/jira/browse/FLUME-2936
> Project: Flume
> Issue Type: Bug
> Components: Test
>Affects Versions: v1.7.0
>Reporter: Lior Zeno
> Fix For: v1.7.0
>
> Attachments: FLUME-2936-0.patch
>
>
> KafkaSource tests are flaky, and tend to fail. I investigated this issue and
> discovered the following issues:
> * KafkaSource hangs in doStart at line 413:
> {code:title=KafkaSourceEmbeddedKafka.java|borderStyle=solid}
> it = consumer.poll(1000).iterator();
> {code}
> I fixed that by simply changing HOST in KafkaSourceEmbeddedKafka to localhost.
> * Out of memory exception. I fixed that by properly configuring MAVEN_OPTS,
> as stated in the documentation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-2936) KafkaSource tests arbitrarily fail
Lior Zeno created FLUME-2936:
Summary: KafkaSource tests arbitrarily fail
Key: FLUME-2936
URL: https://issues.apache.org/jira/browse/FLUME-2936
Project: Flume
Issue Type: Bug
Components: Test
Affects Versions: v1.7.0
Reporter: Lior Zeno
Fix For: v1.7.0
KafkaSource tests are flaky, and tend to fail. I investigated this issue and
discovered the following issues:
* KafkaSource hangs in doStart at line 413:
{code:title=KafkaSourceEmbeddedKafka.java|borderStyle=solid}
it = consumer.poll(1000).iterator();
{code}
I fixed that by simply changing HOST in KafkaSourceEmbeddedKafka to localhost.
* Out of memory exception. I fixed that by properly configuring MAVEN_OPTS, as
stated in the documentation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-2935) Bump java target version to 1.7
[ https://issues.apache.org/jira/browse/FLUME-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno reassigned FLUME-2935: Assignee: Lior Zeno > Bump java target version to 1.7 > --- > > Key: FLUME-2935 > URL: https://issues.apache.org/jira/browse/FLUME-2935 > Project: Flume > Issue Type: Task >Affects Versions: v1.7.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2935) Bump java target version to 1.7
[ https://issues.apache.org/jira/browse/FLUME-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343684#comment-15343684 ] Lior Zeno commented on FLUME-2935: -- This includes pom modification, documentation update, making sure that Jenkins is configured correctly with the right environment and jvm version, and finally, making sure that the project build succeeds and that all tests pass. > Bump java target version to 1.7 > --- > > Key: FLUME-2935 > URL: https://issues.apache.org/jira/browse/FLUME-2935 > Project: Flume > Issue Type: Task >Affects Versions: v1.7.0 >Reporter: Lior Zeno > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2807) Add a simple split interceptor
[ https://issues.apache.org/jira/browse/FLUME-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2807: - Fix Version/s: (was: v1.6.0) > Add a simple split interceptor > --- > > Key: FLUME-2807 > URL: https://issues.apache.org/jira/browse/FLUME-2807 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0, v1.7.0 >Reporter: seekerak > Labels: features, patch > Fix For: v1.7.0 > > Attachments: FLUME-2807.patch > > > a simple split interceptor , aims to deal with the situation as follows: > the source data like this: > “ > 1,tom,boy,13 > 2,lili,girl,14 > 3,jack,boy,10 > ... > ” > and i hope to sink source data into two different hdfs directories named by > boy and girl like this: > “ > hdfs:///sink/boy/ > hdfs:///sink/girl/ > ” > we can use this interceptor to accomplish this goal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2132) Exception while syncing from Flume to HDFS
[
https://issues.apache.org/jira/browse/FLUME-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2132:
-
Fix Version/s: (was: v1.3.0)
v1.7.0
> Exception while syncing from Flume to HDFS
> --
>
> Key: FLUME-2132
> URL: https://issues.apache.org/jira/browse/FLUME-2132
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.3.0
> Environment: Flume 1.3.0, Hadoop 1.2.0, 8GB RAM, Intel Pentium core 2
> duo
>Reporter: Divya R
> Labels: flume, hadoop
> Fix For: v1.7.0
>
>
> I'm running hadoop 1.2.0 and flume 1.3.0. Every thing works fine if its
> independently run. When I start my tomcat I get the below exception after
> some time.
> {quote}2013-07-17 12:40:35,640 (ResponseProcessor for block
> blk_5249456272858461891_436734) [WARN -
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:3015)]
> DFSOutputStream ResponseProcessor exception for block
> blk_5249456272858461891_436734java.net.SocketTimeoutException: 63000 millis
> timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/127.0.0.1:24433
> remote=/127.0.0.1:50010]
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at
> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:124)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2967){quote}
> {quote}2013-07-17 12:40:35,800 (hdfs-hdfs-write-roll-timer-0) [WARN -
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:277)]
> failed to close() HDFSWriter for file
> (hdfs://localhost:9000/flume/Broadsoft_App2/20130717/jboss/Broadsoft_App2.1374044838498.tmp).
> Exception follows.
> java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:3096)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2589)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2793){quote}
> Java snippet for Configuraion
> {quote}configuration.set("fs.default.name", "hdfs://localhost:9000");
> configuration.set("mapred.job.tracker", "hdfs://localhost:9000");{quote}
> I'm using a single datanode to read the files that where written to hdfs by
> flume, my java program just reads the files from hdfs to show it on the
> screen nothing much.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2882) Add Generic configuration provider
[ https://issues.apache.org/jira/browse/FLUME-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2882: - Fix Version/s: (was: v1.6.0) v1.7.0 > Add Generic configuration provider > -- > > Key: FLUME-2882 > URL: https://issues.apache.org/jira/browse/FLUME-2882 > Project: Flume > Issue Type: Improvement > Components: Configuration >Reporter: Enrique Ruiz Garcia > Fix For: v1.7.0 > > Attachments: FLUME-2882.patch > > > Add the ability to specify custom configuration provider to flume node (use > new optional '-confprovider' option to a specify class name that you can > implement from GenericConfigurationProvider class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2717) Add commons-io dependency into hadoop-2 profile to enable Flume 1.5 to support Hadoop 2.7
[ https://issues.apache.org/jira/browse/FLUME-2717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lior Zeno updated FLUME-2717: - Fix Version/s: (was: v1.5.0) v1.7.0 > Add commons-io dependency into hadoop-2 profile to enable Flume 1.5 to > support Hadoop 2.7 > - > > Key: FLUME-2717 > URL: https://issues.apache.org/jira/browse/FLUME-2717 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.5.0 >Reporter: sam liu >Assignee: li xiang > Fix For: v1.7.0 > > Attachments: FLUME-2717-001.patch > > > By default, for branch origin/flume-1.5, the hadoop2.version is 2.4.0. > However, if we want to use hadoop-2.7.0 in flume-1.5, some hadoop regarding > tests will fail with exception 'java.lang.NoClassDefFoundError: > org/apache/commons/io/Charsets'. > The missed class Charsets is a new class in commons-io 2.4 jar file which is > invoked by hadoop-2.7.0, however flume-1.5 depends on commons-io 2.1 which > does not include the class 'Charsets'. > Therefore the solution to enable flume-1.5 to support hadoop-2.7.0 is to add > commons-io 2.4 as a dependency into the hadoop-2 profile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2461) memoryChannel bytesRemaining counting error
[
https://issues.apache.org/jira/browse/FLUME-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lior Zeno updated FLUME-2461:
-
Fix Version/s: (was: v1.5.0.1)
v1.7.0
> memoryChannel bytesRemaining counting error
> ---
>
> Key: FLUME-2461
> URL: https://issues.apache.org/jira/browse/FLUME-2461
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.5.0.1
>Reporter: yangwei
>Priority: Minor
> Labels: patch
> Fix For: v1.7.0
>
> Attachments: FLUME-2461.patch
>
>
> In doRollback function putByteCounter permits are released by bytesRemaining.
> This is wrong for the below cases:
> In the doCommit function:
> 1)
> if(!bytesRemaining.tryAcquire(putByteCounter, keepAlive,
> TimeUnit.SECONDS)) {
> throw new ChannelException("Cannot commit transaction. Heap space "
> +
> "limit of " + byteCapacity + "reached. Please increase heap
> space" +
> " allocated to the channel as the sinks may not be keeping up " +
> "with the sources");
> }
> 2)
> if(!queueRemaining.tryAcquire(-remainingChange, keepAlive,
> TimeUnit.SECONDS)) {
> bytesRemaining.release(putByteCounter);
> throw new ChannelFullException("Space for commit to queue couldn't
> be acquired." +
> " Sinks are likely not keeping up with sources, or the buffer
> size is too tight");
> }
> When they throw ChannelException, bytesRemaining should not release any
> permits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
