[jira] (FLUME-3414) upgrade jackson due to CVE-2020-36518
[ https://issues.apache.org/jira/browse/FLUME-3414 ] Ralph Goers deleted comment on FLUME-3414: was (Author: jira-bot): Commit eaa1d29dd141c9db5bf6cfb8e689ca32b67859a2 in flume's branch refs/heads/trunk from Ralph Goers [ https://gitbox.apache.org/repos/asf?p=flume.git;h=eaa1d29d ] FLUME-3414 - Move elasticsearch sink to its own project > upgrade jackson due to CVE-2020-36518 > - > > Key: FLUME-3414 > URL: https://issues.apache.org/jira/browse/FLUME-3414 > Project: Flume > Issue Type: Bug >Reporter: PJ Fanning >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > https://github.com/advisories/GHSA-57j2-w4cx-62h2 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3417) Remove Elasticsearch sink that requires Elasticsearch 0.90.1
[ https://issues.apache.org/jira/browse/FLUME-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17517120#comment-17517120 ] Ralph Goers commented on FLUME-3417: Commit eaa1d29dd141c9db5bf6cfb8e689ca32b67859a2 in flume's branch refs/heads/trunk from Ralph Goers [ [https://gitbox.apache.org/repos/asf?p=flume.git;h=eaa1d29d] ] FLUME-3417 - Move elasticsearch sink to its own project > Remove Elasticsearch sink that requires Elasticsearch 0.90.1 > > > Key: FLUME-3417 > URL: https://issues.apache.org/jira/browse/FLUME-3417 > Project: Flume > Issue Type: Task > Components: Sinks+Sources >Affects Versions: 1.9.0 >Reporter: Ralph Goers >Priority: Major > Fix For: 1.10.0 > > > The elasticsearch sink is removed in 1.10.0. Flume 2.0 will be broken into > multiple repositories and support for a newer version of Elasticsearch will > be implemented there. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3414) upgrade jackson due to CVE-2020-36518
[ https://issues.apache.org/jira/browse/FLUME-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516687#comment-17516687 ] ASF subversion and git services commented on FLUME-3414: Commit eaa1d29dd141c9db5bf6cfb8e689ca32b67859a2 in flume's branch refs/heads/trunk from Ralph Goers [ https://gitbox.apache.org/repos/asf?p=flume.git;h=eaa1d29d ] FLUME-3414 - Move elasticsearch sink to its own project > upgrade jackson due to CVE-2020-36518 > - > > Key: FLUME-3414 > URL: https://issues.apache.org/jira/browse/FLUME-3414 > Project: Flume > Issue Type: Bug >Reporter: PJ Fanning >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > https://github.com/advisories/GHSA-57j2-w4cx-62h2 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Updated] (FLUME-3338) Doc Flume Recoverability with Kafka
[
https://issues.apache.org/jira/browse/FLUME-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ralph Goers updated FLUME-3338:
---
Fix Version/s: 1.10.0
> Doc Flume Recoverability with Kafka
> ---
>
> Key: FLUME-3338
> URL: https://issues.apache.org/jira/browse/FLUME-3338
> Project: Flume
> Issue Type: Improvement
> Components: Docs
>Affects Versions: 1.9.0
>Reporter: David Mollitor
>Priority: Minor
> Fix For: 1.10.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {quote}
> Recoverability
> The events are staged in the channel, which manages recovery from failure.
> Flume supports a durable file channel which is backed by the local file
> system. There’s also a memory channel which simply stores the events in an
> in-memory queue, which is faster but any events still left in the memory
> channel when an agent process dies can’t be recovered.
> https://flume.apache.org/FlumeUserGuide.html#recoverability
> {quote}
> Include information here regarding the Kafka channel
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org
For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Updated] (FLUME-3403) The parquet-avro version used by flume is 1.4.1, which is vulnerabel.
[ https://issues.apache.org/jira/browse/FLUME-3403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers updated FLUME-3403: --- Fix Version/s: 1.10.0 > The parquet-avro version used by flume is 1.4.1, which is vulnerabel. > - > > Key: FLUME-3403 > URL: https://issues.apache.org/jira/browse/FLUME-3403 > Project: Flume > Issue Type: Improvement > Components: Node >Affects Versions: 1.9.0 >Reporter: zhou yong >Priority: Blocker > Fix For: 1.10.0 > > > flume-ng-dist-1.9.0 requires the parquet-avro component, and the required > version is as follows: > > com.twitter > parquet-avro > 1.4.1 > > > The parquet-avro is maintained by apache from 1.6.0, but there are > vulnerabilities with each version. There is also a vulnerability in > parquet-avro version 1.4.1，as detailed : Improper Input Validation > vulnerability in Parquet-MR of Apache Parquet allows an attacker to DoS by > malicious Parquet files. This issue affects Apache Parquet-MR version 1.9.0 > and later versions. [https://nvd.nist.gov/vuln/detail/CVE-2021-41561] > Do you have any good solutions? -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Updated] (FLUME-3395) Fix for CVE-2021-44228
[ https://issues.apache.org/jira/browse/FLUME-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers updated FLUME-3395: --- Fix Version/s: 1.10.0 > Fix for CVE-2021-44228 > -- > > Key: FLUME-3395 > URL: https://issues.apache.org/jira/browse/FLUME-3395 > Project: Flume > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Blocker > Fix For: 1.10.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We reverted Log4J2 from 1.9.0, so this only affects trunk. Should be simple > enough to upgrade the log4j2 dependency to 2.15.0 and check tests are > passing. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Updated] (FLUME-3386) flume-ng-sdk uses vulnerable version of netty
[ https://issues.apache.org/jira/browse/FLUME-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers updated FLUME-3386: --- Fix Version/s: 1.10.0 > flume-ng-sdk uses vulnerable version of netty > - > > Key: FLUME-3386 > URL: https://issues.apache.org/jira/browse/FLUME-3386 > Project: Flume > Issue Type: Dependency upgrade >Affects Versions: 1.9.0 >Reporter: Lily Warner >Priority: Major > Fix For: 1.10.0 > > > Vulnerabilities: > * [https://nvd.nist.gov/vuln/detail/CVE-2019-16869] > * [https://nvd.nist.gov/vuln/detail/CVE-2019-20444] > * [https://nvd.nist.gov/vuln/detail/CVE-2019-20445] > * sonatype-2020-0103 > * sonatype-2020-0029 > Version to migrate to: 4.1.59 or above -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Closed] (FLUME-3368) Update Jackson to 2.9.10
[ https://issues.apache.org/jira/browse/FLUME-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers closed FLUME-3368. -- > Update Jackson to 2.9.10 > > > Key: FLUME-3368 > URL: https://issues.apache.org/jira/browse/FLUME-3368 > Project: Flume > Issue Type: Improvement >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Major > Fix For: 1.10.0 > > > Cross ref Jackson security problem: > https://github.com/FasterXML/jackson-databind/issues/2295 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Resolved] (FLUME-3368) Update Jackson to 2.9.10
[ https://issues.apache.org/jira/browse/FLUME-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers resolved FLUME-3368. Fix Version/s: 1.10.0 Resolution: Resolved > Update Jackson to 2.9.10 > > > Key: FLUME-3368 > URL: https://issues.apache.org/jira/browse/FLUME-3368 > Project: Flume > Issue Type: Improvement >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Major > Fix For: 1.10.0 > > > Cross ref Jackson security problem: > https://github.com/FasterXML/jackson-databind/issues/2295 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Commented] (FLUME-3368) Update Jackson to 2.9.10
[ https://issues.apache.org/jira/browse/FLUME-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17516681#comment-17516681 ] Ralph Goers commented on FLUME-3368: Jackson was subsequently modified to 2.13.2 > Update Jackson to 2.9.10 > > > Key: FLUME-3368 > URL: https://issues.apache.org/jira/browse/FLUME-3368 > Project: Flume > Issue Type: Improvement >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Major > > Cross ref Jackson security problem: > https://github.com/FasterXML/jackson-databind/issues/2295 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Reopened] (FLUME-3368) Update Jackson to 2.9.10
[ https://issues.apache.org/jira/browse/FLUME-3368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers reopened FLUME-3368: > Update Jackson to 2.9.10 > > > Key: FLUME-3368 > URL: https://issues.apache.org/jira/browse/FLUME-3368 > Project: Flume > Issue Type: Improvement >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Major > > Cross ref Jackson security problem: > https://github.com/FasterXML/jackson-databind/issues/2295 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Updated] (FLUME-3328) Fix Deprecated Properties table of HDFS Sink
[ https://issues.apache.org/jira/browse/FLUME-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers updated FLUME-3328: --- Fix Version/s: 1.10.0 > Fix Deprecated Properties table of HDFS Sink > > > Key: FLUME-3328 > URL: https://issues.apache.org/jira/browse/FLUME-3328 > Project: Flume > Issue Type: Bug > Components: Docs >Affects Versions: 1.9.0 >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Fix For: 1.10.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Updated] (FLUME-3316) Syslog Rfc3164Date test fails when the test date falls on a leap day
[ https://issues.apache.org/jira/browse/FLUME-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers updated FLUME-3316: --- Fix Version/s: 1.10.0 > Syslog Rfc3164Date test fails when the test date falls on a leap day > > > Key: FLUME-3316 > URL: https://issues.apache.org/jira/browse/FLUME-3316 > Project: Flume > Issue Type: Bug >Reporter: Peter Turcsanyi >Assignee: Peter Turcsanyi >Priority: Major > Fix For: 1.10.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Created] (FLUME-3417) Remove Elasticsearch sink that requires Elasticsearch 0.90.1
Ralph Goers created FLUME-3417: -- Summary: Remove Elasticsearch sink that requires Elasticsearch 0.90.1 Key: FLUME-3417 URL: https://issues.apache.org/jira/browse/FLUME-3417 Project: Flume Issue Type: Task Components: Sinks+Sources Affects Versions: 1.9.0 Reporter: Ralph Goers Fix For: 1.10.0 The elasticsearch sink is removed in 1.10.0. Flume 2.0 will be broken into multiple repositories and support for a newer version of Elasticsearch will be implemented there. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Closed] (FLUME-3021) flume Elasticsearch 5.0 not support
[ https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers closed FLUME-3021. -- > flume Elasticsearch 5.0 not support > --- > > Key: FLUME-3021 > URL: https://issues.apache.org/jira/browse/FLUME-3021 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: 1.7.0 >Reporter: tycho_yang >Assignee: Yong-Hao Zou >Priority: Major > Attachments: FLUME-3021-0.patch, FLUME-3021-1.patch, > FLUME-3021-Enable-ElasticSearch-6.1.x-support.patch, > image-2018-01-29-13-53-59-412.png, image-2018-01-31-12-00-06-854.png, > image-2018-01-31-12-00-50-050.png, image-2018-01-31-12-01-55-481.png, > image-2018-02-01-17-42-37-816.png, image-2018-02-01-17-44-24-082.png, > image-2018-02-01-17-45-54-668.png, image-2018-02-01-18-13-05-753.png, > screenshot-1.png, screenshot-2.png, screenshot-3.png > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
[jira] [Resolved] (FLUME-2816) Unable to get apache-flume-1.6.0 to work with ElasticSearch 2.0 beta 2 - Works with Elasticsearch 1.7.3
[
https://issues.apache.org/jira/browse/FLUME-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ralph Goers resolved FLUME-2816.
Resolution: Won't Fix
Elasticsearch 2.x has been EOL for 4 years. Support for a more recent version
of Elasticsearch is planned for Flume 2.0.
> Unable to get apache-flume-1.6.0 to work with ElasticSearch 2.0 beta 2 -
> Works with Elasticsearch 1.7.3
> ---
>
> Key: FLUME-2816
> URL: https://issues.apache.org/jira/browse/FLUME-2816
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Affects Versions: 1.6.0
> Environment: Centos 6
>Reporter: David Waugh
>Priority: Major
> Attachments: flume-1.6-elastic-2.0.patch
>
>
> Hello
> I am unable to get apache-flume-1.6.0 to work with ElasticSearch 2.0 beta 2.
> The same configuration works fine with Elasticsearch 1.7.3
> My flume-en.sh contains
> {noformat}
> Give Flume more memory and pre-allocate, enable remote monitoring via JMX
> export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
> Note that the Flume conf directory is always included in the classpath.
> FLUME_CLASSPATH="/usr/share/elasticsearch/lib/elasticsearch-2.0.0-beta2.jar"
> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre
> {noformat}
> I have copied the lucene-core-5.2.1.jar from /usr/share/elasticsearch/lib to
> /opt/flume/lib
> my elastic.conf file is:
> elastic.conf
> {noformat}
> #elastic.conf: A single-node Flume configuration
> #Reading avro files via spool dir and use the elastic sink
> #Name the components on this agent
> a1.sources = avrofld
> a1.sinks = k1
> a1.channels = c1
> #Describe/configure the folder with the avro files from Warehouse Connector
> a1.sources.avrofld.type = spooldir
> a1.sources.avrofld.channels = c1
> a1.sources.avrofld.spoolDir = /var/export
> a1.sources.avrofld.fileHeader = true
> a1.sources.avrofld.deserializer = avro
> Describe the elasticsearch sink
> a1.sinks.k1.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink
> a1.sinks.k1.hostNames = 127.0.0.1:9300
> a1.sinks.k1.indexName = sa
> a1.sinks.k1.indexType = data
> a1.sinks.k1.clusterName = elasticsearch
> a1.sinks.k1.batchSize = 1000
> a1.sinks.k1.ttl = 2d
> a1.sinks.k1.serializer =
> com.rsa.flume.serialization.FlumeAvroEventDeserializer
> #Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 100
> a1.channels.c1.transactionCapacity = 10
> #Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
> {noformat}
> Elasticsearch 2.0 is successfully started and listening on the port
> {noformat}
> [root@CentosB lib]# netstat -na |grep 900
> tcp 0 0 127.0.0.1:9200 0.0.0.0: LISTEN
> tcp 0 0 127.0.0.1:9300 0.0.0.0:* LISTEN
> {noformat}
> However when starting my flume I get the following error
> {noformat}
> [root@CentosB flume]# bin/flume-ng agent --conf conf --conf-file
> conf/elastic.conf --name a1 -Dflume.root.logger=DEBUG,console
> 2015-10-19 10:24:31,152 (lifecycleSupervisor-1-1) [ERROR -
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)]
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@5c5dc0a5 counterGroup:{
> name:null counters:{} } } - Exception follows.
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.(Ljava/lang/String;I)V
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.(ElasticSearchTransportClient.java:77)
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Th
[jira] [Closed] (FLUME-2816) Unable to get apache-flume-1.6.0 to work with ElasticSearch 2.0 beta 2 - Works with Elasticsearch 1.7.3
[
https://issues.apache.org/jira/browse/FLUME-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ralph Goers closed FLUME-2816.
--
> Unable to get apache-flume-1.6.0 to work with ElasticSearch 2.0 beta 2 -
> Works with Elasticsearch 1.7.3
> ---
>
> Key: FLUME-2816
> URL: https://issues.apache.org/jira/browse/FLUME-2816
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Affects Versions: 1.6.0
> Environment: Centos 6
>Reporter: David Waugh
>Priority: Major
> Attachments: flume-1.6-elastic-2.0.patch
>
>
> Hello
> I am unable to get apache-flume-1.6.0 to work with ElasticSearch 2.0 beta 2.
> The same configuration works fine with Elasticsearch 1.7.3
> My flume-en.sh contains
> {noformat}
> Give Flume more memory and pre-allocate, enable remote monitoring via JMX
> export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
> Note that the Flume conf directory is always included in the classpath.
> FLUME_CLASSPATH="/usr/share/elasticsearch/lib/elasticsearch-2.0.0-beta2.jar"
> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre
> {noformat}
> I have copied the lucene-core-5.2.1.jar from /usr/share/elasticsearch/lib to
> /opt/flume/lib
> my elastic.conf file is:
> elastic.conf
> {noformat}
> #elastic.conf: A single-node Flume configuration
> #Reading avro files via spool dir and use the elastic sink
> #Name the components on this agent
> a1.sources = avrofld
> a1.sinks = k1
> a1.channels = c1
> #Describe/configure the folder with the avro files from Warehouse Connector
> a1.sources.avrofld.type = spooldir
> a1.sources.avrofld.channels = c1
> a1.sources.avrofld.spoolDir = /var/export
> a1.sources.avrofld.fileHeader = true
> a1.sources.avrofld.deserializer = avro
> Describe the elasticsearch sink
> a1.sinks.k1.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink
> a1.sinks.k1.hostNames = 127.0.0.1:9300
> a1.sinks.k1.indexName = sa
> a1.sinks.k1.indexType = data
> a1.sinks.k1.clusterName = elasticsearch
> a1.sinks.k1.batchSize = 1000
> a1.sinks.k1.ttl = 2d
> a1.sinks.k1.serializer =
> com.rsa.flume.serialization.FlumeAvroEventDeserializer
> #Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 100
> a1.channels.c1.transactionCapacity = 10
> #Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
> {noformat}
> Elasticsearch 2.0 is successfully started and listening on the port
> {noformat}
> [root@CentosB lib]# netstat -na |grep 900
> tcp 0 0 127.0.0.1:9200 0.0.0.0: LISTEN
> tcp 0 0 127.0.0.1:9300 0.0.0.0:* LISTEN
> {noformat}
> However when starting my flume I get the following error
> {noformat}
> [root@CentosB flume]# bin/flume-ng agent --conf conf --conf-file
> conf/elastic.conf --name a1 -Dflume.root.logger=DEBUG,console
> 2015-10-19 10:24:31,152 (lifecycleSupervisor-1-1) [ERROR -
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)]
> Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@5c5dc0a5 counterGroup:{
> name:null counters:{} } } - Exception follows.
> java.lang.NoSuchMethodError:
> org.elasticsearch.common.transport.InetSocketTransportAddress.(Ljava/lang/String;I)V
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143)
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.(ElasticSearchTransportClient.java:77)
> at
> org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48)
> at
> org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Full Log Below
> {noformat}
> Info: Sourcing environment configuration script /opt/flume/conf/flume-env.sh
> Inf
[jira] [Updated] (FLUME-3021) flume Elasticsearch 5.0 not support
[ https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ralph Goers updated FLUME-3021: --- Fix Version/s: (was: 1.10.0) Resolution: Won't Fix Status: Resolved (was: Patch Available) Elasticsearch 5.x has been EOL for over 3 years. The current plan is to implement support for a supported version of Elasticsearch in Flume 2.0 > flume Elasticsearch 5.0 not support > --- > > Key: FLUME-3021 > URL: https://issues.apache.org/jira/browse/FLUME-3021 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: 1.7.0 >Reporter: tycho_yang >Assignee: Yong-Hao Zou >Priority: Major > Attachments: FLUME-3021-0.patch, FLUME-3021-1.patch, > FLUME-3021-Enable-ElasticSearch-6.1.x-support.patch, > image-2018-01-29-13-53-59-412.png, image-2018-01-31-12-00-06-854.png, > image-2018-01-31-12-00-50-050.png, image-2018-01-31-12-01-55-481.png, > image-2018-02-01-17-42-37-816.png, image-2018-02-01-17-44-24-082.png, > image-2018-02-01-17-45-54-668.png, image-2018-02-01-18-13-05-753.png, > screenshot-1.png, screenshot-2.png, screenshot-3.png > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@flume.apache.org For additional commands, e-mail: issues-h...@flume.apache.org
