[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739185#comment-14739185 ] Martin Bonica commented on FLUME-2225: -- Thank you for all your work implementing this. It looks like documentation for this feature didn't make it into the user guide for 1.5.0 or 1.6.0. I'd like to use this feature; could I ask for some documentation on how to use this? I apologize if this isn't the right place; I wasn't sure where else to ask. > Elasticsearch Sink for ES HTTP API > -- > > Key: FLUME-2225 > URL: https://issues.apache.org/jira/browse/FLUME-2225 > Project: Flume > Issue Type: New Feature >Affects Versions: v1.5.0 >Reporter: Otis Gospodnetic >Assignee: Pawel Rog > Fix For: v1.4.1, v1.5.0 > > Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, > FLUME-2225-5.patch, FLUME-2225-6.patch > > > Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use > the ES HTTP API, which is sometimes easier, and doesn't have issues around > client and server/cluster components using incompatible versions - currently, > both client and server/cluster need to be on the same version. > See > http://search-hadoop.com/m/k76HH9Te68/otis=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739203#comment-14739203 ] Roshan Naik commented on FLUME-2225: [~mbonica] canu open a new jira for this ? > Elasticsearch Sink for ES HTTP API > -- > > Key: FLUME-2225 > URL: https://issues.apache.org/jira/browse/FLUME-2225 > Project: Flume > Issue Type: New Feature >Affects Versions: v1.5.0 >Reporter: Otis Gospodnetic >Assignee: Pawel Rog > Fix For: v1.4.1, v1.5.0 > > Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, > FLUME-2225-5.patch, FLUME-2225-6.patch > > > Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use > the ES HTTP API, which is sometimes easier, and doesn't have issues around > client and server/cluster components using incompatible versions - currently, > both client and server/cluster need to be on the same version. > See > http://search-hadoop.com/m/k76HH9Te68/otis=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739216#comment-14739216 ] Martin Bonica commented on FLUME-2225: -- [~roshan_naik] I have created issue FLUME-2791 for this. > Elasticsearch Sink for ES HTTP API > -- > > Key: FLUME-2225 > URL: https://issues.apache.org/jira/browse/FLUME-2225 > Project: Flume > Issue Type: New Feature >Affects Versions: v1.5.0 >Reporter: Otis Gospodnetic >Assignee: Pawel Rog > Fix For: v1.4.1, v1.5.0 > > Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, > FLUME-2225-5.patch, FLUME-2225-6.patch > > > Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use > the ES HTTP API, which is sometimes easier, and doesn't have issues around > client and server/cluster components using incompatible versions - currently, > both client and server/cluster need to be on the same version. > See > http://search-hadoop.com/m/k76HH9Te68/otis=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048025#comment-14048025 ] Roshan Naik commented on FLUME-2225: Looks like this feature was not documented in the User guide. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Assignee: Pawel Rog Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch, FLUME-2225-6.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048030#comment-14048030 ] Pawel Rog commented on FLUME-2225: -- Right. Should I document it somehow? Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Assignee: Pawel Rog Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch, FLUME-2225-6.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941888#comment-13941888 ] Edward Sargisson commented on FLUME-2225: - Oh, I thought I'd replied 2 days ago but PEBKAC with using Review Board. I'm happy with the change - I think Hari, etc. should commit it. I could argue to adding something to the docs about strategies for de-duplication with elasticsearch - however, that's outside the scope of this work item. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942073#comment-13942073 ] Hari Shreedharan commented on FLUME-2225: - Ok, great. +1 then. I am going to run tests and commit. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942087#comment-13942087 ] Pawel Rog commented on FLUME-2225: -- But I hope you take the patch from review or I should upload that patch also here? Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942092#comment-13942092 ] Hari Shreedharan commented on FLUME-2225: - Is the version the jira outdated? Please upload it here to grant the copyright etc to ASF. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942498#comment-13942498 ] ASF subversion and git services commented on FLUME-2225: Commit e12f0a7a46752eff2ff3aaddfa0b93fb606074d7 in flume's branch refs/heads/trunk from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=e12f0a7 ] FLUME-2225. Elasticsearch Sink for ES HTTP API (Pawel Rog via Edward Sargisson, Hari Shreedharan) Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch, FLUME-2225-6.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942499#comment-13942499 ] ASF subversion and git services commented on FLUME-2225: Commit 042f53a5f45391a064aa96cba16d9f7f0866db1e in flume's branch refs/heads/flume-1.5 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=042f53a ] FLUME-2225. Elasticsearch Sink for ES HTTP API (Pawel Rog via Edward Sargisson, Hari Shreedharan) Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch, FLUME-2225-6.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942501#comment-13942501 ] Hari Shreedharan commented on FLUME-2225: - I committed this. Thanks Pawel for spending so much time and effort working on this. Thanks a lot Edward for your help reviewing it (gave you credit in the commit message too!). Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch, FLUME-2225-6.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942513#comment-13942513 ] Pawel Rog commented on FLUME-2225: -- Nice, Thanks for commiting this :-) Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch, FLUME-2225-6.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13942785#comment-13942785 ] Hudson commented on FLUME-2225: --- FAILURE: Integrated in flume-trunk #628 (See [https://builds.apache.org/job/flume-trunk/628/]) FLUME-2225. Elasticsearch Sink for ES HTTP API (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo/?p=flume.gita=commith=e12f0a7a46752eff2ff3aaddfa0b93fb606074d7) * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/RoundRobinList.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimestampedEvent.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchClient.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/client/RoundRobinListTest.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchSink.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchDynamicSerializer.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/NoSuchClientTypeException.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TimestampedEventTest.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/IndexNameBuilder.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilderTest.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchClientFactory.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java * flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchLogStashEventSerializer.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/client/TestElasticSearchRestClient.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkConstants.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/client/TestElasticSearchClientFactory.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/client/TestElasticSearchTransportClient.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchIndexRequestBuilderFactory.java Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Assignee: Pawel Rog Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch, FLUME-2225-6.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13941316#comment-13941316 ] Pawel Rog commented on FLUME-2225: -- Hi, Any news about the change? Mostly question to Edward about solution for handling error during bulk on ES Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937971#comment-13937971 ] Edward Sargisson commented on FLUME-2225: - Jeremy Karlson and I will take a look. Thanks for all your efforts Pawel! Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13938421#comment-13938421 ] Otis Gospodnetic commented on FLUME-2225: - Btw. FLUME- mentions Flume + Elasticsearch and duplication. I see this mentioned on the Review board. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937136#comment-13937136 ] Pawel Rog commented on FLUME-2225: -- Hi, I attach patch without Jest library. Only direct http library calls Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937261#comment-13937261 ] Hari Shreedharan commented on FLUME-2225: - If someone familiar with elastic search could review this patch, I will commit it. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch, FLUME-2225-5.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13936127#comment-13936127 ] Pawel Rog commented on FLUME-2225: -- Hi, I tried to use shading here. But I'm not sure if it is possible to shade dependency of dependency (Jest uses guava 14.0 ... ElasticsearchSink doesn't use it directly). I didn't find any examples how to shade dependencies of dependency. Do you have any idea? Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13936172#comment-13936172 ] Otis Gospodnetic commented on FLUME-2225: - 2 thoughts: # does Jest by some chance also work withe older Guava? # Flume does just one type of call to ES - to index docs, right? Maybe it would be much easier to then just do that manually with HttpClient and ditch Jest. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13932448#comment-13932448 ] Otis Gospodnetic commented on FLUME-2225: - [~hshreedharan] if you think shading is the way to go, I think we can do it. You're OK with that approach? Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13932546#comment-13932546 ] Hari Shreedharan commented on FLUME-2225: - At this point, if we really need this patch in, but not affect HDFS, I don't see another option - do you? We will need to shade the guava deps for this patch so elastic search sink uses guava 14.x while HDFS sink uses Guava 11.x. It might be a bit of a pain though, as it looks like it is a dependency introduced by this patch that requires the guava packages - so we'd need to shade guava in the dependency too. As I hear it, it is possible to do this in the binary, without modifying the dependencies of the source code, but just painful to do. So yeah, I am ok with it, as long as we can do it. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13931399#comment-13931399 ] Hari Shreedharan commented on FLUME-2225: - The guava upgrade in this patch seems like a problem. Maybe shading guava would help? Otherwise we'd have compatibility issues between Hadoop and Guava. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13913276#comment-13913276 ] Otis Gospodnetic commented on FLUME-2225: - [~hshreedharan] - I see another Guava upgrade to 14.x in FLUME-2286. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898136#comment-13898136 ] Edward Sargisson commented on FLUME-2225: - Hi [~prog], Let me acknowledge that it's super-frustrating to get code through the Flume review process. Thank you for your contributin. To replace Jest, can I suggest that there is Netty and Jackson JSON already in the dependencies. They are older versions but trustworthy. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898533#comment-13898533 ] Hari Shreedharan commented on FLUME-2225: - Sorry this is taking so long to be committed, but if what Edward suggested above is possible, then could we do that? Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898539#comment-13898539 ] Pawel Rog commented on FLUME-2225: -- Hi, OK I'll try to do it using httpclient or netty. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13897061#comment-13897061 ] Hari Shreedharan commented on FLUME-2225: - [~prog] - I am wary of bumping Guava from 11 to 14 since Guava breaks compatibility between releases. Hadoop currently depends on Guava 11.x and we should stay on that version to make sure we don't end up breaking some of the current components Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13895452#comment-13895452 ] Pawel commented on FLUME-2225: -- New patchset has been added. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13890996#comment-13890996 ] Pawel commented on FLUME-2225: -- Hi, I attached a new patchset. I'll analyze it and try to find something to change, but I'll be also glad if you can watch that patchset. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13885226#comment-13885226 ] Ashish Paliwal commented on FLUME-2225: --- [~prog_] I am reviewing it in chunks (short on time), overall I like the way the transports have been forked. Shall try to push more comments today. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13885256#comment-13885256 ] Pawel Rog commented on FLUME-2225: -- Thanks for that review. Now I have a lot of other work, so that's why I didn't send any patches for comments you've posted. When you make review of next chunk please write comments. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13885262#comment-13885262 ] Ashish Paliwal commented on FLUME-2225: --- Thanks! IMHO, better to wait for review comments from Edward before updating the patch. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13885704#comment-13885704 ] Edward Sargisson commented on FLUME-2225: - Hi Pawel, Thank you for submitting this patch; it clearly took a lot of work and should be useful to many other people. I've finished my review and added lots of comments. My biggest concern is that backwards compatibility with custom derived classes has been broken. We did extensive work on keeping backwards compatibility in previous versions; we should continue to keep this so that we do not needlessly break code. I look forward to seeing the results of your rework. Cheers, Edward Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13885743#comment-13885743 ] Pawel Rog commented on FLUME-2225: -- Hi, Thank you for that review. I have a lot of work to do there to fix all the things you wrote. Please clarify here if you mean that I just should repair the backwards compatibility in all places. Do I understand it correctly? Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13885822#comment-13885822 ] Edward Sargisson commented on FLUME-2225: - repair the backwards compatibility in all places Where our docs say that a custom class can be specified, then we should support that custom class in a backwards compatible fashion. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13885828#comment-13885828 ] Pawel Rog commented on FLUME-2225: -- Ok, so you only mena external/custom classes? Right? For a while I tought you suggest to make more changes. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13884249#comment-13884249 ] Ashish Paliwal commented on FLUME-2225: --- I can spend some time to review this. Given my ES experience is beginner, not sure how much I can help. Doc changes are missing, so may be worth adding in next patch. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13884330#comment-13884330 ] Edward Sargisson commented on FLUME-2225: - Pawel - Having the complete patch 1 in the review as you have it is fine. Ashish - sure! great! There's a lot to go through and I have a few minor concerns so hopefully I'll find some time today. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13883086#comment-13883086 ] Edward Sargisson commented on FLUME-2225: - I'm still working through reviewing this. However, could you please put the -1 patch into a review board please. Preferably as a diff from the -0 patch but cumulative would be okay too. Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13883106#comment-13883106 ] Paweł commented on FLUME-2225: -- Accidentally I pushed patch 1 to review instead of patch 0, but I cannot change a name of existing link. Should I send only patch 0 to review or just difference between patch 0 and patch 1? Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13873034#comment-13873034 ] Otis Gospodnetic commented on FLUME-2225: - Any chance this could be reviewed and committed before the next release? Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Affects Versions: v1.5.0 Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch, FLUME-2225-1.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (FLUME-2225) Elasticsearch Sink for ES HTTP API
[ https://issues.apache.org/jira/browse/FLUME-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863482#comment-13863482 ] Paweł commented on FLUME-2225: -- I think that discovery in http client should be configurable, so I'll prepare a patch later Elasticsearch Sink for ES HTTP API -- Key: FLUME-2225 URL: https://issues.apache.org/jira/browse/FLUME-2225 Project: Flume Issue Type: New Feature Reporter: Otis Gospodnetic Fix For: v1.4.1, v1.5.0 Attachments: FLUME-2225-0.patch Existing ElasticSearchSink uses ES TransportClient. As such, one cannot use the ES HTTP API, which is sometimes easier, and doesn't have issues around client and server/cluster components using incompatible versions - currently, both client and server/cluster need to be on the same version. See http://search-hadoop.com/m/k76HH9Te68/otissubj=Elasticsearch+sink+that+uses+HTTP+API -- This message was sent by Atlassian JIRA (v6.1.5#6160)
