[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691041#comment-15691041
 ] 

Joseph Witt commented on NIFI-3011:
---

certainly.

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Randy Bovay (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691030#comment-15691030
 ] 

Randy Bovay commented on NIFI-3011:
---

Joe,
Thanks for adjusting those items to Improvements.
NIFI-3094 | Bulk API Should accept a list of Hostnames
NIFI-3095 | Bulk API - Add expression language support to 'Elasticsearch Hosts' 
and 'Cluster Name' fields

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690990#comment-15690990
 ] 

Joseph Witt commented on NIFI-3011:
---

[~randy_b] Please go ahead and file one or two JIRAs as enhancements to the 
processor. 

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Randy Bovay (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690965#comment-15690965
 ] 

Randy Bovay commented on NIFI-3011:
---

There are 2 enhancements that would improve this processor. 
1) We'd like to have expression language support for ElasticSearch Hosts, and 
ClusterName field. This is good for developers, and allows us to vary our 
targets per data center and environment. Otherwise we would have to deploy code 
and then manually update it.
2) Also, the bulk transport clients should allow for configuring multiple hosts 
in the elasticsearch hosts field. This allows us to have a Primary, and many 
secondary lists of servers. So that if one goes offline, the processor will 
connect to a backup to retrieve the routing table information.
This shows how to set multiple addresses. 
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684746#comment-15684746
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1233


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684744#comment-15684744
 ] 

ASF subversion and git services commented on NIFI-3011:
---

Commit dba9d5924a29d42f9f914320f6215a1849bbb54f in nifi's branch 
refs/heads/master from [~mattyb149]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=dba9d59 ]

NIFI-3011: Added Elasticsearch5 processors

This closes #1233

Signed-off-by: jpercivall 


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684697#comment-15684697
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1233
  
+1

Visually verified code and did a contrib check build. Ran a stand-alone 
instance hitting an ES 5 cluster with and without the security x-pack. All 
comments and L changes have been addressed. Thanks @mattyb149 I will squash 
and merge.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684438#comment-15684438
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1233
  
@mattyb149 ran into two weird errors while testing the lastest update. 

First, I ran into this when I attempted to do a request with the wrong 
attribute name (and thus there were no actions added to the "bulk":
```
016-11-21 13:40:02,766 ERROR [Timer-Driven Process Thread-10] 
o.a.n.p.elasticsearch.PutElasticsearch5 
org.elasticsearch.action.ActionRequestValidationException: Validation 
Failed: 1: no requests added;
at 
org.elasticsearch.action.ValidateActions.addValidationError(ValidateActions.java:29)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.action.bulk.BulkRequest.validate(BulkRequest.java:530) 
~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:46)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.client.transport.support.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:63)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:233)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:63)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:326)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:67) 
~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.client.support.AbstractClient$1.doExecute(AbstractClient.java:1754)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
 ~[elasticsearch-5.0.1.jar:5.0.1]
at 
org.apache.nifi.processors.elasticsearch.PutElasticsearch5.onTrigger(PutElasticsearch5.java:230)
 ~[nifi-elasticsearch-5-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
 [nifi-api-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_74]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_74]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_74]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
```


Second a bit more concerning, I saw when I tried to "update" a group of 
flowfile. All of which didn't exist already:
```

 PutElasticsearch5[id=b1fc4435-e5c1-153e-b5dc-61b9af32ef03] Failed to 
insert into Elasticsearch due to Index: 50, Size: 50, transferring to failure: 
java.lang.IndexOutOfBoundsException: Index: 50, Size: 50
2016-11-21 13:50:23,985 ERROR [Timer-Driven Process Thread-10] 
o.a.n.p.elasticsearch.PutElasticsearch5 
java.lang.IndexOutOfBoundsException: Index: 50, Size: 50
at java.util.LinkedList.checkElementIndex(LinkedList.java:555) 
~[na:1.8.0_74]
at java.util.LinkedList.get(LinkedList.java:476) ~[na:1.8.0_74]
at 

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684427#comment-15684427
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88963385
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearch5.java
 ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.elasticsearch.ElasticsearchTimeoutException;
+import org.elasticsearch.action.get.GetRequestBuilder;
+import org.elasticsearch.action.get.GetResponse;
+import org.elasticsearch.client.transport.NoNodeAvailableException;
+import org.elasticsearch.node.NodeClosedException;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve. If the cluster has been 
configured for authorization and/or secure "
++ "transport (SSL/TLS), and the X-Pack plugin is available, secure 
connections can be made. This processor "
++ "supports Elasticsearch 5.x clusters.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearch5 extends 
AbstractElasticsearch5TransportClientProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683972#comment-15683972
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88927684
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearch5.java
 ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.elasticsearch.ElasticsearchTimeoutException;
+import org.elasticsearch.action.get.GetRequestBuilder;
+import org.elasticsearch.action.get.GetResponse;
+import org.elasticsearch.client.transport.NoNodeAvailableException;
+import org.elasticsearch.node.NodeClosedException;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve. If the cluster has been 
configured for authorization and/or secure "
++ "transport (SSL/TLS), and the X-Pack plugin is available, secure 
connections can be made. This processor "
++ "supports Elasticsearch 5.x clusters.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearch5 extends 
AbstractElasticsearch5TransportClientProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document 

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683956#comment-15683956
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88924216
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -229,8 +229,122 @@ The following binary components are provided under 
the Apache Software License v
 
   (ASLv2) The Netty Project
   The following NOTICE information applies:
-The Netty Project
-Copyright 2011 The Netty Project
+
+The Netty Project
--- End diff --

This change is not reflected in the NiFi-assembly nar


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683954#comment-15683954
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88924576
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -242,11 +356,66 @@ The following binary components are provided under 
the Apache Software License v
   Apache Commons CLI
   Copyright 2001-2016 The Apache Software Foundation
 
+  (ASLv2) Apache Commons Logging
+The following NOTICE information applies:
+
+  Apache Commons Logging
+  Copyright 2003-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+
+  (ASLv2) Apache Commons Codec
+The following NOTICE information applies:
+  Apache Commons Codec
+  Copyright 2002-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+   src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java
+   contains test data from http://aspell.net/test/orig/batch0.tab.
+   Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org)
+
+   
===
+
+   The content of package org.apache.commons.codec.language.bm has 
been translated
+   from the original php source code available at 
http://stevemorse.org/phoneticinfo.htm
+   with permission from the original authors.
+   Original source copyright:
+   Copyright (c) 2008 Alexander Beider & Stephen P. Morse.
+
+  (ASLv2) Apache Commons Lang
+  The following NOTICE information applies:
+Apache Commons Lang
+Copyright 2001-2015 The Apache Software Foundation
+
+This product includes software from the Spring Framework,
+under the Apache License 2.0 (see: 
StringUtils.containsWhitespace())
+
+  (ASLv2) Apache Http Components
--- End diff --

"HttpComponents" is one word


https://github.com/apache/nifi/blob/b026f0bebe737aa62ab7de48a776d851807b1350/nifi-assembly/NOTICE#L111-L111
https://hc.apache.org/


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683953#comment-15683953
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88924805
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -242,11 +356,66 @@ The following binary components are provided under 
the Apache Software License v
   Apache Commons CLI
   Copyright 2001-2016 The Apache Software Foundation
 
+  (ASLv2) Apache Commons Logging
+The following NOTICE information applies:
+
+  Apache Commons Logging
+  Copyright 2003-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+
+  (ASLv2) Apache Commons Codec
+The following NOTICE information applies:
+  Apache Commons Codec
+  Copyright 2002-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+   src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java
+   contains test data from http://aspell.net/test/orig/batch0.tab.
+   Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org)
+
+   
===
+
+   The content of package org.apache.commons.codec.language.bm has 
been translated
+   from the original php source code available at 
http://stevemorse.org/phoneticinfo.htm
+   with permission from the original authors.
+   Original source copyright:
+   Copyright (c) 2008 Alexander Beider & Stephen P. Morse.
+
+  (ASLv2) Apache Commons Lang
+  The following NOTICE information applies:
+Apache Commons Lang
+Copyright 2001-2015 The Apache Software Foundation
+
+This product includes software from the Spring Framework,
+under the Apache License 2.0 (see: 
StringUtils.containsWhitespace())
+
+  (ASLv2) Apache Http Components
+The following NOTICE information applies:
+  Apache HttpComponents Client
+  Copyright 1999-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+  (ASLv2) Apache Log4J
+The following NOTICE information applies:
+  Apache log4j
+  Copyright 2010 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+
 *
 Public Domain
--- End diff --

You can remove this section now that there are no libraries under the 
Public domain included


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683957#comment-15683957
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88924961
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -242,11 +356,66 @@ The following binary components are provided under 
the Apache Software License v
   Apache Commons CLI
--- End diff --

This was supposed to be removed.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683955#comment-15683955
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88926499
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearch5.java
 ---
@@ -227,6 +227,7 @@ public void process(final InputStream in) throws 
IOException {
 session.transfer(flowFile, REL_FAILURE);
 
 } else {
+session.getProvenanceReporter().send(flowFile, 
response.remoteAddress().getAddress());
--- End diff --

This will only emit provenance event if "response.hasFailures" is true. If 
there are not failures this won't get hit.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678276#comment-15678276
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88766699
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678274#comment-15678274
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r8876
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677430#comment-15677430
 ] 

Joseph Percivall commented on NIFI-3011:


Adding a fix version of 1.1.0 as there is minimal work left (just need to add 
L)

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675691#comment-15675691
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88597130
  
--- Diff: nifi-nar-bundles/nifi-elasticsearch-5-bundle/pom.xml ---
@@ -0,0 +1,44 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-nar-bundles
+1.1.0-SNAPSHOT
+
+
+org.apache.nifi
+nifi-elasticsearch-5-bundle
+pom
+
+
+6.3.0
--- End diff --

It has to be in both the processor POM and NAR POM, so when the two bundles 
were different, I included it in each of the parent bundle POMs. However, I'm 
consolidating the bundles (per your suggestion a la Kafka), and now it can't be 
set in the parent POM, now it will have to be set in each NAR/processors pair. 
This is due to a higher parent POM setting lucene.version for repository stuff, 
and the ES POM using that property to choose a version of Lucene (for some 
reason it's not hard-coded in the published POM). Strange but true :)


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674596#comment-15674596
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88524040
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/LICENSE
 ---
@@ -0,0 +1,262 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other 
modifications
+  represent, as a whole, an original work of authorship. For the 
purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces 
of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright 
owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control 
systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, 

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674594#comment-15674594
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88527105
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

As well as: org.apache.logging.log4j:log4j-core:jar:2.7:compile


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674590#comment-15674590
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88527446
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674593#comment-15674593
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88527394
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674589#comment-15674589
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88520091
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
--- End diff --

Commons lang is brought in through the processor utils and it's notice 
needs to be added too.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674591#comment-15674591
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88525935
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

ES 5 brings in "io.netty:netty:jar:3.10.6.Final:compile" which is ASLv2 
licensed but has the following NOTICE file which is much more than what you 
have listed below.

https://github.com/netty/netty/blob/netty-3.10.6.Final/NOTICE.txt


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674588#comment-15674588
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88526691
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

Also: (org.apache.httpcomponents:httpcore:jar:4.4.1:compile - omitted for 
conflict with 4.4.5)


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674595#comment-15674595
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88526373
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

As well as "(commons-codec:commons-codec:jar:1.10:compile - version managed 
from 1.9; omitted for duplicate)".


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674592#comment-15674592
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88526308
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

It also brings in "commons-logging:commons-logging:jar:1.2:compile - 
omitted for conflict with 1.1.3)", the notice information will need to be added.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673982#comment-15673982
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88475057
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+abstract class AbstractElasticsearch5TransportClientProcessor extends 
AbstractElasticsearch5Processor {
+
+/**
+ * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
+ */
+private static final Validator HOSTNAME_PORT_VALIDATOR = (subject, 
input, context) -> {
+final List esList = Arrays.asList(input.split(","));
+for (String hostnamePort : esList) {
+String[] addresses = hostnamePort.split(":");
+// Protect against invalid input like http://127.0.0.1:9300 
(URL scheme should not be there)
+if (addresses.length != 2) {
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Must be in hostname:port form (no scheme such as 
http://;).valid(false).build();
+}
+}
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Valid cluster definition").valid(true).build();
+};
+
+protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
+.name("el5-cluster-name")
+.displayName("Cluster Name")
+.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("elasticsearch")
+.build();
+
+protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
+.name("el5-hosts")
+.displayName("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
++ "host1:port,host2:port,  For example 
testcluster:9300. This processor uses the Transport Client to "
++ "connect to hosts. The default transport client port 
is 9300.")
+.required(true)
+

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15672259#comment-15672259
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1233
  
I just confirmed functionality with ES 5.0.1 with and without x-pack 
security. Once the issues commented are addressed it should be good to go.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15672052#comment-15672052
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/1233
  
Also, both processors do not emit any provenance events. This should be 
added.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671831#comment-15671831
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88337395
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+abstract class AbstractElasticsearch5TransportClientProcessor extends 
AbstractElasticsearch5Processor {
+
+/**
+ * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
+ */
+private static final Validator HOSTNAME_PORT_VALIDATOR = (subject, 
input, context) -> {
+final List esList = Arrays.asList(input.split(","));
+for (String hostnamePort : esList) {
+String[] addresses = hostnamePort.split(":");
+// Protect against invalid input like http://127.0.0.1:9300 
(URL scheme should not be there)
+if (addresses.length != 2) {
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Must be in hostname:port form (no scheme such as 
http://;).valid(false).build();
+}
+}
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Valid cluster definition").valid(true).build();
+};
+
+protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
+.name("el5-cluster-name")
+.displayName("Cluster Name")
+.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("elasticsearch")
+.build();
+
+protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
+.name("el5-hosts")
+.displayName("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
++ "host1:port,host2:port,  For example 
testcluster:9300. This processor uses the Transport Client to "
++ "connect to hosts. The default transport client port 
is 9300.")
+.required(true)
+

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671835#comment-15671835
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88329420
  
--- Diff: nifi-nar-bundles/pom.xml ---
@@ -54,6 +54,7 @@
 nifi-html-bundle
 nifi-scripting-bundle
 nifi-elasticsearch-bundle
--- End diff --

Could this be changed to match how the Kafka bundle does version hierarchy? 


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671829#comment-15671829
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88329905
  
--- Diff: nifi-nar-bundles/nifi-elasticsearch-5-bundle/pom.xml ---
@@ -0,0 +1,44 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-nar-bundles
+1.1.0-SNAPSHOT
+
+
+org.apache.nifi
+nifi-elasticsearch-5-bundle
+pom
+
+
+6.3.0
--- End diff --

nit-pick, "lucene.version" is defined here but "slf4jversion" and 
"es.version" are defined in the processor pom. Is there a reason for that or 
can it be moved to the processor pom to match?


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671833#comment-15671833
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88337612
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5Processor.java
 ---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A base class for all Elasticsearch processors
+ */
+public abstract class AbstractElasticsearch5Processor extends 
AbstractProcessor {
+
+public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("el5-ssl-context-service")
+.displayName("SSL Context Service")
+.description("The SSL Context Service used to provide client 
certificate information for TLS/SSL "
++ "connections. This service only applies if the 
Elasticsearch endpoint(s) have been secured with TLS/SSL.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("el5-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("el5-username")
+.displayName("Username")
+.description("Username to access the Elasticsearch cluster")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("el5-password")
+.displayName("Password")
+.description("Password to access the Elasticsearch cluster")
+.required(false)
+.sensitive(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected abstract void createElasticsearchClient(ProcessContext 
context) throws ProcessException;
+
+@Override
+protected Collection 
customValidate(ValidationContext validationContext) {
+Set results = new HashSet<>();
+
+// Ensure that if username or password is set, then the other is 
too
+Map propertyMap = 
validationContext.getProperties();
+if (StringUtils.isEmpty(propertyMap.get(USERNAME)) != 
StringUtils.isEmpty(propertyMap.get(PASSWORD))) {
--- End diff --

In addition to this, should do a check that Xpack is at least set if the 
security properties are set (since they won't/can't be used if xpack isn't set)


> Support Elasticsearch 5.0 for Put/FetchElasticsearch 

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671834#comment-15671834
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88341976
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/pom.xml
 ---
@@ -0,0 +1,111 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+nifi-elasticsearch-5-bundle
+org.apache.nifi
+1.1.0-SNAPSHOT
+
+
+nifi-elasticsearch-5-processors
+jar
+
+
+2.7
+5.0.0
+
+
+
+
+org.apache.nifi
+nifi-api
+provided
+
+
+org.apache.nifi
+nifi-properties
+provided
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+org.apache.lucene
+lucene-core
+${lucene.version}
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.elasticsearch.client
+transport
+${es.version}
+
+
+com.squareup.okhttp3
+okhttp
+3.3.1
+
+
+org.apache.nifi
+nifi-ssl-context-service-api
+
+
+commons-io
+commons-io
+
+
+org.codehaus.jackson
+jackson-mapper-asl
--- End diff --

Same comment as lucene, can't find where this is used.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671832#comment-15671832
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88342854
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearch5.java
 ---
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.elasticsearch.ElasticsearchTimeoutException;
+import org.elasticsearch.action.bulk.BulkItemResponse;
+import org.elasticsearch.action.bulk.BulkRequestBuilder;
+import org.elasticsearch.action.bulk.BulkResponse;
+
+import org.elasticsearch.client.transport.NoNodeAvailableException;
+import org.elasticsearch.node.NodeClosedException;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5","insert", "update", "write", 
"put"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document. If the 
cluster has been configured for authorization "
++ "and/or secure transport (SSL/TLS), and the X-Pack plugin is 
available, secure connections can be made. This processor "
++ "supports Elasticsearch 5.x clusters.")
+public class PutElasticsearch5 extends 
AbstractElasticsearch5TransportClientProcessor {
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("el5-put-id-attribute")
+.displayName("Identifier Attribute")
+   

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671828#comment-15671828
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88330726
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+abstract class AbstractElasticsearch5TransportClientProcessor extends 
AbstractElasticsearch5Processor {
+
+/**
+ * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
+ */
+private static final Validator HOSTNAME_PORT_VALIDATOR = (subject, 
input, context) -> {
+final List esList = Arrays.asList(input.split(","));
+for (String hostnamePort : esList) {
+String[] addresses = hostnamePort.split(":");
+// Protect against invalid input like http://127.0.0.1:9300 
(URL scheme should not be there)
+if (addresses.length != 2) {
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Must be in hostname:port form (no scheme such as 
http://;).valid(false).build();
+}
+}
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Valid cluster definition").valid(true).build();
+};
+
+protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
+.name("el5-cluster-name")
+.displayName("Cluster Name")
+.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("elasticsearch")
+.build();
+
+protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
+.name("el5-hosts")
+.displayName("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
++ "host1:port,host2:port,  For example 
testcluster:9300. This processor uses the Transport Client to "
++ "connect to hosts. The default transport client port 
is 9300.")
+.required(true)
+

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671830#comment-15671830
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88341802
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/pom.xml
 ---
@@ -0,0 +1,111 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+nifi-elasticsearch-5-bundle
+org.apache.nifi
+1.1.0-SNAPSHOT
+
+
+nifi-elasticsearch-5-processors
+jar
+
+
+2.7
+5.0.0
+
+
+
+
+org.apache.nifi
+nifi-api
+provided
+
+
+org.apache.nifi
+nifi-properties
+provided
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+org.apache.lucene
+lucene-core
+${lucene.version}
--- End diff --

Where is lucene used in this module? I did a search and only came up with 
this pom


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671836#comment-15671836
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88341213
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/pom.xml
 ---
@@ -0,0 +1,111 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+nifi-elasticsearch-5-bundle
+org.apache.nifi
+1.1.0-SNAPSHOT
+
+
+nifi-elasticsearch-5-processors
+jar
+
+
+2.7
+5.0.0
+
+
+
+
+org.apache.nifi
+nifi-api
+provided
+
+
+org.apache.nifi
+nifi-properties
+provided
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+org.apache.lucene
+lucene-core
+${lucene.version}
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.elasticsearch.client
+transport
+${es.version}
+
+
+com.squareup.okhttp3
--- End diff --

I don't think this dep is needed


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671827#comment-15671827
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88333819
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+abstract class AbstractElasticsearch5TransportClientProcessor extends 
AbstractElasticsearch5Processor {
+
+/**
+ * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
+ */
+private static final Validator HOSTNAME_PORT_VALIDATOR = (subject, 
input, context) -> {
+final List esList = Arrays.asList(input.split(","));
+for (String hostnamePort : esList) {
+String[] addresses = hostnamePort.split(":");
+// Protect against invalid input like http://127.0.0.1:9300 
(URL scheme should not be there)
+if (addresses.length != 2) {
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Must be in hostname:port form (no scheme such as 
http://;).valid(false).build();
+}
+}
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Valid cluster definition").valid(true).build();
+};
+
+protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
+.name("el5-cluster-name")
+.displayName("Cluster Name")
+.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("elasticsearch")
+.build();
+
+protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
+.name("el5-hosts")
+.displayName("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
++ "host1:port,host2:port,  For example 
testcluster:9300. This processor uses the Transport Client to "
++ "connect to hosts. The default transport client port 
is 9300.")
+.required(true)
+

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671837#comment-15671837
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88340765
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearch5.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.elasticsearch.ElasticsearchTimeoutException;
+import org.elasticsearch.action.get.GetRequestBuilder;
+import org.elasticsearch.action.get.GetResponse;
+import org.elasticsearch.client.transport.NoNodeAvailableException;
+import org.elasticsearch.node.NodeClosedException;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve. If the cluster has been 
configured for authorization and/or secure "
++ "transport (SSL/TLS), and the X-Pack plugin is available, secure 
connections can be made. This processor "
++ "supports Elasticsearch 5.x clusters.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearch5 extends 
AbstractElasticsearch5TransportClientProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671455#comment-15671455
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88319339
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5Processor.java
 ---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A base class for all Elasticsearch processors
+ */
+public abstract class AbstractElasticsearch5Processor extends 
AbstractProcessor {
+
+public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("el5-ssl-context-service")
+.displayName("SSL Context Service")
+.description("The SSL Context Service used to provide client 
certificate information for TLS/SSL "
++ "connections. This service only applies if the 
Elasticsearch endpoint(s) have been secured with TLS/SSL.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
--- End diff --

I've received comments on making these properties accept Expression 
Language where possible (to leverage the Variable Registry), will make the 
changes.


> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671062#comment-15671062
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/1233

NIFI-3011: Added Elasticsearch5 processors



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-3011

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1233.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1233


commit 7ad90fc793a1294f6ce9d6aa4780e5432fa92a42
Author: Matt Burgess 
Date:   2016-11-16T17:26:26Z

NIFI-3011: Added Elasticsearch5 processors




> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664940#comment-15664940
 ] 

Andy LoPresto commented on NIFI-3011:
-

It appears it does not mention it in the main X-Pack documentation, but in the 
[advanced 
settings|https://www.elastic.co/guide/en/x-pack/current/security-settings.html#_java_keystore_files],
 there does appear to still be an option for keystore and truststore files and 
passwords. In addition, it appears there is an option for 
{{xpack.ssl.key_passphrase}}, which would allow a password to be used with 
{{xpack.ssl.key}} as mentioned above. 

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664929#comment-15664929
 ] 

Andy LoPresto commented on NIFI-3011:
-

After discussing with [~mattyb149], it appears the way X-Pack (the new client) 
performs TLS configuration is less secure than it previously was. The old 
client lib accepted Java keystore and truststore paths with their respective 
passwords, providing a more secure approach compared to the new client lib 
which wants the raw private key in PEM encoded format, the server certificate 
in PEM format, and the CA certificate(s) in PEM format. This means that the 
server private key (as they refer to it, the *node private key* or 
{{xpack.ssl.key}} is no longer password-protected and is just stored on disk, 
accessible to anyone with OS-level access. 

In fact, the 
[documentation|https://www.elastic.co/guide/en/x-pack/current/ssl-tls.html] was 
clearly not updated completely, as it still refers to "keystore" and 
"password(s)" immediately above. 

{quote}
To enable SSL, make the following changes in elasticsearch.yml:

Specify the location of the node’s keystore and the password(s) needed to 
access the node’s certificate. For example:

{code}
xpack.ssl.key: /home/es/config/x-pack/node01.key 
xpack.ssl.certificate: /home/es/config/x-pack/node01.crt 
xpack.ssl.certificate_authorities: [ "/home/es/config/x-pack/ca.crt" ] 
{code}
The full path to the node key file. This must be a location within the 
Elasticsearch configuration directory.

The full path to the node certificate. This must be a location within the 
Elasticsearch configuration directory.

An array of paths to the CA certificates that should be trusted. These paths 
must be a location within the Elasticsearch configuration directory.
{quote}

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)