[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88963385
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearch5.java
 ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.elasticsearch.ElasticsearchTimeoutException;
+import org.elasticsearch.action.get.GetRequestBuilder;
+import org.elasticsearch.action.get.GetResponse;
+import org.elasticsearch.client.transport.NoNodeAvailableException;
+import org.elasticsearch.node.NodeClosedException;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve. If the cluster has been 
configured for authorization and/or secure "
++ "transport (SSL/TLS), and the X-Pack plugin is available, secure 
connections can be made. This processor "
++ "supports Elasticsearch 5.x clusters.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearch5 extends 
AbstractElasticsearch5TransportClientProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88927684
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearch5.java
 ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.elasticsearch.ElasticsearchTimeoutException;
+import org.elasticsearch.action.get.GetRequestBuilder;
+import org.elasticsearch.action.get.GetResponse;
+import org.elasticsearch.client.transport.NoNodeAvailableException;
+import org.elasticsearch.node.NodeClosedException;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve. If the cluster has been 
configured for authorization and/or secure "
++ "transport (SSL/TLS), and the X-Pack plugin is available, secure 
connections can be made. This processor "
++ "supports Elasticsearch 5.x clusters.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearch5 extends 
AbstractElasticsearch5TransportClientProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 
Relationship.Builder().name("not found")
+.description("A 

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88926499
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearch5.java
 ---
@@ -227,6 +227,7 @@ public void process(final InputStream in) throws 
IOException {
 session.transfer(flowFile, REL_FAILURE);
 
 } else {
+session.getProvenanceReporter().send(flowFile, 
response.remoteAddress().getAddress());
--- End diff --

This will only emit provenance event if "response.hasFailures" is true. If 
there are not failures this won't get hit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88924961
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -242,11 +356,66 @@ The following binary components are provided under 
the Apache Software License v
   Apache Commons CLI
--- End diff --

This was supposed to be removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88924216
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -229,8 +229,122 @@ The following binary components are provided under 
the Apache Software License v
 
   (ASLv2) The Netty Project
   The following NOTICE information applies:
-The Netty Project
-Copyright 2011 The Netty Project
+
+The Netty Project
--- End diff --

This change is not reflected in the NiFi-assembly nar


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88924576
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -242,11 +356,66 @@ The following binary components are provided under 
the Apache Software License v
   Apache Commons CLI
   Copyright 2001-2016 The Apache Software Foundation
 
+  (ASLv2) Apache Commons Logging
+The following NOTICE information applies:
+
+  Apache Commons Logging
+  Copyright 2003-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+
+  (ASLv2) Apache Commons Codec
+The following NOTICE information applies:
+  Apache Commons Codec
+  Copyright 2002-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+   src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java
+   contains test data from http://aspell.net/test/orig/batch0.tab.
+   Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org)
+
+   
===
+
+   The content of package org.apache.commons.codec.language.bm has 
been translated
+   from the original php source code available at 
http://stevemorse.org/phoneticinfo.htm
+   with permission from the original authors.
+   Original source copyright:
+   Copyright (c) 2008 Alexander Beider & Stephen P. Morse.
+
+  (ASLv2) Apache Commons Lang
+  The following NOTICE information applies:
+Apache Commons Lang
+Copyright 2001-2015 The Apache Software Foundation
+
+This product includes software from the Spring Framework,
+under the Apache License 2.0 (see: 
StringUtils.containsWhitespace())
+
+  (ASLv2) Apache Http Components
--- End diff --

"HttpComponents" is one word


https://github.com/apache/nifi/blob/b026f0bebe737aa62ab7de48a776d851807b1350/nifi-assembly/NOTICE#L111-L111
https://hc.apache.org/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88924805
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -242,11 +356,66 @@ The following binary components are provided under 
the Apache Software License v
   Apache Commons CLI
   Copyright 2001-2016 The Apache Software Foundation
 
+  (ASLv2) Apache Commons Logging
+The following NOTICE information applies:
+
+  Apache Commons Logging
+  Copyright 2003-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+
+  (ASLv2) Apache Commons Codec
+The following NOTICE information applies:
+  Apache Commons Codec
+  Copyright 2002-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+   src/test/org/apache/commons/codec/language/DoubleMetaphoneTest.java
+   contains test data from http://aspell.net/test/orig/batch0.tab.
+   Copyright (C) 2002 Kevin Atkinson (kev...@gnu.org)
+
+   
===
+
+   The content of package org.apache.commons.codec.language.bm has 
been translated
+   from the original php source code available at 
http://stevemorse.org/phoneticinfo.htm
+   with permission from the original authors.
+   Original source copyright:
+   Copyright (c) 2008 Alexander Beider & Stephen P. Morse.
+
+  (ASLv2) Apache Commons Lang
+  The following NOTICE information applies:
+Apache Commons Lang
+Copyright 2001-2015 The Apache Software Foundation
+
+This product includes software from the Spring Framework,
+under the Apache License 2.0 (see: 
StringUtils.containsWhitespace())
+
+  (ASLv2) Apache Http Components
+The following NOTICE information applies:
+  Apache HttpComponents Client
+  Copyright 1999-2016 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+  (ASLv2) Apache Log4J
+The following NOTICE information applies:
+  Apache log4j
+  Copyright 2010 The Apache Software Foundation
+
+  This product includes software developed at
+  The Apache Software Foundation (http://www.apache.org/).
+
+
 *
 Public Domain
--- End diff --

You can remove this section now that there are no libraries under the 
Public domain included


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[olegz]

Github user olegz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88806479
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5Processor.java
 ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * A base class for all Elasticsearch processors
+ */
+public abstract class AbstractElasticsearch5Processor extends 
AbstractProcessor {
--- End diff --

Minor, but looking at AbstractElasticsearch5TransportClientProcessor does 
this one have to be public?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r8876
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanLightStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanMinimalStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/es/SpanishLightStemmer.java
+  

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88766699
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanLightStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanMinimalStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/es/SpanishLightStemmer.java
+  

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88597130
  
--- Diff: nifi-nar-bundles/nifi-elasticsearch-5-bundle/pom.xml ---
@@ -0,0 +1,44 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-nar-bundles
+1.1.0-SNAPSHOT
+
+
+org.apache.nifi
+nifi-elasticsearch-5-bundle
+pom
+
+
+6.3.0
--- End diff --

It has to be in both the processor POM and NAR POM, so when the two bundles 
were different, I included it in each of the parent bundle POMs. However, I'm 
consolidating the bundles (per your suggestion a la Kafka), and now it can't be 
set in the parent POM, now it will have to be set in each NAR/processors pair. 
This is due to a higher parent POM setting lucene.version for repository stuff, 
and the ES POM using that property to choose a version of Lucene (for some 
reason it's not hard-coded in the published POM). Strange but true :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88527446
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanLightStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanMinimalStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/es/SpanishLightStemmer.java
+  

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88526691
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

Also: (org.apache.httpcomponents:httpcore:jar:4.4.1:compile - omitted for 
conflict with 4.4.5)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88527105
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

As well as: org.apache.logging.log4j:log4j-core:jar:2.7:compile


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88524040
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/LICENSE
 ---
@@ -0,0 +1,262 @@
+
+ Apache License
+   Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+  "License" shall mean the terms and conditions for use, reproduction,
+  and distribution as defined by Sections 1 through 9 of this document.
+
+  "Licensor" shall mean the copyright owner or entity authorized by
+  the copyright owner that is granting the License.
+
+  "Legal Entity" shall mean the union of the acting entity and all
+  other entities that control, are controlled by, or are under common
+  control with that entity. For the purposes of this definition,
+  "control" means (i) the power, direct or indirect, to cause the
+  direction or management of such entity, whether by contract or
+  otherwise, or (ii) ownership of fifty percent (50%) or more of the
+  outstanding shares, or (iii) beneficial ownership of such entity.
+
+  "You" (or "Your") shall mean an individual or Legal Entity
+  exercising permissions granted by this License.
+
+  "Source" form shall mean the preferred form for making modifications,
+  including but not limited to software source code, documentation
+  source, and configuration files.
+
+  "Object" form shall mean any form resulting from mechanical
+  transformation or translation of a Source form, including but
+  not limited to compiled object code, generated documentation,
+  and conversions to other media types.
+
+  "Work" shall mean the work of authorship, whether in Source or
+  Object form, made available under the License, as indicated by a
+  copyright notice that is included in or attached to the work
+  (an example is provided in the Appendix below).
+
+  "Derivative Works" shall mean any work, whether in Source or Object
+  form, that is based on (or derived from) the Work and for which the
+  editorial revisions, annotations, elaborations, or other 
modifications
+  represent, as a whole, an original work of authorship. For the 
purposes
+  of this License, Derivative Works shall not include works that remain
+  separable from, or merely link (or bind by name) to the interfaces 
of,
+  the Work and Derivative Works thereof.
+
+  "Contribution" shall mean any work of authorship, including
+  the original version of the Work and any modifications or additions
+  to that Work or Derivative Works thereof, that is intentionally
+  submitted to Licensor for inclusion in the Work by the copyright 
owner
+  or by an individual or Legal Entity authorized to submit on behalf of
+  the copyright owner. For the purposes of this definition, "submitted"
+  means any form of electronic, verbal, or written communication sent
+  to the Licensor or its representatives, including but not limited to
+  communication on electronic mailing lists, source code control 
systems,
+  and issue tracking systems that are managed by, or on behalf of, the
+  Licensor for the purpose of discussing and improving the Work, but
+  excluding communication that is conspicuously marked or otherwise
+  designated in writing by the copyright owner as "Not a Contribution."
+
+  "Contributor" shall mean Licensor and any individual or Legal Entity
+  on behalf of whom a Contribution has been received by Licensor and
+  subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  copyright license to reproduce, prepare Derivative Works of,
+  publicly display, publicly perform, sublicense, and distribute the
+  Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+  this License, each Contributor hereby grants to You a perpetual,
+  worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+  (except as stated in this section) patent license to make, have made,
+  use, offer to sell, sell, import, and otherwise transfer the Work,
+  where such license applies only 

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88520091
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
--- End diff --

Commons lang is brought in through the processor utils and it's notice 
needs to be added too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88526373
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

As well as "(commons-codec:commons-codec:jar:1.10:compile - version managed 
from 1.9; omitted for duplicate)".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88527394
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanLightStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanMinimalStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/es/SpanishLightStemmer.java
+  

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88525935
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

ES 5 brings in "io.netty:netty:jar:3.10.6.Final:compile" which is ASLv2 
licensed but has the following NOTICE file which is much more than what you 
have listed below.

https://github.com/netty/netty/blob/netty-3.10.6.Final/NOTICE.txt


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88526308
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
--- End diff --

It also brings in "commons-logging:commons-logging:jar:1.2:compile - 
omitted for conflict with 1.1.3)", the notice information will need to be added.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88475057
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+abstract class AbstractElasticsearch5TransportClientProcessor extends 
AbstractElasticsearch5Processor {
+
+/**
+ * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
+ */
+private static final Validator HOSTNAME_PORT_VALIDATOR = (subject, 
input, context) -> {
+final List esList = Arrays.asList(input.split(","));
+for (String hostnamePort : esList) {
+String[] addresses = hostnamePort.split(":");
+// Protect against invalid input like http://127.0.0.1:9300 
(URL scheme should not be there)
+if (addresses.length != 2) {
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Must be in hostname:port form (no scheme such as 
http://;).valid(false).build();
+}
+}
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Valid cluster definition").valid(true).build();
+};
+
+protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
+.name("el5-cluster-name")
+.displayName("Cluster Name")
+.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("elasticsearch")
+.build();
+
+protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
+.name("el5-hosts")
+.displayName("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
++ "host1:port,host2:port,  For example 
testcluster:9300. This processor uses the Transport Client to "
++ "connect to hosts. The default transport client port 
is 9300.")
+.required(true)
+.expressionLanguageSupported(false)
+.addValidator(HOSTNAME_PORT_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROP_XPACK_LOCATION = new 
PropertyDescriptor.Builder()
+

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88329420
  
--- Diff: nifi-nar-bundles/pom.xml ---
@@ -54,6 +54,7 @@
 nifi-html-bundle
 nifi-scripting-bundle
 nifi-elasticsearch-bundle
--- End diff --

Could this be changed to match how the Kafka bundle does version hierarchy? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88337612
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5Processor.java
 ---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A base class for all Elasticsearch processors
+ */
+public abstract class AbstractElasticsearch5Processor extends 
AbstractProcessor {
+
+public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("el5-ssl-context-service")
+.displayName("SSL Context Service")
+.description("The SSL Context Service used to provide client 
certificate information for TLS/SSL "
++ "connections. This service only applies if the 
Elasticsearch endpoint(s) have been secured with TLS/SSL.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
+.name("el5-charset")
+.displayName("Character Set")
+.description("Specifies the character set of the document 
data.")
+.required(true)
+.defaultValue("UTF-8")
+.addValidator(StandardValidators.CHARACTER_SET_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor USERNAME = new 
PropertyDescriptor.Builder()
+.name("el5-username")
+.displayName("Username")
+.description("Username to access the Elasticsearch cluster")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PASSWORD = new 
PropertyDescriptor.Builder()
+.name("el5-password")
+.displayName("Password")
+.description("Password to access the Elasticsearch cluster")
+.required(false)
+.sensitive(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+protected abstract void createElasticsearchClient(ProcessContext 
context) throws ProcessException;
+
+@Override
+protected Collection 
customValidate(ValidationContext validationContext) {
+Set results = new HashSet<>();
+
+// Ensure that if username or password is set, then the other is 
too
+Map propertyMap = 
validationContext.getProperties();
+if (StringUtils.isEmpty(propertyMap.get(USERNAME)) != 
StringUtils.isEmpty(propertyMap.get(PASSWORD))) {
--- End diff --

In addition to this, should do a check that Xpack is at least set if the 
security properties are set (since they won't/can't be used if xpack isn't set)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a 

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88340765
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/FetchElasticsearch5.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.elasticsearch.ElasticsearchTimeoutException;
+import org.elasticsearch.action.get.GetRequestBuilder;
+import org.elasticsearch.action.get.GetResponse;
+import org.elasticsearch.client.transport.NoNodeAvailableException;
+import org.elasticsearch.node.NodeClosedException;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5", "fetch", "read", "get"})
+@CapabilityDescription("Retrieves a document from Elasticsearch using the 
specified connection properties and the "
++ "identifier of the document to retrieve. If the cluster has been 
configured for authorization and/or secure "
++ "transport (SSL/TLS), and the X-Pack plugin is available, secure 
connections can be made. This processor "
++ "supports Elasticsearch 5.x clusters.")
+@WritesAttributes({
+@WritesAttribute(attribute = "filename", description = "The 
filename attributes is set to the document identifier"),
+@WritesAttribute(attribute = "es.index", description = "The 
Elasticsearch index containing the document"),
+@WritesAttribute(attribute = "es.type", description = "The 
Elasticsearch document type")
+})
+public class FetchElasticsearch5 extends 
AbstractElasticsearch5TransportClientProcessor {
+
+public static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are read from Elasticsearch 
are routed to this relationship").build();
+
+public static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from 
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
document cannot be fetched but attempting the operation again may succeed")
+.build();
+
+public static final Relationship REL_NOT_FOUND = new 

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88341802
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/pom.xml
 ---
@@ -0,0 +1,111 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+nifi-elasticsearch-5-bundle
+org.apache.nifi
+1.1.0-SNAPSHOT
+
+
+nifi-elasticsearch-5-processors
+jar
+
+
+2.7
+5.0.0
+
+
+
+
+org.apache.nifi
+nifi-api
+provided
+
+
+org.apache.nifi
+nifi-properties
+provided
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+org.apache.lucene
+lucene-core
+${lucene.version}
--- End diff --

Where is lucene used in this module? I did a search and only came up with 
this pom


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88333819
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+abstract class AbstractElasticsearch5TransportClientProcessor extends 
AbstractElasticsearch5Processor {
+
+/**
+ * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
+ */
+private static final Validator HOSTNAME_PORT_VALIDATOR = (subject, 
input, context) -> {
+final List esList = Arrays.asList(input.split(","));
+for (String hostnamePort : esList) {
+String[] addresses = hostnamePort.split(":");
+// Protect against invalid input like http://127.0.0.1:9300 
(URL scheme should not be there)
+if (addresses.length != 2) {
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Must be in hostname:port form (no scheme such as 
http://;).valid(false).build();
+}
+}
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Valid cluster definition").valid(true).build();
+};
+
+protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
+.name("el5-cluster-name")
+.displayName("Cluster Name")
+.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("elasticsearch")
+.build();
+
+protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
+.name("el5-hosts")
+.displayName("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
++ "host1:port,host2:port,  For example 
testcluster:9300. This processor uses the Transport Client to "
++ "connect to hosts. The default transport client port 
is 9300.")
+.required(true)
+.expressionLanguageSupported(false)
+.addValidator(HOSTNAME_PORT_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROP_XPACK_LOCATION = new 
PropertyDescriptor.Builder()
+

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88342854
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearch5.java
 ---
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.AttributeExpression;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.elasticsearch.ElasticsearchTimeoutException;
+import org.elasticsearch.action.bulk.BulkItemResponse;
+import org.elasticsearch.action.bulk.BulkRequestBuilder;
+import org.elasticsearch.action.bulk.BulkResponse;
+
+import org.elasticsearch.client.transport.NoNodeAvailableException;
+import org.elasticsearch.node.NodeClosedException;
+import org.elasticsearch.transport.ReceiveTimeoutTransportException;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@SupportsBatching
+@Tags({"elasticsearch", "elasticsearch 5","insert", "update", "write", 
"put"})
+@CapabilityDescription("Writes the contents of a FlowFile to 
Elasticsearch, using the specified parameters such as "
++ "the index to insert into and the type of the document. If the 
cluster has been configured for authorization "
++ "and/or secure transport (SSL/TLS), and the X-Pack plugin is 
available, secure connections can be made. This processor "
++ "supports Elasticsearch 5.x clusters.")
+public class PutElasticsearch5 extends 
AbstractElasticsearch5TransportClientProcessor {
+
+static final Relationship REL_SUCCESS = new 
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Elasticsearch 
are routed to this relationship").build();
+
+static final Relationship REL_FAILURE = new 
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to 
Elasticsearch are routed to this relationship").build();
+
+static final Relationship REL_RETRY = new 
Relationship.Builder().name("retry")
+.description("A FlowFile is routed to this relationship if the 
database cannot be updated but attempting the operation again may succeed")
+.build();
+
+public static final PropertyDescriptor ID_ATTRIBUTE = new 
PropertyDescriptor.Builder()
+.name("el5-put-id-attribute")
+.displayName("Identifier Attribute")
+.description("The name of the attribute containing the 
identifier for each FlowFile")
+.required(true)
+.expressionLanguageSupported(false)
+

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88337395
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+abstract class AbstractElasticsearch5TransportClientProcessor extends 
AbstractElasticsearch5Processor {
+
+/**
+ * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
+ */
+private static final Validator HOSTNAME_PORT_VALIDATOR = (subject, 
input, context) -> {
+final List esList = Arrays.asList(input.split(","));
+for (String hostnamePort : esList) {
+String[] addresses = hostnamePort.split(":");
+// Protect against invalid input like http://127.0.0.1:9300 
(URL scheme should not be there)
+if (addresses.length != 2) {
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Must be in hostname:port form (no scheme such as 
http://;).valid(false).build();
+}
+}
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Valid cluster definition").valid(true).build();
+};
+
+protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
+.name("el5-cluster-name")
+.displayName("Cluster Name")
+.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("elasticsearch")
+.build();
+
+protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
+.name("el5-hosts")
+.displayName("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
++ "host1:port,host2:port,  For example 
testcluster:9300. This processor uses the Transport Client to "
++ "connect to hosts. The default transport client port 
is 9300.")
+.required(true)
+.expressionLanguageSupported(false)
+.addValidator(HOSTNAME_PORT_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROP_XPACK_LOCATION = new 
PropertyDescriptor.Builder()
+

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88341213
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/pom.xml
 ---
@@ -0,0 +1,111 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+nifi-elasticsearch-5-bundle
+org.apache.nifi
+1.1.0-SNAPSHOT
+
+
+nifi-elasticsearch-5-processors
+jar
+
+
+2.7
+5.0.0
+
+
+
+
+org.apache.nifi
+nifi-api
+provided
+
+
+org.apache.nifi
+nifi-properties
+provided
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+org.apache.lucene
+lucene-core
+${lucene.version}
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.elasticsearch.client
+transport
+${es.version}
+
+
+com.squareup.okhttp3
--- End diff --

I don't think this dep is needed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88330726
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5TransportClientProcessor.java
 ---
@@ -0,0 +1,289 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+import org.elasticsearch.client.Client;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.transport.InetSocketTransportAddress;
+import org.elasticsearch.transport.client.PreBuiltTransportClient;
+
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.net.InetSocketAddress;
+import java.net.MalformedURLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+
+
+abstract class AbstractElasticsearch5TransportClientProcessor extends 
AbstractElasticsearch5Processor {
+
+/**
+ * This validator ensures the Elasticsearch hosts property is a valid 
list of hostname:port entries
+ */
+private static final Validator HOSTNAME_PORT_VALIDATOR = (subject, 
input, context) -> {
+final List esList = Arrays.asList(input.split(","));
+for (String hostnamePort : esList) {
+String[] addresses = hostnamePort.split(":");
+// Protect against invalid input like http://127.0.0.1:9300 
(URL scheme should not be there)
+if (addresses.length != 2) {
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Must be in hostname:port form (no scheme such as 
http://;).valid(false).build();
+}
+}
+return new 
ValidationResult.Builder().subject(subject).input(input).explanation(
+"Valid cluster definition").valid(true).build();
+};
+
+protected static final PropertyDescriptor CLUSTER_NAME = new 
PropertyDescriptor.Builder()
+.name("el5-cluster-name")
+.displayName("Cluster Name")
+.description("Name of the ES cluster (for example, 
elasticsearch_brew). Defaults to 'elasticsearch'")
+.required(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.defaultValue("elasticsearch")
+.build();
+
+protected static final PropertyDescriptor HOSTS = new 
PropertyDescriptor.Builder()
+.name("el5-hosts")
+.displayName("ElasticSearch Hosts")
+.description("ElasticSearch Hosts, which should be comma 
separated and colon for hostname/port "
++ "host1:port,host2:port,  For example 
testcluster:9300. This processor uses the Transport Client to "
++ "connect to hosts. The default transport client port 
is 9300.")
+.required(true)
+.expressionLanguageSupported(false)
+.addValidator(HOSTNAME_PORT_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROP_XPACK_LOCATION = new 
PropertyDescriptor.Builder()
+

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88341976
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/pom.xml
 ---
@@ -0,0 +1,111 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+nifi-elasticsearch-5-bundle
+org.apache.nifi
+1.1.0-SNAPSHOT
+
+
+nifi-elasticsearch-5-processors
+jar
+
+
+2.7
+5.0.0
+
+
+
+
+org.apache.nifi
+nifi-api
+provided
+
+
+org.apache.nifi
+nifi-properties
+provided
+
+
+org.apache.nifi
+nifi-processor-utils
+
+
+org.apache.lucene
+lucene-core
+${lucene.version}
+
+
+org.apache.nifi
+nifi-mock
+test
+
+
+org.elasticsearch.client
+transport
+${es.version}
+
+
+com.squareup.okhttp3
+okhttp
+3.3.1
+
+
+org.apache.nifi
+nifi-ssl-context-service-api
+
+
+commons-io
+commons-io
+
+
+org.codehaus.jackson
+jackson-mapper-asl
--- End diff --

Same comment as lucene, can't find where this is used.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88329905
  
--- Diff: nifi-nar-bundles/nifi-elasticsearch-5-bundle/pom.xml ---
@@ -0,0 +1,44 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+
+org.apache.nifi
+nifi-nar-bundles
+1.1.0-SNAPSHOT
+
+
+org.apache.nifi
+nifi-elasticsearch-5-bundle
+pom
+
+
+6.3.0
--- End diff --

nit-pick, "lucene.version" is defined here but "slf4jversion" and 
"es.version" are defined in the processor pom. Is there a reason for that or 
can it be moved to the processor pom to match?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88319339
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/AbstractElasticsearch5Processor.java
 ---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.apache.nifi.util.StringUtils;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * A base class for all Elasticsearch processors
+ */
+public abstract class AbstractElasticsearch5Processor extends 
AbstractProcessor {
+
+public static final PropertyDescriptor PROP_SSL_CONTEXT_SERVICE = new 
PropertyDescriptor.Builder()
+.name("el5-ssl-context-service")
+.displayName("SSL Context Service")
+.description("The SSL Context Service used to provide client 
certificate information for TLS/SSL "
++ "connections. This service only applies if the 
Elasticsearch endpoint(s) have been secured with TLS/SSL.")
+.required(false)
+.identifiesControllerService(SSLContextService.class)
+.build();
+
+protected static final PropertyDescriptor CHARSET = new 
PropertyDescriptor.Builder()
--- End diff --

I've received comments on making these properties accept Expression 
Language where possible (to leverage the Variable Registry), will make the 
changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/1233

NIFI-3011: Added Elasticsearch5 processors



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-3011

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1233.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1233


commit 7ad90fc793a1294f6ce9d6aa4780e5432fa92a42
Author: Matt Burgess 
Date:   2016-11-16T17:26:26Z

NIFI-3011: Added Elasticsearch5 processors




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---