[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414340#comment-16414340 ] ASF GitHub Bot commented on NIFI-4325: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2113 > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414337#comment-16414337 ] ASF subversion and git services commented on NIFI-4325: --- Commit aa947e4d3e0e54877d24683fefa5da651332c433 in nifi's branch refs/heads/master from [~mike.thomsen] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=aa947e4 ] NIFI-4325 Added new processor that uses the JSON DSL. NIFI-4325 Cleaned up how ElasticSearch client service builds SSLContext, added query attribute to flowfiles and other changes requested in a code review. This closes #2113. Signed-off-by: Joe Percivall > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414338#comment-16414338 ] ASF subversion and git services commented on NIFI-4325: --- Commit aa947e4d3e0e54877d24683fefa5da651332c433 in nifi's branch refs/heads/master from [~mike.thomsen] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=aa947e4 ] NIFI-4325 Added new processor that uses the JSON DSL. NIFI-4325 Cleaned up how ElasticSearch client service builds SSLContext, added query attribute to flowfiles and other changes requested in a code review. This closes #2113. Signed-off-by: Joe Percivall > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414285#comment-16414285 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 +1 Overall looks good and works as expected. I've built with contrib check and tested using a vanilla instance. I will squash and merge to master. Thanks @MikeThomsen. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413732#comment-16413732 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall I just got done testing it w/ X-Pack for SSL and u/p auth based on Elastic docs. Everything was in working order there. [Steps if you want to validate](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/configuring-tls.html) I also added the documentation there. Sorry about that. AFAIK, that warning is going to be more of a polite warning than a real "here be dragons" sort of thing because IIRC ES limits you to 10k results unless you explicitly up the number in the index configuration. Probably a good reminder, but I doubt anyone is going to blow up their NiFi installation without having to take deliberate steps to facilitate that. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413230#comment-16413230 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Looks good, was able to do a couple simple queries and aggregations. I wasn't able to test HTTPS though as I don't have access to the X-Pack. Were you able to do so @MikeThomsen? The only lingering issue, that I've mentioned multiple times now, is to add a note about the entire body of the response being loaded into memory. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412605#comment-16412605 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall I fixed the dependency issue and tried a simple flow. Everything seemed to work. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412450#comment-16412450 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @MikeThomsen, just reviewed the changes. They look good and I think we're almost there. One thing that is missing is the comment regarding pulling the results into memory. Also, I tried to build NiFi and test the processor an instance but hit the following error when I attempted to start up NiFi (./bin/nifi.sh start). I tried rebasing it to the latest master but to no avail. > 2018-03-24 02:06:53,171 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch could not be instantiated java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:148) at org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:123) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:771) at org.apache.nifi.NiFi.(NiFi.java:157) at org.apache.nifi.NiFi.(NiFi.java:71) at org.apache.nifi.NiFi.main(NiFi.java:292) Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper at org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch.(JsonQueryElasticsearch.java:206) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 8 common frames omitted Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ObjectMapper at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 15 common frames omitted > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409650#comment-16409650 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall FYI, part of the reason I want to get this done is I'm planning on doing a whole new set of ES processors that are based on bring the total CRUD functionality over to the official Elastic-provided APIs. The current set of processors use the transport API (deprecated) and make manual REST calls that AFAIK don't do master detection and stuff like that which comes baked-in with the Elastic APIs. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409648#comment-16409648 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 Thanks. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409627#comment-16409627 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @MikeThomsen, I'm planning on reviewing this tomorrow evening > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409605#comment-16409605 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall Any chance we can close this out? > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404848#comment-16404848 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall changes checked in. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404746#comment-16404746
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175420184
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Prope
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404690#comment-16404690
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175411532
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Prope
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404689#comment-16404689
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175411128
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -61,22 +58,23 @@
public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
private ObjectMapper mapper = new ObjectMapper();
-private List properties;
+static final private List properties;
private RestClient client;
private String url;
-@Override
-protected void init(ControllerServiceInitializationContext config) {
-properties = new ArrayList<>();
-properties.add(ElasticSearchClientService.HTTP_HOSTS);
-properties.add(ElasticSearchClientService.USERNAME);
-properties.add(ElasticSearchClientService.PASSWORD);
-
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
-properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
-properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
-properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+static {
+List _props = new ArrayList();
--- End diff --
Done.
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404688#comment-16404688
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175411050
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -121,7 +119,7 @@ private void setupClient(ConfigurationContext context)
throws Exception {
RestClientBuilder builder = RestClient.builder(hh)
.setHttpClientConfigCallback(httpClientBuilder -> {
-if (sslService != null) {
+if (sslService != null &&
sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
try {
--- End diff --
Done. I moved that logic out and have the it bubble it up with an
InitializationException if any of those exceptions are thrown.
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404677#comment-16404677
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175407337
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+static final private List properties;
+
+private RestClient client;
+
+private String url;
+
+static {
+List _props = new ArrayList();
+_props.add(ElasticSearchClientService.HTTP_HOSTS);
+_props.add(ElasticSearchClientService.USERNAME);
+_props.add(ElasticSearchClientService.PASSWORD);
+_props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+_props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+_props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+_props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+properties = Collections.unmodifiableList(_props);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
MalformedURLException {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getVal
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404273#comment-16404273 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 > I hope my review doesn't come off in the wrong way, there is a lot of great work here and I just want to make sure the usability is top notch. Not at all. It's all fair and good feedback. I'll take a crack at these tomorrow. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404219#comment-16404219 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @MikeThomsen, just finished up another round of reviewing. Again, mostly focused on the error handling portion. I hope my review doesn't come off in the wrong way, there is a lot of great work here and I just want to make sure the usability is top notch. Error handling is a huge focus in NiFi because we're writing up a toolbox of processors for potentially non-technical users to work with however they want. Those users need to be able to see what they messed up and what went wrong for efficient and effective rectification. Thanks for the hard work, and the willingness to iterate and learn! > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404218#comment-16404218
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306495
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
--- End diff --
I don't see this in the latest commit
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404214#comment-16404214
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306340
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+static final private List properties;
+
+private RestClient client;
+
+private String url;
+
+static {
+List _props = new ArrayList();
+_props.add(ElasticSearchClientService.HTTP_HOSTS);
+_props.add(ElasticSearchClientService.USERNAME);
+_props.add(ElasticSearchClientService.PASSWORD);
+_props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+_props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+_props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+_props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+properties = Collections.unmodifiableList(_props);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
MalformedURLException {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValu
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404210#comment-16404210
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175306133
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+static final private List properties;
+
+private RestClient client;
+
+private String url;
+
+static {
+List _props = new ArrayList();
+_props.add(ElasticSearchClientService.HTTP_HOSTS);
+_props.add(ElasticSearchClientService.USERNAME);
+_props.add(ElasticSearchClientService.PASSWORD);
+_props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+_props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+_props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+_props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+properties = Collections.unmodifiableList(_props);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
MalformedURLException {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValu
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404200#comment-16404200
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305604
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+static final private List properties;
+
+private RestClient client;
+
+private String url;
+
+static {
+List _props = new ArrayList();
+_props.add(ElasticSearchClientService.HTTP_HOSTS);
+_props.add(ElasticSearchClientService.USERNAME);
+_props.add(ElasticSearchClientService.PASSWORD);
+_props.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+_props.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+_props.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+_props.add(ElasticSearchClientService.RETRY_TIMEOUT);
+
+properties = Collections.unmodifiableList(_props);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+private void setupClient(ConfigurationContext context) throws
MalformedURLException {
+final String hosts =
context.getProperty(HTTP_HOSTS).evaluateAttributeExpressions().getValu
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404199#comment-16404199
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305459
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Proper
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404197#comment-16404197
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175305389
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Proper
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404194#comment-16404194
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175304957
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -61,22 +58,23 @@
public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
private ObjectMapper mapper = new ObjectMapper();
-private List properties;
+static final private List properties;
private RestClient client;
private String url;
-@Override
-protected void init(ControllerServiceInitializationContext config) {
-properties = new ArrayList<>();
-properties.add(ElasticSearchClientService.HTTP_HOSTS);
-properties.add(ElasticSearchClientService.USERNAME);
-properties.add(ElasticSearchClientService.PASSWORD);
-
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
-properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
-properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
-properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+static {
+List _props = new ArrayList();
--- End diff --
Bit of a nit pick but can this init be changed so it's not of raw type:
> List\ _props = new ArrayList\<\>();
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404191#comment-16404191
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175304809
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -121,7 +119,7 @@ private void setupClient(ConfigurationContext context)
throws Exception {
RestClientBuilder builder = RestClient.builder(hh)
.setHttpClientConfigCallback(httpClientBuilder -> {
-if (sslService != null) {
+if (sslService != null &&
sslService.isKeyStoreConfigured() && sslService.isTrustStoreConfigured()) {
try {
--- End diff --
Given that this is a callback, we probably want to move as much of the
logic out of here as possible. That way we can better handle any errors and
reduce performance cost if this is called multiple times. Currently, if there
is an error which hits line 145, the error will be caught and logged but it
will continue to create the http client as if nothing went wrong. What we
should probably do is have a final SSLContext which gets created above,
bubbling up any errors so that the CS doesn't enable in a bad state, and set
the SSLContext same way as below in the context.
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404102#comment-16404102 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Taking a look now. WRT: > Ok, I can do that from now on. Shouldn't they be squashed before a merge into master? Yup, they will be squashed prior to merge to master but the reviewer can do that at the same time they add the comment to the commit to close the PR. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403825#comment-16403825 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 Ok. It should all be there now. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403772#comment-16403772
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271885
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
--- End diff --
Ok. I should also note that scroll queries are not supported.
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403771#comment-16403771
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271865
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+p
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403768#comment-16403768
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271747
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+p
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403769#comment-16403769
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271759
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+p
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403767#comment-16403767
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271738
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
--- End diff --
Done.
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403755#comment-16403755
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271557
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Prope
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403754#comment-16403754
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271547
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Prope
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403750#comment-16403750
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175271522
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Prope
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403749#comment-16403749 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r175271497 --- Diff: nifi-nar-bundles/nifi-standard-services/pom.xml --- @@ -15,6 +15,18 @@ --> http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 + --- End diff -- Done. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403748#comment-16403748 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall Thanks for the feedback. I'll get to working on these. WRT: > Lastly, it's preferred if you don't squash your commits every time. If you don't, that allows the reviewer to more easily see exactly what changed since they last reviewed it. Also allows reviewers to see how the PR evolved over time in response to different comments. Ok, I can do that from now on. Shouldn't they be squashed before a merge into master? > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403662#comment-16403662
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265212
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
--- End diff --
Related to the note about pagination, should add something indicating that
very large searches should be done as the response is loaded into memory.
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403658#comment-16403658
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265070
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+pr
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403657#comment-16403657
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175265049
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+pr
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403654#comment-16403654
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264989
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
+
+private RestClient client;
+
+private String url;
+
+@Override
+protected void init(ControllerServiceInitializationContext config) {
+properties = new ArrayList<>();
+properties.add(ElasticSearchClientService.HTTP_HOSTS);
+properties.add(ElasticSearchClientService.USERNAME);
+properties.add(ElasticSearchClientService.PASSWORD);
+
properties.add(ElasticSearchClientService.PROP_SSL_CONTEXT_SERVICE);
+properties.add(ElasticSearchClientService.CONNECT_TIMEOUT);
+properties.add(ElasticSearchClientService.SOCKET_TIMEOUT);
+properties.add(ElasticSearchClientService.RETRY_TIMEOUT);
+}
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return properties;
+}
+
+@OnEnabled
+public void onEnabled(final ConfigurationContext context) throws
InitializationException {
+try {
+setupClient(context);
+} catch (Exception ex) {
+getLogger().error("Could not initialize ElasticSearch
client.", ex);
+throw new InitializationException(ex);
+}
+}
+
+@OnDisabled
+public void onDisabled() throws IOException {
+this.client.close();
+this.url = null;
+}
+
+pr
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403653#comment-16403653
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264961
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java
---
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.ControllerServiceInitializationContext;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.ssl.SSLContextService;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.PrintWriter;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class ElasticSearchClientServiceImpl extends
AbstractControllerService implements ElasticSearchClientService {
+private ObjectMapper mapper = new ObjectMapper();
+
+private List properties;
--- End diff --
Can this be declared similar to the properties in the processor and marked
final?
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403652#comment-16403652
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264906
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Proper
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403651#comment-16403651
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264876
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Proper
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403649#comment-16403649
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175264845
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Proper
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403589#comment-16403589 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r175264784 --- Diff: nifi-nar-bundles/nifi-standard-services/pom.xml --- @@ -15,6 +15,18 @@ --> http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> 4.0.0 + --- End diff -- This dependencies section shouldn't be needed here. This is the parent pom for all of the standard services. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403566#comment-16403566 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Thanks @MikeThomsen, looks like that fixed the NPE. Still needs the License and Notice in the client service nar. Also, it's a bit of an anti-pattern to have the logging and error handling in the Controller Service for methods called by the processor. This is because any time there is a problem with the search, the bulletin is logged at the Controller Service. This means that not only will a user not know which processor is the cause of the error (since it could be shared) but also when looking at the map the processor appears to be working without issue. Lastly, since the error isn't bubbled back to the processor, it never yields so if there is something wrong with the query it will continually error and log without pause. Related, there's also some weirdness with error handling when there is an incoming connection. Currently, input is only sent to Failure instead of Original when an error is caught at the top level. This means that if a query is messed up and is caught as a JsonProcessingException in the Controller service, the input will be routed to original and no other outputs. As a user, I would expect that flowfile to be sent to Failure. So in summary, the CS should let the calling processor decide how to handle any exceptions encountered during processing. This allows the processor to more granularlly decide what to do under different situations. For example, was it a JSON parsing exception and it's acting as a source processor? well we better yield the whole processor since it's probably just going to continually fail. Is there an incoming connection but some network error happened when searching? again, later FlowFiles will probably fail too, so we should yield. If you have any questions on when to yield the processor vs penalizing the FlowFile, feel free to ask. Lastly, it's preferred if you don't squash your commits every time. If you don't, that allows the reviewer to more easily see exactly what changed since they last reviewed it. Also allows reviewers to see how the PR evolved over time in response to different comments. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403413#comment-16403413 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall Done. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403286#comment-16403286
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r175249262
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch.java
---
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.SearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_ALLOWED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
Pr
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401471#comment-16401471 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Apologises, won't be able to finish the review this evening. I reviewed the restapi nar License and Notice files they look good to go. Just need to do a bit of functional testing and verify @mattyb149 comments were addressed and we should be ready to merge (pending the L/N for the service nar). > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401452#comment-16401452 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 The nifi-elasticsearch-client-service-nar is missing a license and notice. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400947#comment-16400947 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Thanks @MikeThomsen I should be able to take another look later tonight > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400945#comment-16400945 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall It's ready for review. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396886#comment-16396886 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 2/3 builds pass now. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396866#comment-16396866 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @JPercivall @mattyb149 Once I changed the code to use `transfer(FlowFile, Relationship)` and not `transfer(List, Relationship)` the error stopped happening. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396808#comment-16396808
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2113
@JPercivall @mattyb149 That exception only happens for me when I stop the
processor. It happens here:
``
if (hitsFlowFiles.size() > 0) {
session.transfer(hitsFlowFiles, REL_HITS);
for (FlowFile ff : hitsFlowFiles) {
session.getProvenanceReporter().send(ff,
clientService.getTransitUrl(index, type));
}
}
``
On the provenance reporting line. Any ideas?
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396786#comment-16396786 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r174078307 --- Diff: nifi-assembly/pom.xml --- @@ -445,6 +445,24 @@ language governing permissions and limitations under the License. --> 1.6.0-SNAPSHOT nar + +org.apache.nifi + nifi-elasticsearch-client-service-api-nar --- End diff -- Weird, I thought that I took that out. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396509#comment-16396509
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user JPercivall commented on the issue:
https://github.com/apache/nifi/pull/2113
I'm seeing NPE and FlowFile handling exceptions. I didn't do anything
special, just using the example query. I'll attach a template. Errors from the
logs below.
> 2018-03-12 23:58:11,386 ERROR [Timer-Driven Process Thread-10]
o.a.n.p.e.JsonQueryElasticsearch
JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42] Error
processing flowfile.: java.lang.NullPointerException
java.lang.NullPointerException: null
at
org.apache.nifi.processors.elasticsearch.JsonQueryElasticsearch.onTrigger(JsonQueryElasticsearch.java:248)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1123)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-03-12 23:58:11,389 ERROR [Timer-Driven Process Thread-10]
o.a.n.p.e.JsonQueryElasticsearch
JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42]
JsonQueryElasticsearch[id=1d5d83d1-0162-1000-cdf2-92fa9e61ef42] failed to
process due to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=77ad5fff-d4b6-4e11-8d8c-b07ac1d0a6cb,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1520913052434-1, container=default,
section=1], offset=30, length=228],offset=226,name=46172028496405,size=2]
transfer relationship not specified; rolling back session: {}
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=77ad5fff-d4b6-4e11-8d8c-b07ac1d0a6cb,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1520913052434-1, container=default,
section=1], offset=30, length=228],offset=226,name=46172028496405,size=2]
transfer relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:251)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:321)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1123)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396511#comment-16396511 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Template for FlowFile handling exception. [FlowFile_Handling_Exception.txt](https://github.com/apache/nifi/files/1805420/FlowFile_Handling_Exception.txt) > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396380#comment-16396380 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r173996495 --- Diff: nifi-assembly/pom.xml --- @@ -445,6 +445,24 @@ language governing permissions and limitations under the License. --> 1.6.0-SNAPSHOT nar + +org.apache.nifi + nifi-elasticsearch-client-service-api-nar --- End diff -- My build is failing due to this. I don't see this artifact created anywhere. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396358#comment-16396358 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Looks like the master build is failing with some check style issues, hence why your builds failed. I'll work on fixing them. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396326#comment-16396326 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Reviewing > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395889#comment-16395889 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 Rebased and building. @mattyb149 I changed the functionality to match the GetMongo functionality you reviewed recently. Had to spend a while getting the build to work again. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395609#comment-16395609 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2113 I'm not sure I'll have time to close the loop before 1.6.0, so if you'd like to finish the review/merge after Mike's rebase that would be very cool, thanks! > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395597#comment-16395597 ] ASF GitHub Bot commented on NIFI-4325: -- Github user JPercivall commented on the issue: https://github.com/apache/nifi/pull/2113 Hey @mattyb149, what's your status for this review? From a cursory look, it appears like it just needs an updated pom from @MikeThomsen and the final approval. With talks of 1.6.0 happening it would be nice to get this in so those using ES 6 aren't limited to the HTTP processor. If help is needed to finalize things just let me know where I can help. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352244#comment-16352244 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @mattyb149 I think all of the change requests are done. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16334270#comment-16334270 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @mattyb149 I moved it to a new NAR and rebased against the latest master last night. So once this is committed, I'll start working on the new processors and service methods. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327294#comment-16327294 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2113 Makes sense to me, we had to do the same with Elasticsearch 2.x and 5.x because of the difference in transport client versions. As folks move exclusively to ES5+, it would be nice to deprecate the other NARs and go forward with versions of all the processors (Get,Put,Search, etc.) using the new Java client. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327257#comment-16327257 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @mattyb149 I've been thinking about what to do with this PR, and I think it needs to be put into a new NAR. I'm thinking "nifi-elasticsearch-restapi.nar" or something like that. What I'd ideally like to do is position this to be a new package that does full CRUD using only the official REST API from Elastic Co. It might also help with the issue that @joewitt raised about the sizing because the current mix of ElasticSearch processors use different toolkits which needlessly raises the size of the NARs. Thoughts? > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301902#comment-16301902 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @mattyb149 I think I got your changes done now. I renamed a few classes to remove obvious references to ElasticSearch 5.X because the goal is to make it an obvious fit with 6.X too now that that is out. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297635#comment-16297635
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157903183
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchResponse.java
---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.elasticsearch;
+
+import java.util.List;
+import java.util.Map;
+
+public class ElasticSearchResponse {
+private List> hits;
+private Map aggregations;
+private long numberOfHits;
+private int took;
+private boolean timedOut;
+
+public ElasticSearchResponse(List> hits,
Map aggregations,
+ int numberOfHits, int took, boolean
timedOut) {
+this.hits = hits;
+this.aggregations = aggregations;
+this.numberOfHits = numberOfHits;
+this.took = took;
+this.timedOut = timedOut;
+}
+
+public Map getAggregations() {
+return aggregations;
+}
+
+public List> getHits() {
+return hits;
+}
+
+public long getNumberOfHits() {
+return numberOfHits;
+}
+
+public boolean isTimedOut() {
+return timedOut;
+}
+
+public int getTook() {
+return took;
+}
+
+@Override
+public String toString() {
+return "ElasticSearchResponse{" +
--- End diff --
Nice!
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297634#comment-16297634
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902956
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fe
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297636#comment-16297636
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902546
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
--- End diff --
It looks like you're not using the incoming flow file contents, in which
case we might relax this constraint to INPUT_ALLOWED, and if no flow file is
available (you can evaluate EL against a null FF) you can create one, what do
you think?
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297631#comment-16297631
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902648
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displ
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297633#comment-16297633 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r157903679 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service-nar/src/main/resources/META-INF/LICENSE --- @@ -0,0 +1,357 @@ + + Apache License + Version 2.0, January 2004 +http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby g
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297632#comment-16297632
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902770
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fe
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297629#comment-16297629
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r157902719
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type", description =
"application/json"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fe
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297630#comment-16297630 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r157903403 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,332 @@ +nifi-hbase_1_1_2-client-service-nar --- End diff -- This appears to be a copy from a different NAR ;) The first line should be updated, and the rest of the NOTICES below should only be included if we are including the component(s) in the artifacts. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16297578#comment-16297578 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r157902041 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java --- @@ -0,0 +1,291 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.annotation.lifecycle.OnUnscheduled; +import org.apache.nifi.components.AllowableValue; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.elasticsearch.ElasticSearchClientService; +import org.apache.nifi.elasticsearch.ElasticSearchResponse; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.OutputStreamCallback; --- End diff -- Lines 38 and 42 are unused imports (i.e. CheckStyle violations), but I can fix these on final merge if there are no other issues :) > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274354#comment-16274354 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @mattyb149 I think everything should be up to date now. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259512#comment-16259512
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r152056531
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.dis
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250746#comment-16250746
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726684
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displ
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250744#comment-16250744
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726813
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type"),
+@WritesAttribute(attribute = "aggregation.name", description = "The
name of the aggregation whose results are in the output flowfile")
+})
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_ORIGINAL = new
Relationship.Builder().name("original")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"split up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displ
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250743#comment-16250743 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r150726016 --- Diff: nifi-assembly/pom.xml --- @@ -338,6 +338,16 @@ org.apache.nifi +nifi-elasticsearch-client-service-nar +nar + + > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250742#comment-16250742
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726318
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,451 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.entity.ContentType;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
+import org.apache.http.nio.entity.NStringEntity;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.URL;
+import java.security.KeyStore;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
--- End diff --
Having the 5 and Http can be confusing as well, so I'm good with the name
as-is :)
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user nee
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250745#comment-16250745
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150726237
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+@WritesAttributes({
+@WritesAttribute(attribute = "mime.type"),
--- End diff --
This should have a description that the mime.type will be set to
"application/json"
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248876#comment-16248876 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 @mattyb149 I think everything except the proxy issue is now in there. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246525#comment-16246525 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r150086797 --- Diff: nifi-assembly/pom.xml --- @@ -338,6 +338,16 @@ org.apache.nifi +nifi-elasticsearch-client-service-nar +nar + + > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246533#comment-16246533
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150087559
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"break up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
+.description("The name of the index to read from")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Pro
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246531#comment-16246531
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r150087242
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"break up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
+.description("The name of the index to read from")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Pro
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16245682#comment-16245682 ] ASF GitHub Bot commented on NIFI-4325: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2113 Thanks @mattyb149 I'll try to find some time to address these today or tomorrow. I've been knee deep in some Mongo-related work for a client. I have two pull requests for Mongo and record api-related fixes/updates if you're up for reviewing two small patches (one is 3 lines of code :) ) > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240509#comment-16240509 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r148787152 --- Diff: nifi-assembly/pom.xml --- @@ -338,6 +338,16 @@ org.apache.nifi +nifi-elasticsearch-client-service-nar +nar + + > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240512#comment-16240512
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149121878
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"break up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
+.description("The name of the index to read from")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Prope
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240511#comment-16240511
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149127323
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientService.java
---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.elasticsearch;
+
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.controller.ControllerService;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.ssl.SSLContextService;
+
+import java.util.Map;
+import java.util.Optional;
+
+@Tags({"elasticsearch", "client"})
+@CapabilityDescription("A controller service for accessing an
ElasticSearch client.")
+public interface ElasticSearchClientService extends ControllerService {
--- End diff --
The current Http versions of the ES processors have the future capability
of configuring a proxy, although these properties are not yet exposed to the
user (see [NIFI-4198](https://issues.apache.org/jira/browse/NIFI-4198) and
#2094); consider adding proxy capabilities here, or creating an improvement
Jira after this PR/Jira has been merged.
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240513#comment-16240513
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149120565
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
+"This applies even if you select the \"break up hits\"
option to send individual hits to the " +
+"\"hits\" relationship.").build();
+
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be read from
Elasticsearch are routed to this relationship").build();
+
+public static final Relationship REL_HITS = new
Relationship.Builder().name("hits")
+.description("Search hits are routed to this relationship.")
+.build();
+
+public static final Relationship REL_AGGREGATIONS = new
Relationship.Builder().name("aggregations")
+.description("Aggregations are routed to this relationship.")
+.build();
+
+public static final PropertyDescriptor INDEX = new
PropertyDescriptor.Builder()
+.name("el5-fetch-index")
+.displayName("Index")
+.description("The name of the index to read from")
+.required(true)
+.expressionLanguageSupported(true)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final Prope
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[
https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240507#comment-16240507
]
ASF GitHub Bot commented on NIFI-4325:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2113#discussion_r149119932
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-5-processors/src/main/java/org/apache/nifi/processors/elasticsearch/JsonQueryElasticsearch5.java
---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.elasticsearch;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnUnscheduled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.elasticsearch.ElasticSearchClientService;
+import org.apache.nifi.elasticsearch.ElasticSearchResponse;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.OutputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+
+@InputRequirement(InputRequirement.Requirement.INPUT_REQUIRED)
+@EventDriven
+@Tags({"elasticsearch", "elasticsearch 5", "query", "read", "get", "json"})
+@CapabilityDescription("A processor that allows the user to run a query
(with aggregations) written with the " +
+"ElasticSearch JSON DSL. It currently does not support
pagination.")
+public class JsonQueryElasticsearch5 extends AbstractProcessor {
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All original flowfiles that don't cause an error
to occur go to this relationship. " +
--- End diff --
Should this be called "original" if the incoming flow files (that don't
cause an error) are always routed here?
> Create a new ElasticSearch processor that supports the JSON DSL
> ---
>
> Key: NIFI-4325
> URL: https://issues.apache.org/jira/browse/NIFI-4325
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Mike Thomsen
>Priority: Minor
>
> The existing ElasticSearch processors use the Lucene-style syntax for
> querying, not the JSON DSL. A new processor is needed that can take a full
> JSON query and execute it. It should also support aggregation queries in this
> syntax. A user needs to be able to take a query as-is from Kibana and drop it
> into NiFi and have it just run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFI-4325) Create a new ElasticSearch processor that supports the JSON DSL
[ https://issues.apache.org/jira/browse/NIFI-4325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16240514#comment-16240514 ] ASF GitHub Bot commented on NIFI-4325: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2113#discussion_r149123022 --- Diff: nifi-nar-bundles/nifi-standard-services/nifi-elasticsearch-client-service-bundle/nifi-elasticsearch-client-service/pom.xml --- @@ -0,0 +1,158 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> +4.0.0 + + +org.apache.nifi +nifi-elasticsearch-client-service-bundle +1.5.0-SNAPSHOT + + +nifi-elasticsearch-client-service +jar + + + +org.apache.nifi +nifi-lookup-service-api +provided + + +org.apache.nifi +nifi-api +provided + + +org.apache.nifi +nifi-utils + + +org.apache.nifi +nifi-hadoop-utils --- End diff -- Does this package need nifi-hadoop-utils? Please remove if not. > Create a new ElasticSearch processor that supports the JSON DSL > --- > > Key: NIFI-4325 > URL: https://issues.apache.org/jira/browse/NIFI-4325 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike Thomsen >Priority: Minor > > The existing ElasticSearch processors use the Lucene-style syntax for > querying, not the JSON DSL. A new processor is needed that can take a full > JSON query and execute it. It should also support aggregation queries in this > syntax. A user needs to be able to take a query as-is from Kibana and drop it > into NiFi and have it just run. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
