[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448783#comment-16448783 ] ASF GitHub Bot commented on NIFI-3576: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2601 Thanks for the review time! > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448762#comment-16448762 ] ASF GitHub Bot commented on NIFI-3576: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2601 Tested it out with a live instance of ElasticSearch and it behaved the way I expected it. Though there appears to be a bug in the pagination itself with getPage, but that's a separate Jira ticket from what I can tell. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448758#comment-16448758 ] ASF subversion and git services commented on NIFI-3576: --- Commit 45bc1f1b4f590db0e4a3cb76fcd4cd89ca1a06bf in nifi's branch refs/heads/master from [~ottobackwards] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=45bc1f1 ] NIFI-3576 Support for QueryInfo relationship, can be used to track no-hits Squashed commit includes related commit from GitHub user wietze. This closes #2601 Signed-off-by: Mike Thomsen > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448760#comment-16448760 ] ASF GitHub Bot commented on NIFI-3576: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2601 > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448628#comment-16448628
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183491100
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -81,12 +87,21 @@
description = "Adds the specified property name/value as a query
parameter in the Elasticsearch URL used for processing")
public class QueryElasticsearchHttp extends
AbstractElasticsearchHttpProcessor {
+public enum QueryInfoRouteStrategy {
+NEVER,
--- End diff --
sorry check latest
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448595#comment-16448595
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183484904
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -81,12 +87,21 @@
description = "Adds the specified property name/value as a query
parameter in the Elasticsearch URL used for processing")
public class QueryElasticsearchHttp extends
AbstractElasticsearchHttpProcessor {
+public enum QueryInfoRouteStrategy {
+NEVER,
--- End diff --
Like NOHIT("noHits")
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448591#comment-16448591
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183484851
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -81,12 +87,21 @@
description = "Adds the specified property name/value as a query
parameter in the Elasticsearch URL used for processing")
public class QueryElasticsearchHttp extends
AbstractElasticsearchHttpProcessor {
+public enum QueryInfoRouteStrategy {
+NEVER,
--- End diff --
You need to set these to the values used in the AllowableValues.
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448588#comment-16448588
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183484608
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -267,32 +267,18 @@ public void setup(ProcessContext context) {
@Override
public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
-final Set routeQueryInfoRels = new HashSet<>();
-routeQueryInfoRels.add(REL_SUCCESS);
-routeQueryInfoRels.add(REL_FAILURE);
-routeQueryInfoRels.add(REL_RETRY);
-
-final Set successRels = new HashSet<>();
-successRels.add(REL_SUCCESS);
-successRels.add(REL_FAILURE);
-successRels.add(REL_RETRY);
if (ROUTING_QUERY_INFO_STRATEGY.equals(descriptor)) {
-if (ALWAYS.getValue().equalsIgnoreCase(newValue)) {
-routeQueryInfoRels.add(REL_QUERY_INFO);
-this.relationships = routeQueryInfoRels;
-
-this.queryInfoRouteStrategy =
QueryInfoRouteStrategy.ALWAYS;
-} else if (NO_HITS.getValue().equalsIgnoreCase(newValue)) {
-routeQueryInfoRels.add(REL_QUERY_INFO);
-this.relationships = routeQueryInfoRels;
-
-this.queryInfoRouteStrategy = QueryInfoRouteStrategy.NOHIT;
-} else {
-this.relationships = successRels;
+final Set relationshipSet = new HashSet<>();
+relationshipSet.add(REL_SUCCESS);
+relationshipSet.add(REL_FAILURE);
+relationshipSet.add(REL_RETRY);
-this.queryInfoRouteStrategy = QueryInfoRouteStrategy.NEVER;
+if (ALWAYS.getValue().equalsIgnoreCase(newValue) ||
NO_HITS.getValue().equalsIgnoreCase(newValue)) {
+relationshipSet.add(REL_QUERY_INFO);
}
+this.queryInfoRouteStrategy =
QueryInfoRouteStrategy.valueOf(newValue);
--- End diff --
The newValue value does not correspond to the enum values, so it blows up 3
unit tests.
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448585#comment-16448585 ] ASF GitHub Bot commented on NIFI-3576: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2601 Unit test failures: ``` [ERROR] testQueryElasticsearchOnTrigger_NoHits_NoHits(org.apache.nifi.processors.elasticsearch.TestQueryElasticsearchHttpNoHits) Time elapsed: 0.001 s <<< ERROR! java.lang.IllegalArgumentException: No enum constant org.apache.nifi.processors.elasticsearch.QueryElasticsearchHttp.QueryInfoRouteStrategy.noHits at org.apache.nifi.processors.elasticsearch.TestQueryElasticsearchHttpNoHits.testQueryElasticsearchOnTrigger_NoHits_NoHits(TestQueryElasticsearchHttpNoHits.java:75) [ERROR] testQueryElasticsearchOnTrigger_Hits_Always(org.apache.nifi.processors.elasticsearch.TestQueryElasticsearchHttpNoHits) Time elapsed: 0.002 s <<< ERROR! java.lang.IllegalArgumentException: No enum constant org.apache.nifi.processors.elasticsearch.QueryElasticsearchHttp.QueryInfoRouteStrategy.always at org.apache.nifi.processors.elasticsearch.TestQueryElasticsearchHttpNoHits.testQueryElasticsearchOnTrigger_Hits_Always(TestQueryElasticsearchHttpNoHits.java:185) [ERROR] testQueryElasticsearchOnTrigger_Hits_NoHits(org.apache.nifi.processors.elasticsearch.TestQueryElasticsearchHttpNoHits) Time elapsed: 0.001 s <<< ERROR! java.lang.IllegalArgumentException: No enum constant org.apache.nifi.processors.elasticsearch.QueryElasticsearchHttp.QueryInfoRouteStrategy.noHits at org.apache.nifi.processors.elasticsearch.TestQueryElasticsearchHttpNoHits.testQueryElasticsearchOnTrigger_Hits_NoHits(TestQueryElasticsearchHttpNoHits.java:141) ``` > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448408#comment-16448408
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183461688
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -267,30 +267,28 @@ public void setup(ProcessContext context) {
@Override
public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
+final Set routeQueryInfoRels = new HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+
+final Set successRels = new HashSet<>();
--- End diff --
I thought the same thing. Thanks for the feedback my latest did take that
approach
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448400#comment-16448400
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183458816
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -267,30 +267,28 @@ public void setup(ProcessContext context) {
@Override
public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
+final Set routeQueryInfoRels = new HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+
+final Set successRels = new HashSet<>();
--- End diff --
I don't think even needs to exist at this point because
`routequeryInfoRels` effectively does all three use cases and only gets the 4th
relationship based on the routing strategy. Might want to refactor that and
change the name to be more appropriate to the overlap.
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448398#comment-16448398
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183458469
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -267,30 +267,28 @@ public void setup(ProcessContext context) {
@Override
public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
+final Set routeQueryInfoRels = new HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+
+final Set successRels = new HashSet<>();
+successRels.add(REL_SUCCESS);
+successRels.add(REL_FAILURE);
+successRels.add(REL_RETRY);
+
if (ROUTING_QUERY_INFO_STRATEGY.equals(descriptor)) {
if (ALWAYS.getValue().equalsIgnoreCase(newValue)) {
--- End diff --
Not a deal breaker, but I actually had this location in mind.
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448278#comment-16448278
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183427371
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -237,6 +265,39 @@ public void setup(ProcessContext context) {
super.setup(context);
}
+@Override
+public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
+if (ROUTING_QUERY_INFO_STRATEGY.equals(descriptor)) {
+if (ALWAYS.getValue().equalsIgnoreCase(newValue)) {
+final Set routeQueryInfoRels = new
HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
--- End diff --
Thanks! done
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448239#comment-16448239
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r183417572
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -237,6 +265,39 @@ public void setup(ProcessContext context) {
super.setup(context);
}
+@Override
+public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
+if (ROUTING_QUERY_INFO_STRATEGY.equals(descriptor)) {
+if (ALWAYS.getValue().equalsIgnoreCase(newValue)) {
+final Set routeQueryInfoRels = new
HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
--- End diff --
You have a lot of duplicate code here. You could move:
```
final Set routeQueryInfoRels = new HashSet<>();
routeQueryInfoRels.add(REL_SUCCESS);
routeQueryInfoRels.add(REL_FAILURE);
routeQueryInfoRels.add(REL_RETRY);
```
to before the if/else if blocks and then add `REL_QUERY_INFO` only in those
two blocks. Also the `this.relationships` assignment could be done there as
well.
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16444013#comment-16444013
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r182733953
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/test/java/org/apache/nifi/processors/elasticsearch/TestQueryElasticsearchHttpNoHits.java
---
@@ -142,7 +142,7 @@ public void
testQueryElasticsearchOnTrigger_Hits_NoHits() throws IOException {
runner.assertValid();
runner.setIncomingConnection(false);
-runAndVerify(3,1,0,true);
+runAndVerify(3,0,0,true);
--- End diff --
Nevermind, I didn't know what that parameter was for, zero is the correct
value
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443301#comment-16443301
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r182592328
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/test/java/org/apache/nifi/processors/elasticsearch/TestQueryElasticsearchHttpNoHits.java
---
@@ -142,7 +142,7 @@ public void
testQueryElasticsearchOnTrigger_Hits_NoHits() throws IOException {
runner.assertValid();
runner.setIncomingConnection(false);
-runAndVerify(3,1,0,true);
+runAndVerify(3,0,0,true);
--- End diff --
It's been a long one, what scenario are you thinking about?
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443286#comment-16443286
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r182589001
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/test/java/org/apache/nifi/processors/elasticsearch/TestQueryElasticsearchHttpNoHits.java
---
@@ -142,7 +142,7 @@ public void
testQueryElasticsearchOnTrigger_Hits_NoHits() throws IOException {
runner.assertValid();
runner.setIncomingConnection(false);
-runAndVerify(3,1,0,true);
+runAndVerify(3,0,0,true);
--- End diff --
Should we run this with both 1 and 0 for coverage? I haven't looked into
the parameter, just shooting from the hip :)
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443253#comment-16443253
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r182582965
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -352,6 +414,15 @@ private int getPage(final Response getResponse, final
URL url, final ProcessCont
JsonNode responseJson = parseJsonResponse(new
ByteArrayInputStream(bodyBytes));
JsonNode hits = responseJson.get("hits").get("hits");
+if ( (hits.size() == 0 && queryInfoRouteStrategy ==
QueryInfoRouteStrategy.NOHIT)
--- End diff --
OK, you can check my fix
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442856#comment-16442856
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r182499545
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -352,6 +414,15 @@ private int getPage(final Response getResponse, final
URL url, final ProcessCont
JsonNode responseJson = parseJsonResponse(new
ByteArrayInputStream(bodyBytes));
JsonNode hits = responseJson.get("hits").get("hits");
+if ( (hits.size() == 0 && queryInfoRouteStrategy ==
QueryInfoRouteStrategy.NOHIT)
--- End diff --
So, you are saying that, if you set it to NO_HITS and you get all 20 at a
time, you get a query info on the last try?
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442848#comment-16442848
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r182498324
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -352,6 +414,15 @@ private int getPage(final Response getResponse, final
URL url, final ProcessCont
JsonNode responseJson = parseJsonResponse(new
ByteArrayInputStream(bodyBytes));
JsonNode hits = responseJson.get("hits").get("hits");
+if ( (hits.size() == 0 && queryInfoRouteStrategy ==
QueryInfoRouteStrategy.NOHIT)
--- End diff --
This part doesn't quite work, not necessarily due to a logic error here,
but it needs to be addressed either here or in the onTrigger do-while loop. If
I don't set a limit and I set my page size to 20, then fetch all results (I
have 100), then I get a flow file on the query-info connection. This is because
the hitLimit variable is never set as the limit variable is always null, so we
attempt to get all records starting from index 100, and zero are returned.
However this code checks that hits.size() is zero (which it is the last time).
No documents are output of course (due to the for-loop below), but the
query-info flow file is sent. Perhaps this method needs to know the current
total number of hits?
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442518#comment-16442518 ] ASF GitHub Bot commented on NIFI-3576: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2601 bump > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437676#comment-16437676 ] ASF GitHub Bot commented on NIFI-3576: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2601 @MikeThomsen This isn't my use case. I am just helping with the work. Please refer to the jira > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16437669#comment-16437669 ] ASF GitHub Bot commented on NIFI-3576: -- Github user MikeThomsen commented on the issue: https://github.com/apache/nifi/pull/2601 @ottobackwards Follow up from your reference to this in the `QuerySolr` PR... If you've got a use case where you need to get stats from ES, check out `JsonQueryElasticsearch`. It uses the JSON DSL and has relationships for aggregations so you can run them and split them up into separate JSON documents. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16434025#comment-16434025
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r180783317
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -175,16 +197,21 @@
.allowableValues(TARGET_FLOW_FILE_CONTENT,
TARGET_FLOW_FILE_ATTRIBUTES)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build();
-private static final Set relationships;
+public static final PropertyDescriptor ROUTING_QUERY_INFO_STRATEGY =
new PropertyDescriptor.Builder()
+.name("routing-query-info-strategy")
+.displayName("Routing Strategy for Query Info")
+.description("Specifies when to generate and route Query Info
after a successful query")
+.expressionLanguageSupported(false)
--- End diff --
Fixed thanks
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16433995#comment-16433995
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r180773767
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -175,16 +197,21 @@
.allowableValues(TARGET_FLOW_FILE_CONTENT,
TARGET_FLOW_FILE_ATTRIBUTES)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build();
-private static final Set relationships;
+public static final PropertyDescriptor ROUTING_QUERY_INFO_STRATEGY =
new PropertyDescriptor.Builder()
+.name("routing-query-info-strategy")
+.displayName("Routing Strategy for Query Info")
+.description("Specifies when to generate and route Query Info
after a successful query")
+.expressionLanguageSupported(false)
--- End diff --
As of PR #2205 I believe this has to be changed from false to
ExpressionLanguageScope.NONE
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425884#comment-16425884
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r179217708
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -68,8 +72,10 @@
+ "To retrieve more records, use the ScrollElasticsearchHttp
processor.")
@WritesAttributes({
@WritesAttribute(attribute = "filename", description = "The
filename attribute is set to the document identifier"),
+@WritesAttribute(attribute = "es.hitCount", description = "The
number of hits for a query"),
--- End diff --
nah, it is fine by me
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425878#comment-16425878
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r179216547
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -68,8 +72,10 @@
+ "To retrieve more records, use the ScrollElasticsearchHttp
processor.")
@WritesAttributes({
@WritesAttribute(attribute = "filename", description = "The
filename attribute is set to the document identifier"),
+@WritesAttribute(attribute = "es.hitCount", description = "The
number of hits for a query"),
--- End diff --
Just found out that some processors have used camel cases in writing
attributes. Sorry about that. Feel free to revert back although `es.hitcount`
itself looks good to me.
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424085#comment-16424085
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178839932
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -175,16 +197,30 @@
.allowableValues(TARGET_FLOW_FILE_CONTENT,
TARGET_FLOW_FILE_ATTRIBUTES)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build();
-private static final Set relationships;
+public static final PropertyDescriptor ROUTING_QUERY_INFO_STRATEGY =
new PropertyDescriptor.Builder()
+.name("routing-query-info-strategy")
+.displayName("Routing Strategy for Query Info")
+.description("Specifies when to generate and route Query Info
after a successful query")
+.expressionLanguageSupported(false)
+.allowableValues(ALWAYS, NEVER, NO_HITS)
+.defaultValue(NEVER.getValue())
+.required(false)
+.build();
+
+public static final PropertyDescriptor INCLUDE_QUERY_IN_ATTRS = new
PropertyDescriptor.Builder()
--- End diff --
done
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424023#comment-16424023
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178824015
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -175,16 +197,30 @@
.allowableValues(TARGET_FLOW_FILE_CONTENT,
TARGET_FLOW_FILE_ATTRIBUTES)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build();
-private static final Set relationships;
+public static final PropertyDescriptor ROUTING_QUERY_INFO_STRATEGY =
new PropertyDescriptor.Builder()
+.name("routing-query-info-strategy")
+.displayName("Routing Strategy for Query Info")
+.description("Specifies when to generate and route Query Info
after a successful query")
+.expressionLanguageSupported(false)
+.allowableValues(ALWAYS, NEVER, NO_HITS)
+.defaultValue(NEVER.getValue())
+.required(false)
+.build();
+
+public static final PropertyDescriptor INCLUDE_QUERY_IN_ATTRS = new
PropertyDescriptor.Builder()
--- End diff --
Having the attribute by default sounds fine to me, usually when we have a
property to choose it's because the value in the attribute could be fairly
large (such as a compound SQL query used in ExecuteSQL, e.g.).
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424013#comment-16424013
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178822326
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -106,6 +121,13 @@
+ "based on the processor properties and the
results of the fetch operation.")
.build();
+public static final Relationship REL_QUERY_INFO = new
Relationship.Builder()
+.name("query-info")
+.description(
+"Depending on the setting of the Generate Query Info
property, a FlowFile is routed to this relationship with " +
--- End diff --
nice catch, fixed
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424014#comment-16424014
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178822510
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -175,16 +197,30 @@
.allowableValues(TARGET_FLOW_FILE_CONTENT,
TARGET_FLOW_FILE_ATTRIBUTES)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build();
-private static final Set relationships;
+public static final PropertyDescriptor ROUTING_QUERY_INFO_STRATEGY =
new PropertyDescriptor.Builder()
+.name("routing-query-info-strategy")
+.displayName("Routing Strategy for Query Info")
+.description("Specifies when to generate and route Query Info
after a successful query")
+.expressionLanguageSupported(false)
+.allowableValues(ALWAYS, NEVER, NO_HITS)
+.defaultValue(NEVER.getValue())
+.required(false)
+.build();
+
+public static final PropertyDescriptor INCLUDE_QUERY_IN_ATTRS = new
PropertyDescriptor.Builder()
--- End diff --
@mattyb149 thoughts?
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424010#comment-16424010
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178821673
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -68,8 +72,10 @@
+ "To retrieve more records, use the ScrollElasticsearchHttp
processor.")
@WritesAttributes({
@WritesAttribute(attribute = "filename", description = "The
filename attribute is set to the document identifier"),
+@WritesAttribute(attribute = "es.hitCount", description = "The
number of hits for a query"),
--- End diff --
done
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423995#comment-16423995
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178817080
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -175,16 +197,30 @@
.allowableValues(TARGET_FLOW_FILE_CONTENT,
TARGET_FLOW_FILE_ATTRIBUTES)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build();
-private static final Set relationships;
+public static final PropertyDescriptor ROUTING_QUERY_INFO_STRATEGY =
new PropertyDescriptor.Builder()
+.name("routing-query-info-strategy")
+.displayName("Routing Strategy for Query Info")
+.description("Specifies when to generate and route Query Info
after a successful query")
+.expressionLanguageSupported(false)
+.allowableValues(ALWAYS, NEVER, NO_HITS)
+.defaultValue(NEVER.getValue())
+.required(false)
+.build();
+
+public static final PropertyDescriptor INCLUDE_QUERY_IN_ATTRS = new
PropertyDescriptor.Builder()
--- End diff --
Should we really need to have a property for this? Writing the
`es.query.url` as an attribute by default sounds safe to me. Thoughts?
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423993#comment-16423993
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178816076
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -68,8 +72,10 @@
+ "To retrieve more records, use the ScrollElasticsearchHttp
processor.")
@WritesAttributes({
@WritesAttribute(attribute = "filename", description = "The
filename attribute is set to the document identifier"),
+@WritesAttribute(attribute = "es.hitCount", description = "The
number of hits for a query"),
--- End diff --
I believe, in general, it is a standard practice in the project to avoid
casing `WritesAttributes` so shall we change it to `es.hit.count` or
`es.hitcount` ?
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423994#comment-16423994
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user zenfenan commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178816096
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -106,6 +121,13 @@
+ "based on the processor properties and the
results of the fetch operation.")
.build();
+public static final Relationship REL_QUERY_INFO = new
Relationship.Builder()
+.name("query-info")
+.description(
+"Depending on the setting of the Generate Query Info
property, a FlowFile is routed to this relationship with " +
--- End diff --
I believe "Generate Query Info" has to be changed to "Routing Strategy for
Query Info".
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423188#comment-16423188 ] ASF GitHub Bot commented on NIFI-3576: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2601 Added checkstyle fixes > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423085#comment-16423085
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178641076
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -219,6 +256,39 @@ public void setup(ProcessContext context) {
super.setup(context);
}
+@Override
+public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
+if (ROUTING_QUERY_INFO_STRATEGY.equals(descriptor)) {
+if (ALWAYS.getValue().equalsIgnoreCase(newValue)) {
+final Set routeQueryInfoRels = new
HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+routeQueryInfoRels.add(REL_QUERY_INFO);
+this.relationships = routeQueryInfoRels;
+
+this.queryInfoRouteStrategy =
QueryInfoRouteStrategy.ALWAYS;
+}else if (NO_HITS.getValue().equalsIgnoreCase(newValue)) {
+final Set routeQueryInfoRels = new
HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+routeQueryInfoRels.add(REL_QUERY_INFO);
+this.relationships = routeQueryInfoRels;
+
+this.queryInfoRouteStrategy = QueryInfoRouteStrategy.NOHIT;
+}
+}else {
--- End diff --
I believe that will work. Also MockProcessContext calls onPropertyModified
on set/removeProperty(), so you might be able to parlay that into a unit test
to check that the relationship is available or not (but I'm not sure)
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423074#comment-16423074
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178637480
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -219,6 +256,39 @@ public void setup(ProcessContext context) {
super.setup(context);
}
+@Override
+public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
+if (ROUTING_QUERY_INFO_STRATEGY.equals(descriptor)) {
+if (ALWAYS.getValue().equalsIgnoreCase(newValue)) {
+final Set routeQueryInfoRels = new
HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+routeQueryInfoRels.add(REL_QUERY_INFO);
+this.relationships = routeQueryInfoRels;
+
+this.queryInfoRouteStrategy =
QueryInfoRouteStrategy.ALWAYS;
+}else if (NO_HITS.getValue().equalsIgnoreCase(newValue)) {
+final Set routeQueryInfoRels = new
HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+routeQueryInfoRels.add(REL_QUERY_INFO);
+this.relationships = routeQueryInfoRels;
+
+this.queryInfoRouteStrategy = QueryInfoRouteStrategy.NOHIT;
+}
+}else {
--- End diff --
if (ROUTING_QUERY_INFO_STRATEGY.equals(descriptor)) ?
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423068#comment-16423068 ] ASF GitHub Bot commented on NIFI-3576: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/2601 I apologize, I had thought I ran contrib-check, but I obviously didn't > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423066#comment-16423066 ] ASF GitHub Bot commented on NIFI-3576: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2601#discussion_r178636382 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/test/java/org/apache/nifi/processors/elasticsearch/TestQueryElasticsearchHttpNoHits.java --- @@ -0,0 +1,365 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.elasticsearch; + +import static org.junit.Assert.assertEquals; --- End diff -- Checkstyle violation here, unused import, please remove (and see my comment above about running your own style check before submitting the PR, definitely helps the review/merge process), please and thanks! > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423065#comment-16423065
]
ASF GitHub Bot commented on NIFI-3576:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2601#discussion_r178635886
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java
---
@@ -219,6 +256,39 @@ public void setup(ProcessContext context) {
super.setup(context);
}
+@Override
+public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
+if (ROUTING_QUERY_INFO_STRATEGY.equals(descriptor)) {
+if (ALWAYS.getValue().equalsIgnoreCase(newValue)) {
+final Set routeQueryInfoRels = new
HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+routeQueryInfoRels.add(REL_QUERY_INFO);
+this.relationships = routeQueryInfoRels;
+
+this.queryInfoRouteStrategy =
QueryInfoRouteStrategy.ALWAYS;
+}else if (NO_HITS.getValue().equalsIgnoreCase(newValue)) {
+final Set routeQueryInfoRels = new
HashSet<>();
+routeQueryInfoRels.add(REL_SUCCESS);
+routeQueryInfoRels.add(REL_FAILURE);
+routeQueryInfoRels.add(REL_RETRY);
+routeQueryInfoRels.add(REL_QUERY_INFO);
+this.relationships = routeQueryInfoRels;
+
+this.queryInfoRouteStrategy = QueryInfoRouteStrategy.NOHIT;
+}
+}else {
--- End diff --
This whole if-else should be wrapped in a check to see if the
PropertyDescriptor is the Query Info Strategy property. Otherwise, changing any
other property results in entering this final else clause, thereby resetting
the Query Info Strategy to "Never".
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423067#comment-16423067 ] ASF GitHub Bot commented on NIFI-3576: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/2601#discussion_r178636130 --- Diff: nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java --- @@ -255,6 +325,8 @@ public void onTrigger(final ProcessContext context, final ProcessSession session .evaluateAttributeExpressions(flowFile).getValue() : null; final boolean targetIsContent = context.getProperty(TARGET).getValue() .equals(TARGET_FLOW_FILE_CONTENT); +final boolean includeQueryInAttrs = context.getProperty(INCLUDE_QUERY_IN_ATTRS).isSet() ? --- End diff -- CheckStyle violation here, the ? should be on the next line. You can run your Maven build for the nifi-elasticsearch-bundle with the contrib-check profile activated (`-Pcontrib-check`) and it will run a style check for you. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423023#comment-16423023 ] ASF GitHub Bot commented on NIFI-3576: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/2601 Reviewing... > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422882#comment-16422882 ] ASF GitHub Bot commented on NIFI-3576: -- GitHub user ottobackwards opened a pull request: https://github.com/apache/nifi/pull/2601 NIFI-3576 Support for QueryInfo relationship, useful for nohit queries This pr adds a new relationship, and a property/strategy for when it is used. The new relationship has an empty flow file with attributes about the query, right now the query string and the number of hits. The relationship is dynamically made available based on the strategy selected: Never : um, never send query information Always: always send it NoHit: Send it when there are no hits. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/nifi nohit-support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2601.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2601 commit ad1c4614104c541c68227658f784ec21f6e06694 Author: Otto Fowler Date: 2018-04-02T17:16:39Z NIFI-3576 Support for QueryInfo relationship, can be used to track no-hits > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419736#comment-16419736 ] Otto Fowler commented on NIFI-3576: --- What i'm doing: * new optional route with property to enable * property sets strategy for sending query info to ALWAYS, NEVER or NOHIT * another property to include the query URL in the Success FF > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419722#comment-16419722
]
Otto Fowler commented on NIFI-3576:
---
{code:java}
if ( (hits.size() == 0 && queryInfoRouteStrategy ==
QueryInfoRouteStrategy.NOHIT)
|| queryInfoRouteStrategy == QueryInfoRouteStrategy.ALWAYS) {
FlowFile queryInfo = flowFile == null ? session.create() :
session.create(flowFile);
session.putAttribute(queryInfo, "es.query.url", url.toExternalForm());
session.putAttribute(queryInfo, "es.query.hitCount",
String.valueOf(hits.size()));
session.putAttribute(queryInfo, MIME_TYPE.key(), "application/json");
session.importFrom(new ByteArrayInputStream(bodyBytes),queryInfo);
}
{code}
like that
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419656#comment-16419656 ] Joseph Percivall commented on NIFI-3576: [~ottobackwards] I agree [~panchicore] and [~wietze], would this path forward solve your use-cases? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419653#comment-16419653 ] Otto Fowler commented on NIFI-3576: --- OK, this feels like something to start with. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419639#comment-16419639 ] Otto Fowler commented on NIFI-3576: --- I like : never, no-hits, always! > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419637#comment-16419637 ] Otto Fowler commented on NIFI-3576: --- that and it would have the query as the FF content > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419631#comment-16419631 ] Joseph Percivall commented on NIFI-3576: Hmm, the attributes on a no-hits REL FF could be the same as a query-info REL FF. So the main differences would be the name and when it emits a FF. What if we took a best of both worlds and added the query-info relationship and for the property, it is an option of when to emit the FF to that relationship? The options would either be "never", "when there are no hits", or "always" (default to never). > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419621#comment-16419621 ] Otto Fowler commented on NIFI-3576: --- Still conforming to your above concept > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419620#comment-16419620 ] Otto Fowler commented on NIFI-3576: --- So - here is a question. Would a no-hits be more useful than a query-info which had the query and the hitcount and other meta that we think of? You are going to check attrs either way, and the query is good for debug > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419599#comment-16419599 ] Joseph Percivall commented on NIFI-3576: There are a couple examples, such as RouteOnAttribute, RouteOnContent, and QueryRecord but the best example for this use case is probably LookUpRecord[1]. Ah looks like it's still an open ticket to add DisplayName to Relationships[2]. Definitely, would rather not break people's flows. Let's just change up the description to better match. [1] [https://github.com/apache/nifi/blob/d7347a2dc3f73e76d03b9b3ef95015dd539e16ac/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/LookupRecord.java#L237] [2] https://issues.apache.org/jira/browse/NIFI-3591 > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419586#comment-16419586 ] Otto Fowler commented on NIFI-3576: --- I also, don't see a displayName() for RELs like there is for properties, should I worry about breaking people calling getRelationship(String name)? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419569#comment-16419569 ] Otto Fowler commented on NIFI-3576: --- I am not sure how you make a relationship optional based on a property. Is there an example? I like the success to hits, that de-conflates things. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419555#comment-16419555 ] Joseph Percivall commented on NIFI-3576: Ah, I see your point about the FlowFile per hit and agree that there's some weirdness with the attributes for an empty "hit" FlowFile. The actual "hit per FF" functionality kinda goes against what is documented so we should change that a bit. With that, I also agree that 3 isn't what we want. So what if we changed the displayName for "success" to "hits" and add a new property to say whether to emit a FF to a 'no hits' relationship. The property would default to not emitting and the relationship would only be available when that prop is true. That way the functionality is fully backwards compatible. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419548#comment-16419548 ] Otto Fowler commented on NIFI-3576: --- I'm sorry, I meant 3 is valid as first step, and instead of 1-2 the new REL_QUERY_INFO. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[
https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419537#comment-16419537
]
Otto Fowler commented on NIFI-3576:
---
I agree that it is valid to return no hits, if this were returning all hits as
one OR one flow file per hit you can imagine just outputting "\{"hits":[]}.
Since this outputs a file per hit, what the attributes would be on the 'empty'
FF is where I'm left wondering. I would think that doing hit.number,
nit.total ( so 1 of 20 ) etc, like segment numbers would work ( and make sense
if you did get hits as well).
I think 1 is a valid step.
Instead of 2 or 3, I was thinking it would be better to have a REL_QUERY_INFO,
that had the query as the document, and the hits etc as attributes.
> QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
>
>
> Key: NIFI-3576
> URL: https://issues.apache.org/jira/browse/NIFI-3576
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Joseph Percivall
>Assignee: Otto Fowler
>Priority: Minor
>
> In the event of a successful call, QueryElasticsearchHttp always drops the
> incoming flowfile and then emits pages of results to the success
> relationship. If the search returns no results then no pages of results are
> emitted to the success relationship.
> The processor should offer other options for handling when there are no
> results returned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419524#comment-16419524 ] Joseph Percivall commented on NIFI-3576: So to summarize we have a couple of different options: # A new 'original' relationship to route the original FlowFile which has an attribute like 'es.hits.total' - precedent set by InvokeHttp # A 'none found' relationship which routes a FlowFile when there are no hits - precedent set by FetchElasticsearch(5) # Emit a FlowFile to "Success" for but add an attribute 'es.hits.total' - matches the 'Success' relationship documentation which states "All FlowFiles that are read from Elasticsearch are routed to this relationship." I'm leaning more towards option 3. It seems more in line with the original intent of the relationship, and the processor as a whole. It's querying for potential hits, not fetching a single document, so having no results but, a successful query is valid. Also, this means that users won't have an invalid processor on migration (the new relationships wouldn't be routed anywhere). Then we also aren't duplicating logic which could just be in a follow-up RouteOnAttribute processor. Thoughts? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419492#comment-16419492 ] Otto Fowler commented on NIFI-3576: --- [~JPercivall] thoughts? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419194#comment-16419194 ] Otto Fowler commented on NIFI-3576: --- OK, I'll take a stab at it > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Otto Fowler >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419185#comment-16419185 ] Luis Pallares commented on NIFI-3576: - true > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419170#comment-16419170 ] Otto Fowler commented on NIFI-3576: --- So, you want to route the original to a new relationship? Like invokeHttp does with the request relationship? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419160#comment-16419160 ] Luis Pallares commented on NIFI-3576: - >would every flow file have a 'es.hits.total' and maybe a 'es.hit.number'? yes, 'es.hits.total' to keep the ES convention for this field. > just empty to a new relationship? it would be great to have a "not found" relationship when 'es.hits.total' is 0, if not, I bet developers will be using an "evaluator" connected to a "router" to make it usable, hence my advice to use a new rel to have the (if not results then) implicitly done by the processor. > What would the flow file output be? > empty to success with the attributes? flowfile is OK as it is, later we can work on a QueryElasticsearchRecord now that everything trends to the record-oriented methodology which is great. thanks. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418996#comment-16418996 ] Otto Fowler commented on NIFI-3576: --- So, would every flow file have a 'es.hits.total' and maybe a 'es.hit.number'? What would the flow file output be? just empty to a new relationship? or empty to success with the attributes? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418991#comment-16418991 ] Luis Pallares commented on NIFI-3576: - I propose, if you are querying something, you might want to know the total hits that match with the query search, so no matter if the query matches or not, it has to be a success relation response with the "es.hits.total". > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418978#comment-16418978 ] Otto Fowler commented on NIFI-3576: --- What would the correct behavior be? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418974#comment-16418974 ] Luis Pallares commented on NIFI-3576: - +1 I confirm too, in my use case I can't use a simple InvokeHTTP because I don't want to overwrite the original flowfile. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418776#comment-16418776 ] Wietze B commented on NIFI-3576: Are there plans to fix this issue anytime soon? I can confirm the problem is still there. I can't think of an easy workaround either, as the flow literally stops when Elastic doesn't return results. > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15905780#comment-15905780 ] Joseph Percivall commented on NIFI-3576: [~puspendu.baner...@gmail.com] nope, from my testing there is nothing emitted. I believe this is because when it hits this for loop[1] (where it creates the flowfiles to send) the hits.size() is 0 so it never does any iterations and the session.transfer(page, REL_SUCCESS) doesn't have any flowfiles in the list. [1] https://github.com/JPercivall/nifi/blob/69b23adf1b0fdfb7e1e344c833adfbeb1f4980e5/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L336-L336 > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3576) QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship
[ https://issues.apache.org/jira/browse/NIFI-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902625#comment-15902625 ] Puspendu Banerjee commented on NIFI-3576: - [~JPercivall] Shouldn't it still be a success with a blank flowfile? > QueryElasticsearchHttp should have a "Not Found"/"Zero results" relationship > > > Key: NIFI-3576 > URL: https://issues.apache.org/jira/browse/NIFI-3576 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Priority: Minor > > In the event of a successful call, QueryElasticsearchHttp always drops the > incoming flowfile and then emits pages of results to the success > relationship. If the search returns no results then no pages of results are > emitted to the success relationship. > The processor should offer other options for handling when there are no > results returned. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
