vjgorla commented on code in PR #9806:
URL: https://github.com/apache/nifi/pull/9806#discussion_r1999932538
##########
nifi-extension-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/SearchElasticsearch.java:
##########
@@ -88,26 +90,40 @@ public class SearchElasticsearch extends
AbstractPaginatedJsonQueryElasticsearch
static final String STATE_PAGE_EXPIRATION_TIMESTAMP =
"pageExpirationTimestamp";
static final String STATE_PAGE_COUNT = "pageCount";
static final String STATE_HIT_COUNT = "hitCount";
+ static final String STATE_FINISHED = "finished";
static final PropertyDescriptor QUERY = new
PropertyDescriptor.Builder().fromPropertyDescriptor(ElasticsearchRestProcessor.QUERY)
.description("A query in JSON syntax, not Lucene syntax. Ex:
{\"query\":{\"match\":{\"somefield\":\"somevalue\"}}}. " +
"If the query is empty, a default JSON Object will be
used, which will result in a \"match_all\" query in Elasticsearch.")
.build();
- private static final Set<Relationship> relationships = Set.of(REL_HITS,
REL_AGGREGATIONS);
+ static final PropertyDescriptor RESTART_ON_FINISH = new
PropertyDescriptor.Builder()
Review Comment:
The use case is when you know that query will not match any new data once it
reaches the end of the result set (e.g. reading all logs from yesterday).
Problem with using `ConsumeElasticsearch` for this use case is
* Having to provide an additional range field
* Elastic API calls would continue to be made even after reaching the end of
the result set, putting load on elastic
* Processor does not yield after reaching the end of the result set, putting
load on NiFi
On a separate note, `ConsumeElasticsearch` requires range field to have
unique values. It should support multiple fields, one of them a tie breaker,
similar to `search_after`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
