[
https://issues.apache.org/jira/browse/NIFI-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424252#comment-17424252
]
ASF subversion and git services commented on NIFI-8002:
-------------------------------------------------------
Commit 3a63bd579e059763c21c0b8355d8686ec5c222d4 in nifi's branch
refs/heads/main from Chris Sampson
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3a63bd5 ]
NIFI-8002:
- Creation of a PaginatedJsonQueryElasticsearch and ConsumeElasticsearch
processors
- Integration Tests for ES Client Service with version and flavour-specific
tests for applicable functionality
Signed-off-by: Joe Gresock <[email protected]>
This closes #5193.
> Create new Elasticsearch Scroll/Paginated Query processor using REST API
> ------------------------------------------------------------------------
>
> Key: NIFI-8002
> URL: https://issues.apache.org/jira/browse/NIFI-8002
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.12.1
> Reporter: Chris Sampson
> Priority: Minor
> Attachments: ES_Paginated_Queries.json, ES_Paginated_Queries.xml
>
> Original Estimate: 5h
> Time Spent: 57h 20m
> Remaining Estimate: 0h
>
> The existing ScrollElasticsearch processor does not fully work with newer
> versions of Elasticsearch (e.g. 7.x).
> A new version of this processor should be created using the REST API
> approach, allowing for a query to be passed as input FlowFile content.
> Appropriate parameters should be set as processor properties (e.g. index,
> scroll size, scroll time) and allow for additional query string parameters to
> be set as dynamic properties (see NIFI-8001).
> The processor should support both
> [_scroll|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results]
> and
> [_pit/search_after|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after].
> While scroll may be backward compatible, _pit was introduced in ES 7 (an
> earlier version of search_before exists for <7, but suggest not trying to
> support that as part of this ticket).
> It may be possible to simply extend the existing REST API's Query processor
> to page through results with the selected method, rather than create a new
> processor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
