[
https://issues.apache.org/jira/browse/NIFI-10948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844461#comment-17844461
]
Chris Sampson commented on NIFI-10948:
--------------------------------------
I tested the REST API based Elasticsearch components of NiFi against AWS
OpenSearch after upgrading the Elasticsearch (low-level) REST Client library to
the latest version (8.13.3) as part of NIFI-11686.
Aside from the lack of Point in Time queries (an Elastic XPack feature that
provides an improved pagination approach over the older Scroll functionality),
things seemed to work without issue.
Document types have been deprecated in Elasticsearch since long before AWS
forked the libraries to create OpenSearch. Use of OS without Types works fine
if seems, and I imagine would be the recommended approach for OS use as it is
in Elasticsearch (ES 8 no longer supports Types at all, NiFi allows for Types
but discourages their use).
So I'd suggest this isn't something that will be fixed in the NiFi 1 branch,
and works fine in the NiFi 2 branch for both OS & ES.
Work to create OS specific implementations could be raised as separate tickets
to include:
* use of the AWS OpenSearch client libraries (rather than Elastic libraries)
* Allowing AWS-style auth, e.g. assume IAM roles (hopefully via the existing
NiFi AWS controller services), rather than simple BASIC auth (username &
password)
> QueryElasticsearchHTTP doesnt work on Opensearch (and possibly newer
> Elasticsearch versions)
> --------------------------------------------------------------------------------------------
>
> Key: NIFI-10948
> URL: https://issues.apache.org/jira/browse/NIFI-10948
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.15.3
> Environment: Nifi 1.15.3 on centos machine
> Reporter: David Vassallo
> Priority: Minor
> Attachments: image-2022-12-05-16-34-20-513.png
>
>
> The QueryElasticsearchHTTP processor will return a "Null Pointer Exception"
> when running agasint Opensearch and probably newer versions of Elasticsearch.
> I tracked down the null pointer to this line:
>
> [https://github.com/apache/nifi/blob/0ebc6d31489e975dcbbe078fa572332ef8ffa9e2/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L418]
>
> The processor is looking for the "_type" field which is no longer in the
> right place. In opensearch v 1.x and above (possibly Elasticsearch too, but I
> dont have the means to test this), the field has moved to be a child of the
> "@metadata" field:
> !image-2022-12-05-16-34-20-513.png!
>
> If you try to circumvent this issue by populating the "Type" property of the
> processor, you will get an HTTP 400 Bad Message response since the processor
> builds a URL with the "size" and "from" arguments as part of the URL which
> are not supported by Opensearch/Elasticsearch in more recent versions, so you
> are forced to leave this property empty but then run into the null problem
> described above
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
