Further to my last, I've raised a new PR [1] that adds a new GetElasticsearch processor using the aforementioned ElasticsearchClientService (this is intended as a replacement for/alternative to the older FetchElasticsearch/Http processors).
The same PR also adds Proxy capability to the ElasticsearchClientService, updates the Elasticsearch low-level rest client library to 7.13.4 and addresses some comments made about the older processors (namely the ability to run the processor and fetch an Elasticsearch document without the need for an incoming FlowFile connection). The module Mike refers to that I think you'd want to start looking to replicate is the nifi-elasticsearch-client-service [2], which implements the APi defined in nifi-elasticsearch-client-service-api. This APi is what the nifi-elasticsearch-restapi-processors [4] use for their connections to the Elasticsearch service, so having a new OpenSearch based implementation of that API would allow users to easily swap out Elasticsearch for OpenSearch libraries as required and without any other reconfiguration required. [1] https://github.com/apache/nifi/pull/5535 [2] https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service [3] https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service-api [4] https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors --- *Chris Sampson* IT Consultant [email protected] On Wed, 17 Nov 2021 at 23:07, Mike Thomsen <[email protected]> wrote: > Eli, > > The best starting point we have is the > nifi-elasticsearch-restapi-bundle module and its sub-modules. That > uses a client service for most of the connection management. It's > based on the official Elastic client APIs. My recommendation would be > to clone that module and start creating a version agnostic OpenSearch > bundle. > > Mike > > On Mon, Nov 15, 2021 at 1:00 PM Chris Sampson > <[email protected]> wrote: > > > > Note that I'd started to consider this in some recent updates to the NiFi > > Elasticsearch REST Processors[1]. It looks like my comments/the actual > > version in this file may have got a little out of step with one another, > I > > think I intended to use 7.13.4 in this pom.xml as the high-level REST > > Clients from Elastic are still Apache 2.0 licensed, it's the low-level > > client that's no longer Apache 2.0 (I may look into this further if I get > > chance, but that's irrelevant for any OpenSearch work). > > > > OpenSearch is not something I've tested against nor plan to make changes > > for in any ongoing work, but I'd imagine a sensible place to > start/consider > > would be to look at providing a new implementation of the > > ElasticSearchClientService[2] that uses the same API (so far as the NiFi > > processors are concerned). This would then allow users to select an > > appropriate implementation (e.g. the existing > > ElasticSearchClientServiceImpl[3] or a new OpenSearch equivalent) for use > > in any of the existing Elasticsearch REST Processors [4]. > > > > I'd probably suggest not updating the other Elasticsearch processors: > > * nifi-elasticsearch-5-processors have largely been replaced and are only > > intended for use with the now very old Elasticsearch 5.x branch > > * nifi-elasticsearch-processors all have > > equivalent nifi-elasticsearch-restapi-processors processors that use the > > above Service API (except for the FetchElasticsearch processors, for > which > > we should probably raise a Jira ticket to create a newer equivalent, thus > > allowing potential deprecation of all processors other than the restapi > set) > > > > > > [1] > > > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/pom.xml#L142 > > [2] > > > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service-api/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientService.java > > [3] > > > https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java > > [4] > > > https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch > > > > --- > > *Chris Sampson* > > IT Consultant > > [email protected] > > > > > > On Mon, 15 Nov 2021 at 17:37, Joe Witt <[email protected]> wrote: > > > > > Eli, > > > > > > I've added you and your two cc'd colleagues as bcc so you see this. > > > Please subscribe to the list [1] so you'll see the replies and so that > > > your emails won't require moderation. cc's and bcc's and such will not > > > work for replies/etc.. on these lists. > > > > > > Ultimately we'd just need someone in the community to create the > > > appropriate Opensearch equivalent clients to the existing > > > Elasticsearch set. Presuming the License of the client and any > > > transitive deps is fair game we can include it or make it available as > > > a usable nar supported by the NiFi community. And then we need > > > someone to review and ultimately someone with commit privileges to > > > merge. > > > > > > [1] https://nifi.apache.org/mailing_lists.html > > > > > > Thanks > > > > > > On Mon, Nov 15, 2021 at 10:22 AM Fisher, Eli > <[email protected]> > > > wrote: > > > > > > > > Hi Apache NiFi team, > > > > > > > > My name is Eli Fisher. I work at Amazon on the OpenSearch project< > > > http://opensearch.org/>. We have had a couple people in the community > ask > > > about adding NiFi support, and so I wanted to reach out to learn more > about > > > what the process involved would be and to understand if you also > seeing of > > > related issues/requests. OpenSearch is wire compatible with OSS > > > Elasticsearch 7.10.2, and so we have people pointing people to use the > > > existing Elasticsearch integration. However, in the fullness of time, > these > > > projects will likely diverge so we wanted to reach out to learn about > the > > > process of contributing an integration for OpenSearch. I’ve cc’d two > of my > > > colleagues who also work on OpenSearch with me that are interested in > > > learning about the process. > > > > > > > > Thank you for your time, > > > > Eli > > > >
