Hi, We have been using Apache Nifi for a while and its a core part of our system. Our main use case is to consume some documents from a apache kafka cluster and persist it to an Elastic search cluster. The version of these documents is controlled externally. We use the PutElasticSearch5processor to push data to elastic search, currently we are interested in the most recent version of a document, but we insert all the updates in elastic search and rely on searching to get the latest version of the document. As you can see this creates a lot of redundant copies of the same document, which we dont use. We are exploring options to overcome this problem, and one of them would be to extend the PutElasticSearch5Processor and add support for external version of the elastic search documents. We checked and the elastic search _bulk api supports external versioning.
My question is simple, is there any plans to add support for external version on the PuElasticSearch5Processor ? Regards, Ulisses Franca
