eolivelli opened a new pull request #11263: URL: https://github.com/apache/pulsar/pull/11263
### Motivation This patch is a set of enhancements to the ElasticSearch Sink: - support TLS - support authentication - support KeyValue schema and other Pulsar Schemas - support Bulk writes - support "errors action" (FAIL, IGNORE...) - support AVRO records (Avro to JSON conversion) - support for using the "key" of the record as record "_id" in ES (supporting both Message key and KeyValue encoding) - support for creating the "_id" in ES using a selection of values from the Message (in case of AVRO/JSON message) - support DELETING documents in case of non empty key and null value By default the Sink behaves the same way it did with Pulsar 2.8.0, that is basically reading from the Topic and writing to ElasticSearch. Schema related features are to be enabled explicitly by configuring "schemaEnable: true" in the configuration. This patch also re-enables ElasticSearch integration tests, that were disabled in 2.8.0. ### Verifying this change - this change add unit tests that cover the changes and the new features - this change re-enables ES integration tests (disabled in 2.8.0?) We (at DataStax/Kesque) have also tested manually the compatibility with the previous version of the sink. It is hard to setup a backward compatibility test and therefore it will add cost on CI. ### Documentation Once this patch is merged I will be happy to enhance the documentation, because we have many new features. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
