fantapsody commented on issue #5690: Elasticsearch 5.x Sink connector incompatible URL: https://github.com/apache/pulsar/issues/5690#issuecomment-572860087 I have done some research in documents and some change histories of Elasticsearch, the problem is fundamentally related to the long term "[removal of types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html)" project in Elasticsearch. In brief, Elasticsearch supported multiple [types](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/_basic_concepts.html#_type) before 6.0. However, they found it to be a problematic abstraction, and decided to remove it gradually. The road map is to support only a single type start from 6.0, with the default type name set to be "_doc" since [6.2](https://github.com/elastic/elasticsearch/pull/27816), and to support the type name "_doc" only(which will actually be considered as the name of the API endpoint rather than a type name) since 7.0, and finally remove the concept of type since 8.0, while "_doc" will be kept in the document APIs to make them backward compatible. On the other hand, names with a leading underscore are generally considered as reserved for internally use in Elasticsearch, and are not allowed to be used in user provided names (index, field, type, ...). And "_doc" is a special type name added in 6.2 to make the transition to 7.0 smoother, so it will not be supported before version 6.2. As a result, "_doc" is supported as the only type name since 7.0 while not supported before 6.2, it is likely not possible to specify a default type name without knowing the version of the destination cluster(In fact, we have changed the type name from "doc" to "_doc" to support 7.0 in https://github.com/apache/pulsar/pull/4322). To fix the issue, we can provide a configuration "typeName" with the default value set to be "_doc", so it supports versions of Elasticsearch since 6.2 by default, and can also support the previous versions if users configure a valid type name. Auto detection of the version of the cluster is possible but may not be very valuable for current feature sets.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
