SEPURI-SAI-KRISHNA opened a new pull request, #13090:
URL: https://github.com/apache/seatunnel/pull/13090
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Closes #12988.
The Elasticsearch sink fails on an Elasticsearch 9 cluster. When the writer
starts, the sink reads the cluster version to decide whether a bulk action may
carry a mapping type, and `ElasticsearchVersion` only knows the majors 2, 5, 6,
7 and 8:
```
ErrorCode:[ELASTICSEARCH-02], ErrorDescription:[Get elasticsearch version
failed] - version=9,fail fo find ElasticsearchVersion.
```
This PR adds `ES9`. It takes the same path as `ES7` and `ES8` in
`IndexTypeSerializerFactory`, so bulk actions carry no `_type`. Elasticsearch
9.5.5 rejects the whole bulk request otherwise (`Action/metadata line [1]
contains an unknown parameter [_type]`). The source and the catalog do not
parse the version, so they need no change.
Before adding only the enum value, I checked the two Elasticsearch 9
breaking changes that could have touched this connector, against a real
`elasticsearch:9.5.5`:
- **Stricter bulk action parsing (9.0).** Every action line the sink writes
is accepted: `index` with and without `_id`, `update` with `doc_as_upsert`, and
`delete`.
- **Vectors left out of `_source` by default (9.2, new indices).** A search
with no `_source` list drops a `dense_vector` field, but the scroll and PIT
searches always send an explicit field list (the schema fields, `source`, or
every mapped field), and Elasticsearch then returns the vector. So nothing is
lost. One Elasticsearch behaviour to know about: with the default `cosine`
similarity, the returned values come from the normalized indexed form, so some
can differ in the last float digit (`[0.1, 0.2, 0.3]` reads back as
`[0.099999994, 0.19999999, 0.3]`). `l2_norm`, `dot_product` and
`max_inner_product` return the exact values. The connector passes the values
through unchanged.
### Does this PR introduce _any_ user-facing change?
Yes. The Elasticsearch sink now works with Elasticsearch 9.x, and the docs
state the supported range as 2.x to 9.x (EN and ZH, source and sink). Nothing
changes for 2.x to 8.x: the mapping of those versions is untouched. No option
or default changed.
### How was this patch tested?
**Unit test.**
`ElasticsearchRowSerializerTest#testSerializeForElasticsearch9` builds the
serializer for a `9.5.5` cluster with a primary key and `index_type` set, and
checks that the upsert and delete lines carry no `_type`. Without the `ES9`
value it fails with the `ELASTICSEARCH-02` error above.
**E2E.** New `Elasticsearch9IT` on `elasticsearch:9.5.5` with TLS and basic
auth, in its own class so the existing ITs stay on 8.9.0 and no test pays for a
second container. Per engine it runs:
1. FakeSource to the Elasticsearch sink with insert, update and delete rows
by primary key, `CREATE_SCHEMA_WHEN_NOT_EXIST` and `DROP_DATA`, then checks the
documents directly: ids 1 and 2 with the updated value, id 3 deleted.
2. Reads the index back with scroll, PIT with `slice_max = 2`, and SQL
search, each into `Assert` (exactly 2 rows).
**Local runs against a real Elasticsearch 9.5.5** (Docker, plain HTTP, Zeta
local mode):
- On `dev`: the sink job fails with the error above. Equivalent scroll, PIT,
SQL and `dense_vector` read jobs already pass.
- With this change: the four job configs of `Elasticsearch9IT`, with only
the host and credentials pointed at the local container, all pass. A second
sink run with a stale document in the index shows `DROP_DATA` removing it.
- A negative control (the PIT job expecting 3 rows) fails, so the `Assert`
row rules really count across both subtasks.
Checks, JDK 17:
- `./mvnw test -pl seatunnel-connectors-v2/connector-elasticsearch`: 75
tests, 0 failures.
- `./mvnw spotless:check -pl
seatunnel-connectors-v2/connector-elasticsearch,seatunnel-e2e/seatunnel-connector-v2-e2e/connector-elasticsearch-e2e`:
clean.
- `./mvnw -pl
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-elasticsearch-e2e -am
-DskipTests package`: compiles.
### Check list
* [X] Code changed are covered with tests, or it does not need tests for
reason
* [X] If any new Jar binary package adding in your PR, please add License
Notice according to
[License
Notice](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md):
none added.
* [X] If necessary, please update the documentation to describe the new
feature: the supported version range in the EN and ZH source and sink docs.
* [X] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties):
not needed, no new connector.
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml):
not needed, no new module.
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml):
not needed, the module is already labelled.
4. Add e2e test for your connector: `Elasticsearch9IT`.
5. Update [connector-v2
doc](https://github.com/apache/seatunnel/tree/dev/docs/en/connectors): done,
version range only.
* [X] Update the `release-note`: not applicable, the file was removed from
`dev` in #9304.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
