goutamadwant opened a new issue, #12838:
URL: https://github.com/apache/seatunnel/issues/12838
### What happens
The Easysearch and Typesense sinks build a document ID from multiple
`primary_keys` by joining the formatted values with `key_delimiter` (default
`_`). This is ambiguous when a key value contains the delimiter. For example,
`["1_1", "2"]` and `["1", "1_2"]` both become `1_1_2`. The second write targets
the same document ID and can overwrite the first row.
This affects these two connectors independently of the
Elasticsearch/OpenSearch case tracked in #12704. A search of open SeaTunnel PRs
found no implementation for Easysearch or Typesense composite-ID encoding.
### Reproduction
1. Configure either sink with `primary_keys = ["part1", "part2"]` and the
default `key_delimiter = "_"`.
2. Write two rows whose key pairs are `("1_1", "2")` and `("1", "1_2")`.
3. Both rows serialize to document ID `1_1_2`. In a local
Easysearch/Typesense service, writing both IDs leaves one document, not two.
### Expected behavior and compatibility
Provide an explicit, opt-in `key_encoding = LENGTH_PREFIXED` mode for
composite keys. Encode each formatted value as `<length>:<value>`, producing
`3:1_11:2` and `1:13:1_2` for the example. Keep `LEGACY` as the default and
leave single-key IDs unchanged. Existing jobs must retain their current IDs
unless users opt in.
Changing the encoding on an existing composite-key index or collection
changes document identity; users must migrate or recreate existing documents
before switching, or later writes may create duplicates instead of updates.
### Proposed scope and validation
Limit the change to Easysearch and Typesense sink option wiring, key
extraction/serialization, focused unit tests, connector E2E tests, and EN/ZH
sink documentation. Test collision and non-collision cases, exact legacy IDs,
single-key behavior, and real local Easysearch and Typesense writes. No new
dependencies or default changes are needed.
The source-level collision and direct-service overwrite have been reproduced
locally. An opt-in implementation has also passed connector unit tests and one
Zeta E2E test against each local service; the PR will link those details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
