ss666 opened a new pull request, #11414:
URL: https://github.com/apache/seatunnel/pull/11414
### Purpose of this pull request
Closes #9717.
Adds a new `TextChunk` plugin that splits a long text field into multiple
smaller chunks for RAG / embedding pipelines, emitting one row per chunk (a
1-row → N-rows FlatMap). Each output row keeps all original source fields and
appends two columns: the chunk text (`chunk`) and its 0-based index within the
source document (`chunk_index`).
Main points:
1. `TextChunker`: a pure, dependency-free chunking algorithm. It splits
recursively by a priority list of `separators`, then greedily merges pieces up
to `chunk_size`; when `separators` is empty it falls back to fixed-size
windows. Overlap between adjacent chunks is carried as whole trailing pieces
(never mid-word) in the separator path; boundaries are aligned to whole Unicode
code points so a surrogate pair (e.g. an emoji) is never split.
2. `TextChunkTransform` / `TextChunkTransformFactory` /
`TextChunkMultiCatalogTransform`: the SPI wiring — options, check-time
validation, schema extension, and multi-catalog dispatch.
3. Options: `text_field`, `output_field`, `chunk_index_field`, `chunk_size`,
`overlap_size`, `separators`.
4. Docs (`docs/en` + `docs/zh`) and unit/E2E tests added.
### Does this PR introduce _any_ user-facing change?
Yes — a new optional transform plugin `TextChunk`. It is purely additive: no
existing option key, default, public API or SPI contract is changed, so
existing jobs are unaffected. New user-facing docs are added under
`docs/en/transforms/text-chunk.md` and `docs/zh/transforms/text-chunk.md`.
### How was this patch tested?
Scoped to the changed module:
```bash
./mvnw spotless:apply -pl
seatunnel-transforms-v2,seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-1
./mvnw -DskipTests verify -pl seatunnel-transforms-v2
./mvnw test -pl seatunnel-transforms-v2 \
-Dtest='TextChunkerTest,TextChunkTransformTest,TextChunkTransformFactoryTest'
```
E2E: two configs added — `text_chunk_transform.conf` (single table; asserts
exact chunk content and index bounds) and
`text_chunk_transform_multi_table.conf` (per-table override + top-level regex
fallback).
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
— *N/A: no new jar/dependency.*
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs — *Added `docs/en` +
`docs/zh` `text-chunk.md` and registered in `docs/sidebars.js`.*
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. — *N/A: purely additive, no incompatibility.*
* [ ] If you are contributing the connector code, please check that the
following files are updated: — *N/A: this is a Transform-V2 plugin, not a
connector. It registers via `@AutoService` (SPI), and `seatunnel-transforms-v2`
is already in `seatunnel-dist`, so `plugin-mapping.properties`, the dist pom,
and `config/plugin_config` need no change.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
