DanielLeens opened a new issue, #12707: URL: https://github.com/apache/seatunnel/issues/12707
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description ## Background The Debezium community recently shipped **PyDebeziumAI v0.1.0** (a GSoC 2026 project, now on PyPI). It brings real-time change data capture into AI/LLM applications: database changes are captured as they happen and kept in sync with vector stores (Chroma, PGVector, Milvus), with pluggable document transformation and LangChain / LangGraph integration. This confirms a concrete and growing demand: **keeping a vector index continuously consistent with an operational database, end to end, with no custom glue code.** SeaTunnel is already well positioned for this (it has CDC sources, an embedding transform, and Milvus / Qdrant sinks), but the pieces are not yet a first-class, production-grade pipeline. This issue tracks the *CDC -> embedding -> vector store* path specifically, and does not duplicate the document-oriented RAG work. ## What SeaTunnel already has (verified on `dev`) - CDC sources: `connector-cdc-mysql`, `-postgres`, `-oracle`, `-sqlserver`, `-mongodb`, `-db2`, `-oceanbase`, `-opengauss`, `-tidb`, `-vitess`. - `EmbeddingTransform` (`seatunnel-transforms-v2/.../nlpmodel/embedding`) with multi-field `vectorization_fields`, plus a `chunk` transform. - Vector sinks: `connector-milvus`, `connector-qdrant`. ## Gaps for the CDC -> vector scenario 1. **No row-kind-aware embedding.** `EmbeddingTransform` contains no `RowKind` handling, so (inferred from source, to be confirmed by a test) every changelog event, including `DELETE` and `UPDATE_BEFORE`, goes through the embedding model. Embedding calls are the dominant cost of a CDC-to-vector job, and `DELETE` / `UPDATE_BEFORE` should never need one. Unchanged source columns on an `UPDATE_AFTER` should also be able to skip re-embedding. 2. **Changelog semantics on vector sinks.** CDC `INSERT / UPDATE_AFTER` must become upsert by primary key and `DELETE` must become delete by primary key, with a defined behaviour for `UPDATE_BEFORE`. The open Qdrant / Milvus lifecycle PRs (#10917, #10919) cover the document-chunk lifecycle; the row-level (primary key) changelog contract on top of them is not specified anywhere. 3. **Row-to-document conversion.** A database row has to be turned into the text that is embedded plus the metadata that is stored next to the vector (for example a template such as `"{title}: {description}"`, a selectable metadata column set, and the source primary key / table / commit position as payload). Today this requires chaining generic transforms by hand. 4. **Missing popular vector targets.** There is no **pgvector** sink (a PostgreSQL `vector` column through JDBC) and no **Chroma** sink. pgvector is the natural target for teams that already run PostgreSQL and do not want a separate vector database. 5. **No reference pipeline or documentation** that shows `MySQL-CDC / Postgres-CDC -> Embedding -> Milvus / Qdrant / pgvector` with delete propagation, restart / checkpoint recovery and backfill-then-incremental behaviour. ## Proposed scope (design first, to be split into small PRs) 1. Design note (STIP-style) defining the changelog contract between CDC sources, the embedding transform and vector sinks: row kind mapping, primary key requirements, ordering and idempotency, and what happens on checkpoint recovery. 2. Row-kind-aware behaviour in `EmbeddingTransform`: pass `DELETE` / `UPDATE_BEFORE` through without invoking the model, and an optional skip-unchanged mode for updates. 3. Row-to-document support (text template plus metadata mapping), kept inside the transform layer with no engine / core / API change. 4. pgvector sink, preferably as a JDBC dialect capability on top of the existing JDBC sink instead of a new engine abstraction. Chroma sink as a separate, lower-priority item. 5. An end-to-end E2E and a user guide for the CDC -> vector pipeline. ## Non-goals - Query engine, retriever, reranker, chatbot or agent runtime (consistent with the positioning in #10889). - Python / LangChain / LangGraph bindings. SeaTunnel should stay the data movement layer; framework integrations belong to the consumer side of the vector store. - A new shared "vector sink" abstraction in core before at least two concrete sinks have proven the need. ### Usage Scenario - A product catalog or support-ticket table in MySQL / PostgreSQL is continuously indexed into Milvus, Qdrant or pgvector so that a RAG or semantic-search application always reads fresh data. - A row deleted in the source database is removed from the vector index within the CDC latency (compliance and "no stale answers" requirement). - Embedding cost control: only changed rows hit the embedding model; deletes and before-images do not. - First run backfills the existing table (snapshot), then continues with incremental changes, with exactly the same row-to-document mapping. ### Related issues - #10889 [Umbrella][RAG] Knowledge Sync (document-oriented; this issue is the database / CDC counterpart) - #10917 PR-E: Add Qdrant lifecycle sink V1 - #10919 PR-G: Add Milvus lifecycle and vector lifecycle base - #11304 Agent Memory / Context Data Integration (CDC freshness is one of its scenarios) - #11758 Object storage CDC (changelog semantics for object storage) - #9713 Connector prepare for RAG ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
