DanielLeens commented on issue #12707: URL: https://github.com/apache/seatunnel/issues/12707#issuecomment-6091616127
## Maintainer review: how to handle this reasonably The problem is real and SeaTunnel already owns most of the parts (CDC sources, `EmbeddingTransform`, Milvus / Qdrant sinks). But as written this issue bundles three independent families of work, and it cannot be implemented or reviewed as one change: 1. CDC changelog semantics for embedding and vector sinks 2. New sinks (pgvector, Chroma) and row-to-document mapping 3. `pip install` packaging My suggestion is to keep this issue as the umbrella / discussion and split the work into child issues, ordered by dependency and risk. The sections below give my reasoning and a correction to my own packaging argument. ### 1. Packaging (pip install) **Correction first.** I initially treated the 472 MB v3.0.0 binary tarball as too big for PyPI. That is overstated: `apache-flink-libraries` is a 329 MB wheel (split from a 2.7 MB `apache-flink` Python wheel) and PySpark ships a 450 MB sdist, so large jar-bundling packages have ASF precedent. Size is not the blocker. The real decision is the release process: what exactly gets voted, and who publishes it. **Recommendation: thin launcher first, split "libraries" wheel only if needed.** - Thin launcher: the wheel stays small, and the engine is the already voted ASF binary downloaded on first run. There is one source of truth, no second 400+ MB artifact to vote and publish per release, and it matches how connectors already work (`install-plugin.sh` pulls from Maven Central). - Cost: first run needs network. Offline users get an override (existing `SEATUNNEL_HOME`, or a local tarball path / mirror variable). - Flink-style split (small wheel + `...-libraries` wheel with the engine inside) is the follow-up only if there is real air-gapped demand. It needs a PyPI size exception and a decision on how the second artifact is released. **Do not add a third Python package.** The repo already has two, and both would collide on publication: - `tools/seatunnel-python-sdk`: distribution name `seatunnel`, REST client (`httpx`), unpublished. - `seatunnel-cli`: console script `seatunnel`, its own `seatunnel_cli/rest.py` that duplicates REST calls. One naming decision is needed before anything is published (including whether the ASF `apache-` prefix is required, which I would confirm with the PMC / brand policy, not assume). Then the SDK should be the single REST layer, the launcher reuses it for cluster submit, and the AI CLI becomes an optional extra. **Requirements I would put on the launcher:** - Verify sha512 (and ideally the `.asc` against `KEYS`) before extracting. For 3.0.0 all three exist on `downloads.apache.org`. Verify against the main ASF site, not a mirror. - Wheel version equals engine version. No floating "latest". - Fail fast on Java below 11 with an actionable message (same check as `seatunnel.sh`). Do not auto-install a JRE in phase 1. - Do not inherit the default checkpoint location in local mode. `config/seatunnel.yaml` points the checkpoint namespace at `/tmp/seatunnel/checkpoint_snapshot` (`fs.defaultFS: file:///tmp/`). A CDC job resumed after a temp cleanup or reboot would lose its position. The launcher should generate a per-user data directory config. - Document that `-m local` is a single process without HA. pip mode is for development and small jobs; production still goes through a cluster (REST submit). **Tests that can fail:** fresh virtualenv install, then a `FakeSource -> Console` job in local mode; checksum mismatch aborts without extracting; missing / old Java gives the expected error; second run works offline from the cache. ### 2. CDC -> vector: proposed split and order 1. **Row-kind-aware `EmbeddingTransform`, opt-in.** Today `DELETE` / `UPDATE_BEFORE` rows go through the model. Skipping them is the smallest, stateless, independently testable change. It must default to off: existing jobs currently receive vectors on those rows, and null vectors could break a sink with a non-null schema. Test: a `DELETE` row makes no model call and keeps its `RowKind`. 2. **Skip-unchanged is a separate, harder design.** It needs before/after pairing or a content cache. In `nlpmodel` I only found `ModelInvocationCache` as an interface with a no-op implementation, so a real cache needs its own design (key, size bound, persistence, interaction with checkpoint). At the scale we review for (a single job in the tens of billions of rows) an unbounded cache is an OOM risk, so I would not put this in phase 1. 3. **Changelog contract (STIP-style), connector-local.** Row kind to upsert / delete by primary key, `UPDATE_BEFORE` handling, ordering, idempotency, restart behaviour. It must be built on #10917 / #10919 and not duplicate them. Define the model failure policy here too (fail job, null vector, or route out). 4. **pgvector: prove before building.** I found no pgvector reference in the codebase, so it is unproven in both directions whether the existing JDBC sink + PostgreSQL dialect can write a `vector` column. First establish that (type mapping from float arrays, the literal form, schema creation). Only if the JDBC path cannot carry it should a separate connector be proposed, with the usual boundary evidence. 5. **Row-to-document mapping: ask for a concrete example first.** The SQL transform (e.g. `CONCAT`) may already cover simple templates; I have not validated this. Add an operator only for a case it cannot express. 6. **Chroma: defer.** Every sink is a long-term maintenance and E2E-container cost, and there is no demand evidence here beyond one example project. Revisit when a user asks. 7. **Docs and E2E last,** with a mock embedding server so CI makes no external API calls. ### Decisions needed before child issues are opened - PyPI project name and release path (PMC). - Thin launcher vs split libraries wheel. - Whether item 1 (opt-in row-kind skipping) is agreed as phase 1 of the CDC work. Once those are settled, the child issues can be opened and linked here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
