goutamadwant opened a new pull request, #13173: URL: https://github.com/apache/seatunnel/pull/13173
### Purpose of this pull request Yes, I am contributing this vector-correctness fix to SeaTunnel. Related to #9782. Zeta's Gaussian and sparse VECTOR_REDUCE implementations share a mutable random generator. Each call advances that generator, so identical inputs can produce different outputs. Different rows therefore do not share the same projection, making comparisons between reduced embeddings unreliable. This change creates a locally seeded generator for each matrix construction. The same method and source/target dimensions now produce a consistent projection regardless of call order, concurrent evaluation, transform instance or process restart. The fix adds no dependencies, configuration options or global matrix cache. Calcite behavior and existing sparse scaling remain unchanged. ### Does this PR introduce _any_ user-facing change? Yes. Zeta RANDOM_PROJECTION and SPARSE_RANDOM_PROJECTION results become deterministic. Previously stored projected vectors must be rebuilt from their original embeddings before comparing or mixing them with new results. Indexed vectors and query vectors must use the same projection method and dimensions. SQL syntax, options, schemas and checkpoint formats are unchanged. TRUNCATE and no-reduction paths are unchanged. English and Chinese function documentation and incompatible-change notes include the migration guidance. ### How was this patch tested? - Reproduction: 9 of 10 new regression cases failed with the original implementation. - JDK 11: all 1,967 transform tests passed without failures or skips. - JDK 17: all 1,967 transform tests passed without failures or skips. - Regression coverage includes interleaved calls, seeded reference results, linearity, concurrent calls, separate transform instances and fresh JVMs. - Existing Calcite vector tests passed. - Docker Zeta: TestSQLIT#testVectorFunctions passed with assertions requiring zero distance between repeated Gaussian and sparse projections. - Negative control: replacing only VectorFunction with its original base implementation made the same Docker test fail the projection assertion. - The 66-module E2E prerequisite build, scoped verification and Spotless passed. Unit tests: ./mvnw -pl seatunnel-transforms-v2 test Docker E2E, after building its prerequisites: RUN_ENGINES=zeta ./mvnw -f seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/pom.xml -DskipUT=true -DskipIT=false -Dit.test=TestSQLIT#testVectorFunctions verify The full repository suite, other engine matrices and actual worker-crash recovery were not run. Restart determinism is covered by fresh-JVM tests. ### Check list - [ ] New JAR license/notice requirements: not applicable; no new dependencies or binaries. - [x] Update English and Chinese documentation. - [x] Update incompatible-changes.md with migration guidance. - [ ] Connector registration checklist: not applicable; no connector changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
