goutamadwant opened a new pull request, #13173:
URL: https://github.com/apache/seatunnel/pull/13173

   ### Purpose of this pull request
   
   Yes, I am contributing this vector-correctness fix to SeaTunnel.
   
   Related to #9782.
   
   Zeta's Gaussian and sparse VECTOR_REDUCE implementations share a mutable 
random generator. Each call advances that generator, so identical inputs can 
produce different outputs. Different rows therefore do not share the same 
projection, making comparisons between reduced embeddings unreliable.
   
   This change creates a locally seeded generator for each matrix construction. 
The same method and source/target dimensions now produce a consistent 
projection regardless of call order, concurrent evaluation, transform instance 
or process restart.
   
   The fix adds no dependencies, configuration options or global matrix cache. 
Calcite behavior and existing sparse scaling remain unchanged.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. Zeta RANDOM_PROJECTION and SPARSE_RANDOM_PROJECTION results become 
deterministic.
   
   Previously stored projected vectors must be rebuilt from their original 
embeddings before comparing or mixing them with new results. Indexed vectors 
and query vectors must use the same projection method and dimensions.
   
   SQL syntax, options, schemas and checkpoint formats are unchanged. TRUNCATE 
and no-reduction paths are unchanged.
   
   English and Chinese function documentation and incompatible-change notes 
include the migration guidance.
   
   ### How was this patch tested?
   
   - Reproduction: 9 of 10 new regression cases failed with the original 
implementation.
   - JDK 11: all 1,967 transform tests passed without failures or skips.
   - JDK 17: all 1,967 transform tests passed without failures or skips.
   - Regression coverage includes interleaved calls, seeded reference results, 
linearity, concurrent calls, separate transform instances and fresh JVMs.
   - Existing Calcite vector tests passed.
   - Docker Zeta: TestSQLIT#testVectorFunctions passed with assertions 
requiring zero distance between repeated Gaussian and sparse projections.
   - Negative control: replacing only VectorFunction with its original base 
implementation made the same Docker test fail the projection assertion.
   - The 66-module E2E prerequisite build, scoped verification and Spotless 
passed.
   
   Unit tests: ./mvnw -pl seatunnel-transforms-v2 test
   
   Docker E2E, after building its prerequisites: RUN_ENGINES=zeta ./mvnw -f 
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2/pom.xml
 -DskipUT=true -DskipIT=false -Dit.test=TestSQLIT#testVectorFunctions verify
   
   The full repository suite, other engine matrices and actual worker-crash 
recovery were not run. Restart determinism is covered by fresh-JVM tests.
   
   ### Check list
   
   - [ ] New JAR license/notice requirements: not applicable; no new 
dependencies or binaries.
   - [x] Update English and Chinese documentation.
   - [x] Update incompatible-changes.md with migration guidance.
   - [ ] Connector registration checklist: not applicable; no connector changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to