JNSimba opened a new pull request, #69948:
URL: https://github.com/apache/doris/pull/69948
### What problem does this PR solve?
Issue Number: None
Related PR: None
Streaming jobs cannot currently synchronize MySQL VECTOR columns: source DDL
parsing and row-event decoding lack the type, and snapshot reads with the
current JDBC driver do not return the required bytes. This affects ordinary
VECTOR columns and both VIRTUAL and STORED generated columns.
This change reads VECTOR values as numeric arrays and maps them to Doris
`ARRAY<FLOAT>` for automatic table creation and runtime ADD COLUMN. It adds the
required parser, binlog metadata/row decoding, snapshot reads and value
conversion while retaining existing dependency versions. The copied classes
retain their existing compatibility fixes and document their source and changes.
MySQL metadata connections default to `useInformationSchema=true` to
discover the native type name. Explicit settings are preserved, and the CDC
reader's JDBC URL is unchanged. Adding a generated column does not backfill
historical target rows.
The existing version smoke test now includes MySQL 9.7.4. A dedicated VECTOR
ITCase covers PLAIN/VIRTUAL/STORED snapshot and incremental reads, NULL,
multiple dimensions, trailing columns, runtime ADD and reader reconstruction.
### Release note
Support synchronizing MySQL VECTOR columns, including VIRTUAL and STORED
generated columns, into Doris `ARRAY<FLOAT>` through streaming jobs, including
automatic target creation and runtime column additions. Add MySQL 9.7.4
compatibility smoke coverage.
### Check List (For Author)
- Test:
- [x] Unit Test: 31 targeted FE tests and all 273 CDC unit tests passed.
- [x] Integration Test: 3 version smoke scenarios (5.7, 8.4, 9.7.4) and 6
VECTOR scenarios passed, with no failures, errors or skips. These use real
MySQL and a mock Doris receiver.
- [x] Manual test: a separate single-FE/single-BE cluster passed
automatic/precreated target tables, snapshot plus INSERT/UPDATE/DELETE/NULL,
dynamic ADD, idempotent ADD, exclude/ignore, and recovery after actual FE and
BE restarts for all three VECTOR forms. Verified concrete values such as
`[1.25,-2.5,0,4]` and `[5,-6.25,7.5,8]`, target `ARRAY<FLOAT>`, and unaffected
trailing columns. Ordinary dynamic ADD was repeated five times per form.
- [x] FE/CDC standard build and its style checks passed.
- [ ] Regression test: no new Groovy suite or external MySQL CI
environment in this PR.
Validation was performed on the reviewed implementation at base
`04aeab8171a`. The feature patch was then applied cleanly to master
`378404eeb12`, preserving its newer JDBC exception wrapping. Post-test edits
only remove trailing grammar whitespace; all other feature files match the
validated sources except that preserved upstream change. The full build/test
suite has not been rerun after this base update. The remote IT execution copy
also disabled native AIO for its host environment.
Not covered: PolarDB-specific VECTOR encoding, multi-BE failover, real CDC
TVF end-to-end ingestion, TLS/cold-authentication matrices, performance or full
MySQL 9 compatibility certification.
- Behavior changed:
- [x] Yes. VECTOR columns can be synchronized and mapped to array columns
as described above.
- Does this need documentation?
- [x] Yes. The supported source-type mapping and MySQL version validation
scope need a website documentation follow-up; no website PR is included here.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
