Cq-study opened a new pull request, #689: URL: https://github.com/apache/doris-flink-connector/pull/689
# Proposed changes Issue Number: close #620 ## Problem Summary: When mapping MySQL `CHAR`/`VARCHAR` columns to Doris `VARCHAR`, the length is multiplied by 3, assuming at most 3 bytes per character. The widely used `utf8mb4` charset takes up to 4 bytes per character (emoji, supplementary CJK, etc.), so a source column filled with 4-byte characters can exceed the created Doris column's byte limit and cause load failures or truncation. ### What is changed Use a multiplier of 4 instead of 3 for the MySQL paths: - `MysqlType#toDorisType` (`CHAR`/`VARCHAR`) — covers table creation in `mysql-sync-database` and the SQL-parser based schema change (`SQLParserSchemaManager` resolves MySQL types through it) - `JsonDebeziumSchemaChangeImpl` — the legacy MySQL DDL rewrite path Columns whose byte length would exceed 65533 (`length * 4 > 65533`) now fall back to `STRING` accordingly. Add `MysqlTypeTest` covering the new multiplier and the `VARCHAR`/`STRING` boundary (16383 → `VARCHAR(65532)`, 16384 → `STRING`), and update existing test expectations. ### Why MySQL reports column lengths in characters, and JDBC metadata carries no charset info per column, so the connector must assume the widest encoding. `utf8mb4` is the MySQL default since 8.0 and the de-facto standard; multiplying by 3 under-provisions for it. Doris `VARCHAR(n)` is a byte-length upper bound with variable-length storage, so the wider bound does not increase storage cost for existing data. Other database type mappers (Oracle/PostgreSQL/SQL Server/DB2) are intentionally left unchanged in this PR to keep the scope focused; they can follow up separately if desired. ## Checklist(Required) 1. Does it affect the original behavior: Yes (newly created Doris tables get wider VARCHAR columns for MySQL sources; columns between 16384 and 21844 chars now map to STRING instead of VARCHAR) 2. Has unit tests been added: Yes 3. Has document been added or modified: No 4. Does it need to update dependencies: No 5. Are there any changes that cannot be rolled back: No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
