Dev-next-gen opened a new pull request, #12327:
URL: https://github.com/apache/seatunnel/pull/12327
### Purpose of this pull request
I found that `CreateTableParser.getColumnList`, which Doris and MaxCompute
use to pick up the columns a user wrote by hand in `save_mode_create_template`,
splits column definitions on every `,` and stops at the first unbalanced `)`,
including those inside a quoted string. So a template column like this:
```sql
CREATE TABLE IF NOT EXISTS `${database}`.`${table}` (
${rowtype_primary_key},
`level` VARCHAR(10) COMMENT 'low,medium,high',
${rowtype_fields}
) ENGINE=OLAP
UNIQUE KEY (${rowtype_primary_key})
DISTRIBUTED BY HASH (${rowtype_primary_key})
```
is read as three pieces, `` `level` VARCHAR(10) COMMENT 'low ``, `medium`
and `high'`. The bare `medium` looks like a column name with no type, so
`DorisCatalogUtil.mergeColumnInTemplate` tries to expand it and the sink fails
with `java.lang.IllegalArgumentException: Can't find column medium in table.` A
`)` inside a comment, such as `COMMENT 'seconds)'`, ends parsing early, so
later hand-written columns are not recognised and get emitted a second time
through `${rowtype_fields}`.
The fix tracks whether the scanner is inside `'...'`, `"..."` or `` `...` ``
and treats commas and parentheses there as ordinary characters. A
backslash-escaped quote stays inside the literal, matching how
`DorisCatalogUtil.columnToDorisType` escapes comments. The `''` form works on
its own, since it just closes and reopens the quote. Start and end offsets are
still computed the same way, so replacing bare column names in the template
works as before.
The same class exists as identical copies in `connector-common` (used by
Doris), `connector-maxcompute` (used by `MaxComputeCatalogUtil`) and
`connector-clickhouse`. I applied the same change to all three so they stay
identical.
### Does this PR introduce _any_ user-facing change?
Yes, for templates with quoted text that contains `,` or `)`. Before, the
template above failed with `Can't find column medium in table.` Now it produces:
```sql
CREATE TABLE IF NOT EXISTS `db`.`tbl` (
`id` BIGINT NOT NULL ,
`level` VARCHAR(10) COMMENT 'low,medium,high',
`name` STRING NULL
) ENGINE=OLAP
UNIQUE KEY (`id`)
DISTRIBUTED BY HASH (`id`)
```
Templates without such characters produce the same statement as before.
### How was this patch tested?
I added `CreateTableParserTest` in `connector-common`. It uses a template
with a comma-separated comment, a comment ending in `)`, a backslash-escaped
quote followed by a comma, and a plain column after them. It also checks that
the offsets of a bare column still point at its name. Before the change it
fails with `expected: <[note, unit, create_time, level, id]> but was: <[high,
unit, level, id, medium]>`, and after the change it passes.
I also ran the Doris template above through
`DorisCatalogUtil.getCreateTableStatement` in a throwaway test, before and
after, which is where the error message and the statement quoted above come
from.
Existing unit tests of the affected modules pass on JDK 17: connector-doris
130/130 (including `DorisCreateTableTest` and `DorisCatalogUtilTest`),
connector-maxcompute 63/63 and connector-clickhouse 48/48. In connector-common,
`ArrowToSeatunnelRowReaderTest` fails locally because Arrow needs
`--add-opens=java.base/java.nio` on that JDK. That's unrelated to this change,
and the other connector-common tests pass. `spotless:apply` was run.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
Found by a defect-hunting pipeline I build and run
([Dev-next-gen](https://github.com/Dev-next-gen)), using Claude Code with
Anthropic's Claude Opus 5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
