morningman opened a new pull request, #70351:
URL: https://github.com/apache/doris/pull/70351
### Background
Connector plugins (Hudi, and the coming Lance connector) describe their
columns with the SPI's `ConnectorType`, and fe-core's
`ConnectorColumnConverter` turns each into a Doris type. Connectors put a
temporal column's fractional-second scale in the type's precision field:
`("DATETIMEV2", 3, 0)` is `datetime(3)`, and Hudi's `HudiTypeMapping` sends
`("TIMEV2", 3, 0)` for an Avro `time-millis` column and `("TIMEV2", 6, 0)` for
`time-micros`.
### What problem does this PR solve?
Issue Number: None
Related PR: #62183 (introduced the converter without a TIMEV2 case), #64304
(moved Hudi onto it)
PR Type: Bugfix
Problem Summary:
Since Hudi became a connector plugin, its `time-millis` and `time-micros`
columns show as `time` instead of `time(3)` and `time(6)`, which is what 4.x
shows and what `HudiTypeMapping` asks for: the converter drops the scale and
builds a scale-0 TIMEV2. The Lance connector, ported in a follow-up, maps Arrow
`time32(ms)` and `time64` the same way and would lose its scale too. The
converter now reads the TIMEV2 scale from the precision field, as it already
does for DATETIMEV2.
### Details
- **Frequency:** Always, for every connector column whose `ConnectorType` is
TIMEV2 with a scale; today that is every Hudi `time-millis` / `time-micros`
column on master. Found while porting the Lance connector, by reading the
converter, and confirmed by the unit test below.
- **Root cause:** `ConnectorColumnConverter.convertScalarType` reads the
precision field for CHAR, VARCHAR, VARBINARY, the decimals, DATETIMEV2 and
TIMESTAMPTZ, but has no TIMEV2 case, so TIMEV2 falls through to
`ScalarType.createType("TIMEV2")`, which returns the scale-0
`ScalarType.TIMEV2`.
- **Why existing tests missed it:** Unit tests:
`ConnectorColumnConverterTest` checks decimals, CHAR/VARCHAR lengths and the
TIMESTAMPTZ extra info but has no TIMEV2 case, and `HudiTypeMappingTest`
asserts the `("TIMEV2", 3, 0)` the plugin emits but stops before fe-core
converts it. Regression: no suite under `regression-test/suites` reads a Hudi
time column.
- **Fix:** The TIMEV2 case uses the precision field as the scale when one is
given; without one the type stays the scale-0 TIMEV2 it has always been. The
fix sits in the converter because every connector goes through it. The reverse
direction, `toConnectorType`, is unchanged: only the read-only Hudi and Lance
connectors produce TIMEV2, and the JDBC connector maps TIME to STRING.
- **Other paths and branches:** `toConnectorType` is not affected for reads
(see Fix). branch-4.1 and branch-4.2 are not affected: they have no
`ConnectorColumnConverter`, and their fe-core Hudi mapping (`HudiUtils`) builds
`createTimeV2Type(3)` / `createTimeV2Type(6)` directly. The bug is on master
since #64304.
### Testing
- `ConnectorColumnConverterTest#timeScaleRidesInThePrecisionSlot` (new):
fails before the fix with `expected: <timev2(3)> but was: <timev2(0)>` and
passes after; it also pins that a TIMEV2 without a precision stays scale 0. Run
locally on master `f5c993f2c94` with #70338 applied, which touches neither file.
- `ConnectorColumnConverterTest` (35 tests) and `HudiTypeMappingTest` (17
tests): pass locally on the same tree.
### Release note
None
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5)
- Generated: code, tests, this description
- Human review: not stated
- Prompts: The author asked to port the branch-4.2 Lance FE read path to
master as a connector plugin; this fix came out of that port, and the author's
rule for such finds is to fix them in their own PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
