SteNicholas opened a new pull request, #478:
URL: https://github.com/apache/paimon-cpp/pull/478
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #477
<!-- What is the purpose of the change -->
Paimon Java accepts `TIME` columns in Lance data files
(`LanceFileFormat.LanceRowTypeVisitor.visit(TimeType)`) and writes them as
Arrow `TIMEMILLI`
(`time32[ms]`) through `ArrowFieldTypeConversion`. Paimon C++ has
represented `TIME` as
`time32[ms]` since #378, but rejected it for Lance:
- `SchemaValidation::ValidateLanceDataField()` had no `TIME32` case, so a
TIME column, at top
level or nested in ARRAY or ROW, failed with `Lance file format does not
support type
time32[ms]`.
- `LanceStatsExtractor::CreateEmptyStats()` had no `TIME32` case either, so
lifting only the
validation would still fail to finalize an append data file with a TIME
column, with
`cannot create empty Lance statistics for type time32[ms]`.
This change:
- Accepts `TIME32` in `ValidateLanceDataField()`. The existing recursion
covers TIME nested in
ARRAY and ROW, and `TableSchema::Create()` still rejects non-millisecond
units.
- Returns empty integer statistics for `TIME32` in
`LanceStatsExtractor::CreateEmptyStats()`,
as `VortexStatsExtractor` does.
- Needs no reader change: the declared precision comes from the
`paimon.time.precision` field
metadata of the table schema, and `LanceFileBatchReader` aligns each batch
to the read type.
### Tests
<!-- List UT and IT cases to verify this change -->
- `SchemaValidationTest.TestLanceDataTypes`: millisecond TIME at top level
and in
`ROW<ARRAY<TIME>>` moves from the rejected to the accepted fields;
second-unit TIME, top level
and nested, is still rejected.
- `LanceFileFormatTest.WriteThenReadSupportedTypes`: adds a top-level TIME
column and
`ARRAY<TIME>`, `ROW<TIME>` and `ROW<ARRAY<TIME>>` columns declared with
precisions 0, 3, 6 and
9. The data covers 0 and 86399999, null and empty arrays, null elements
and null ROW children,
and is read with batch sizes 1, 2 and 4. A batch imported directly from
the reader checks that
the returned Arrow type keeps the read schema's field metadata, including
`paimon.time.precision`.
- `LanceFileFormatTest.ExtractStatistics`: a TIME column yields an empty
statistics entry.
- `TimeColumnWriteInteTest.TestAppendTimeColumnFinalize`, generalized from
`VortexTimeColumnWriteInteTest`: append write, finalize and read of a TIME
column for both
Lance and Vortex.
The README of the `lance/append_java_compat` fixture no longer says the C++
table path does not
support TIME. A Paimon Java compatibility fixture with TIME columns is left
for a follow-up.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No API changes. No storage format changes: TIME is written to Lance as
`time32[ms]`, as in
Paimon Java.
### Documentation
<!-- Does this change introduce a new feature -->
Yes. `docs/source/user_guide/data_types.rst` no longer lists TIME as not
supported. It describes
the `time32[ms]` mapping, how the declared precision is kept in
`paimon.time.precision` while
values keep millisecond precision, and which file formats read or write TIME.
### Generative AI tooling
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
