SteNicholas opened a new pull request, #479:
URL: https://github.com/apache/paimon-cpp/pull/479
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #476
<!-- What is the purpose of the change -->
Paimon Java accepts `TIMESTAMP_LTZ` (`TIMESTAMP WITH LOCAL TIME ZONE`)
columns in Lance data files
since apache/paimon#10558. Paimon C++ rejected every Arrow timestamp with a
time zone in
`SchemaValidation::ValidateLanceDataField()` with `Lance file format does
not support type
LOCAL_ZONED_TIMESTAMP`, whether Lance was selected by `file.format`,
`file-format-per-level` or
`changelog-file-format`.
Removing that check alone is not enough, because the two sides put different
time zones on the
Arrow type:
- Java always writes LTZ as Arrow `Timestamp(unit, "UTC")`
(`ArrowFieldTypeConversion`), and its
validator relies on that: Lance 0.39 stores the type as
`timestamp:<unit>:<zone>` and cannot
parse a zone id containing `:` such as `GMT-10:00`.
- C++ builds the LTZ Arrow type with
`DateTimeUtils::GetLocalTimezoneName()`, a host-dependent
zone id that may contain `:`.
This change:
- Accepts zoned timestamps in `ValidateLanceDataField()`, including inside
ARRAY and ROW, as
Java's `LanceRowTypeVisitor` does.
- Makes `LanceFormatWriter` import each batch with a copy of the writer
schema whose zoned
timestamps, top level and nested, carry the `UTC` time zone. The values
are instants since the
epoch, so only the Arrow type changes, and the file stores what Java
writes whatever the local
time zone is.
- Relabels the `UTC` timestamps Lance returns with the read schema's local
zone in
`LanceFileBatchReader::AlignBatch()`, before `AlignArrayToReadType()`,
which would otherwise
reject the zone difference. The helper, `AlignTimestampTimezones`, moves
from the Vortex
reader's anonymous namespace into `NestedProjectionUtils` and is used by
both readers. It only
relabels a timestamp when both sides have a zone and the same unit.
- Needs no statistics change: `LanceStatsExtractor::CreateEmptyStats()`
already handles
`TIMESTAMP` whatever its zone.
### Tests
<!-- List UT and IT cases to verify this change -->
- `SchemaValidationTest.TestLanceDataTypes`: LTZ at precisions 0, 3, 6 and 9
with `UTC`,
`Asia/Shanghai`, `GMT-10:00` and `+08:00` zones, in ARRAY and in
`ROW<ARRAY<...>>`, moves from
the rejected to the accepted fields. An LTZ array is also accepted when
Lance is selected
through `file-format-per-level` or `changelog-file-format`.
-
`NestedProjectionUtilsTest.AlignTimestampTimezonesRelabelsZonedLeavesOfSameUnit`:
top-level,
ROW and ARRAY leaves are relabelled; identical types, a unit change and a
timestamp without a
zone on either side are left unchanged. It also covers sliced input and
checks that the input
is not modified.
- `LanceFileFormatTest.WriteThenReadSupportedTypes`: adds LTZ columns at
precisions 0, 3, 6 and 9,
`ARRAY<LTZ>` and `ROW<LTZ>` in the `GMT-10:00` zone, reads them back in
that zone and checks
that the file schema stores `UTC`.
- `WriteAndReadInteTest.TestAppendTimestampType` and `TestPkTimestampType`
now also run for
Lance.
- `LanceTimestampLtzInteTest.TestWriteAndReadInPinnedTimezone`: append and
primary-key tables
under `UTC`, `Asia/Shanghai` and `GMT-10:00`, with nested LTZ, an LTZ
column added by schema
evolution and a reordered projection.
- `ScanAndReadInteTest.TestLanceLtzCompatibility`: reads the new
`lance/append_ltz_java_compat`
fixture (Paimon Java at `535a5c9`, JVM time zone `GMT-10:00`) and
`lance/append_ltz_python_compat` fixture (Paimon Python at `63ffd40`)
under `Asia/Shanghai`, at
both snapshots and with a reordered projection. Both fixtures hold the
same LTZ(0/3/6/9),
`ARRAY<LTZ(6)>` and `ROW<number INT, ts LTZ(3)>` values, and both writers
store every LTZ leaf
as `UTC`. The existing `append_java_compat` and `append_python_compat`
fixtures are unchanged.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No API changes in `include/`. No storage format changes: LTZ is written to
Lance as an Arrow
timestamp with the `UTC` time zone, as in Paimon Java.
### Documentation
<!-- Does this change introduce a new feature -->
Yes. `docs/source/user_guide/data_types.rst` describes how Lance data files
store
`TIMESTAMP WITH LOCAL TIME ZONE`: as an Arrow timestamp with the `UTC` time
zone whatever the
local time zone is, returned in the local time zone on read.
### Generative AI tooling
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
