SteNicholas opened a new issue, #476: URL: https://github.com/apache/paimon-cpp/issues/476
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation Paimon Java accepts `TIMESTAMP_LTZ` (`TIMESTAMP WITH LOCAL TIME ZONE`) columns in Lance data files since apache/paimon#10558. Paimon C++ still rejects them: - `SchemaValidation::ValidateLanceDataField()` returns `Lance file format does not support type LOCAL_ZONED_TIMESTAMP` for every Arrow timestamp with a non-empty time zone (`src/paimon/core/schema/schema_validation.cpp:1009`). The check runs for `file.format`, `file-format-per-level` and `changelog-file-format`, so C++ cannot create a Lance table with an LTZ column that Java can create and write. Reading such a table written by Java is not covered by any test either. Simply removing the check would not be enough, because the two sides put different time zones on the Arrow type: - Java always writes LTZ as Arrow `Timestamp(unit, "UTC")` (`ArrowFieldTypeConversion.visit(LocalZonedTimestampType)`), and its validator relies on that. Lance 0.39 encodes the type as `timestamp:<unit>:<tz>` and cannot parse a zone id that contains `:`. It fails with `Unsupported timestamp type: timestamp:us:GMT-10:00` (`rust/lance-core/src/datatypes.rs`), as the comment in the Java change explains. - C++ builds the LTZ Arrow type with `DateTimeUtils::GetLocalTimezoneName()` (`src/paimon/common/utils/date_time_utils.h:188`), which returns the `TZ` environment variable as-is or the system zone name. Passed through unchanged, this would write a host-dependent zone id that differs from Java's, and writes would fail whenever that id contains `:`. ### Solution 1. **Validation.** Accept timestamps with a time zone in `ValidateLanceDataField()`, including when nested in ARRAY and ROW, matching Java's `LanceRowTypeVisitor`. 2. **Write.** Normalize the zone of every timestamp-with-zone field (top level and nested) to `"UTC"` before the batch reaches Lance. `LanceFormatWriter::AddBatch()` imports each batch with `schema_` (`src/paimon/format/lance/lance_format_writer.cpp:113`). Because the values are UTC epoch offsets, importing with a copy of `schema_` whose timestamp zones are replaced by `"UTC"` changes only type metadata, not data. The file then stores the same type Java writes. 3. **Read.** Lance returns `timestamp[<unit>, UTC]`. `LanceFileBatchReader::AlignBatch()` already passes batches through `NestedProjectionUtils::AlignArrayToReadType()`, which should cast the leaf to the read type's local zone (a metadata-only cast). This needs a test rather than an assumption. `GetFileSchema()` will also report `UTC`; check that the field mapping and schema evolution code that compares file types with table types accepts the zone difference. 4. **Stats.** `LanceStatsExtractor::CreateEmptyStats()` already handles `TIMESTAMP` regardless of the zone, so no change is expected there. ### Anything else? Suggested validation: - Update `SchemaValidationTest.TestLanceDataTypes` (`src/paimon/core/schema/schema_validation_test.cpp:480`), which currently expects `LOCAL_ZONED_TIMESTAMP` to be rejected. - Add a format-level round trip in `src/paimon/format/lance/lance_file_format_test.cpp` for LTZ at precisions 0/3/6/9 and nested in ARRAY. Run it under a non-UTC zone and under a `TZ` value that contains `:`, and assert that the written Lance schema uses `UTC`. - Add a Java compatibility fixture with LTZ columns, written by Paimon Java at or after apache/paimon#10558, and read it from C++. Add it as a new fixture next to `test/test_data/lance/append_java_compat.db` instead of rewriting the existing one. The on-disk representation matches what Java writes, so this needs no storage format or protocol change. `TIME` support for Lance is tracked separately. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
