zhuxiangyi opened a new pull request, #9939:
URL: https://github.com/apache/paimon/pull/9939
### Purpose
**This is a bug fix.** Extracting a timestamp from a Variant converts it
with the JVM default time zone instead of the zone the query asked for.
A variant extraction carries a `timeZoneId` (`VariantCastArgs.zoneId()`,
filled from the Spark session zone when `variant_get` is pushed into the scan),
but `VariantGet` only used it for `toJson`. Every scalar cast went through the
generic `CastExecutors` rules, and `TimestampToStringCastRule` /
`TimestampToTimestampCastRule` / `TimestampToDateCastRule` /
`StringToTimestampCastRule` all use `TimeZone.getDefault()`. So on Spark 4.1
with `spark.sql.variant.pushVariantIntoScan=true` the result depends on the
executor JVM's zone and disagrees with Spark's own evaluation:
```sql
SET spark.sql.session.timeZone = UTC; -- JVM default: America/Los_Angeles
CREATE TABLE T (id INT, v VARIANT);
INSERT INTO T VALUES (1, to_variant_object(named_struct('ts',
timestamp'2023-11-14 22:13:20', 'd', date'2024-10-04')));
SELECT variant_get(v, '$.ts', 'string'), try_variant_get(v, '$.ts',
'timestamp_ntz'), try_variant_get(v, '$.d', 'timestamp') FROM T;
-- pushVariantIntoScan=true : 2023-11-14 14:13:20.000000 | 2023-11-14
14:13:20 | 2024-10-04 07:00:00 <- wrong
-- pushVariantIntoScan=false: 2023-11-14 22:13:20 | 2023-11-14
22:13:20 | 2024-10-04 00:00:00 <- Spark
```
A `date` extraction can land on the previous or next day (`22:13 UTC` read
from a JVM in `Asia/Shanghai` is `06:13` the next day). `'string'` is the one
target type a strict `variant_get` is pushed down for, so the most common
extraction is affected. The shredded `typed_value` reader
(`BaseVariantReader.ScalarReader`) has the same problem through the same rules;
plain and shredded files agree with each other, both wrongly. `toJson` was
already correct, which makes the contrast easy to see in the tests.
The fix handles the zone-dependent casts in the scalar cast helper both legs
share, with the zone from `VariantCastArgs`: timestamp with local time zone to
string / to timestamp without time zone / to date; timestamp without time zone
and date to timestamp with local time zone; and string to timestamp with local
time zone. Every other pair still goes through the cast rules. A timestamp now
renders the way Spark's cast does, without trailing fraction zeros
(`22:13:20.5`, not `22:13:20.500000`).
One existing expectation in `PaimonShreddingUtilsTest#testAssembleAllTypes`
had encoded the JVM-default behaviour (formatting with `TimeZone.getDefault()`
while the cast args asked for UTC); it now expects UTC.
### Tests
The new tests reproduce the bug: on `master` they fail with `expected:
2023-11-15 06:13:20.5 but was: 2023-11-14 22:13:20.500000`, and pass with this
change. They flip `TimeZone.setDefault` between UTC and Los Angeles inside the
test and assert the same result each time, so a JVM that happens to run in the
requested zone cannot hide a regression.
- `GenericVariantTest#testVariantGetTemporalCastsUseRequestedZone`: every
zone-dependent pair (LTZ→string/ntz/date, ntz→LTZ, date→LTZ, string→LTZ) plus
target-precision truncation, under two JVM defaults.
- `PaimonShreddingUtilsTest#testShreddedTimestampCastsUseRequestedZone`: the
same extractions through a shredded and an unshredded `VariantSchema`.
- `VariantShreddingReadTest#testReadTimestampInRequestedZone`: parameterised
over a plain Parquet file and one shredded with `ts TIMESTAMP(6) WITH LOCAL
TIME ZONE`, read with `Asia/Shanghai` under two JVM defaults.
- `VariantTestBase`: `variant_get` / `try_variant_get` on a timestamp,
timestamp_ntz and date with the session zone set to `Asia/Shanghai` while the
test JVM runs in `America/Los_Angeles`; runs under all four Spark 4.x
configurations. Verified on Spark 4.1.2: 132 tests pass.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
