deniskuzZ opened a new pull request, #6890:
URL: https://github.com/apache/hive/pull/6890
### What changes were proposed in this pull request?
Hive's variant decoder reads primitive types 17–19 of the variant spec:
- `VariantUtil`: `TIME` (17, microseconds since midnight), `TIMESTAMP_NANOS`
(18, nanoseconds since the epoch,
UTC-adjusted) and `TIMESTAMP_NTZ_NANOS` (19) in `getType`, `valueSize` and
`getLong`.
- `Variant.toJson` (query results, `to_json`) renders them like the existing
types:
- TIME: `"13:45:30.123456"`;
- TIMESTAMP_NANOS: `"2024-01-15 13:45:30.123456789+00:00"`, in the session
time zone, like TIMESTAMP;
- TIMESTAMP_NTZ_NANOS: `"2024-01-15 13:45:30.123456789"`, like
TIMESTAMP_NTZ.
- `variant_get(v, path, 'string')`: TIME as `13:45:30.123456`; the
nanosecond timestamps as instants, like the
microsecond ones (`2024-01-15T13:45:30.123456789Z`).
### Why are the changes needed?
Iceberg's variant API writes these types (`Variants.ofTime`,
`ofTimestamptzNanos`, `ofTimestampntzNanos`), and Hive
could not read them:
- `to_json` and `variant_get` failed with
`UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT{id=17}`;
- `VariantUtil.valueSize` threw for them, so an object or array that
contains one could not be read either;
- HS2 results showed the raw `{"metadata":…,"value":…}` form instead of JSON.
### Does this PR introduce _any_ user-facing change?
Yes. Variant values of types 17–19 render as shown above instead of failing.
Typed conversions of these types
(`variant_get(v, path, 'timestamp')` and the like) are HIVE-30172.
### How was this patch tested?
- New unit tests:
- `TestVariantTimeTypes`: decoding, JSON rendering in two time zones,
values inside arrays and objects,
pre-epoch values, the HS2 payload through `SerDeUtils`, and malformed
values;
- `TestGenericUDFVariantGet`: `variant_get(v, path, 'string')` for the
three types;
- `TestIcebergVariantObjectInspector`: values written with Iceberg's
`Variants` API read through Hive.
- The new tests fail without the change
(`UNKNOWN_PRIMITIVE_TYPE_IN_VARIANT{id=17}`), and each of the eight new
decoding and rendering branches, when reverted, fails at least one of them.
- The existing variant tests in serde, ql and iceberg-handler pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
