zhuxiangyi opened a new pull request, #9942:
URL: https://github.com/apache/paimon/pull/9942
### Purpose
**This is a bug fix.** A table with a `VARIANT` column can be created with a
per-level or changelog file format that cannot store it; the failure only shows
up on the first write, deep inside the ORC or Avro writer:
```sql
CREATE TABLE T (id INT, v VARIANT) TBLPROPERTIES (
'primary-key' = 'id', 'file.format' = 'parquet', 'file.format.per.level' =
'0:orc'); -- accepted
INSERT INTO T VALUES (1, parse_json('{"a":1}'));
-- java.lang.UnsupportedOperationException: Unsupported type: VARIANT
CREATE TABLE T2 (id INT, v VARIANT) TBLPROPERTIES (
'primary-key' = 'id', 'changelog-producer' = 'input',
'changelog-file.format' = 'avro'); -- accepted
INSERT INTO T2 VALUES (1, parse_json('{"a":1}'));
-- java.lang.UnsupportedOperationException: Unsupported to derive Schema for
type: VARIANT
```
The docs state that Variant data files must use Parquet, but
`SchemaValidation` only enforced it for the main `file.format`, and only as a
side effect of `fileFormat.validateDataFields()` running the format's own type
converter. `file.format.per.level` was inspected for Avro timestamp precision
only, and `changelog-file.format` not at all. Even the main-format rejection
was the converter's bare `Unsupported type: VARIANT`, naming neither the column
nor the option. Geometry and geography columns, which have the same
Parquet-only restriction, already get all three checks with an actionable
message (`validateGeospatialTypes`).
This PR adds the equivalent `validateVariantTypes`: the main format, every
per-level override and the changelog format must be parquet, checked at DDL
time and reported with the offending columns (including a variant nested inside
a struct, array or map) and option:
```
Variant columns [v] require 'file.format'='parquet', but was 'orc'.
Variant columns [v] require parquet at every level, but
'file.format.per.level' contains '0:orc'.
Variant columns [v] require 'changelog-file.format' to be parquet, but was
'avro'.
```
It runs before `validateDataFields`, so the main-format case now reports
this message instead of the converter's.
### Tests
`SchemaValidationTest#testVariantTypeValidation` covers a top-level variant
and one nested in a struct: an all-parquet table passes; `file.format=orc`,
`file.format.per.level=0:orc` and `changelog-file.format=avro` are each
rejected with the message above. On `master` it fails with `Expecting throwable
message: "Unsupported type: VARIANT" to contain: "Variant columns [v, nested]
require 'file.format'='parquet', but was 'orc'"` (the per-level and changelog
cases would not throw at all). `SchemaValidationTest`, `SchemaManagerTest` and
`SchemaEvolutionTest` pass (155 tests).
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
