zhuxiangyi opened a new pull request, #9940:
URL: https://github.com/apache/paimon/pull/9940
### Purpose
**This is a bug fix.** With `variant.inferShreddingSchema=true`, a Variant
object whose keys sort differently by UTF-8 bytes and by UTF-16 code units
fails the whole file write:
```sql
CREATE TABLE T (id INT, v VARIANT) TBLPROPERTIES
('variant.inferShreddingSchema' = 'true');
INSERT INTO T VALUES (1, parse_json('{"￥":100,"😀":"s"}'));
-- java.lang.RuntimeException: Variant object fields must be sorted
alphabetically
-- at
org.apache.paimon.data.variant.InferVariantShreddingSchema.schemaOf(InferVariantShreddingSchema.java:318)
```
`InferVariantShreddingSchema.schemaOf` asserted that the fields of every
sampled object are sorted by `String.compareTo`, i.e. by UTF-16 code units.
Variant writers do not agree on that order: Paimon's own
`GenericVariantBuilder` sorts keys by their UTF-8 bytes
(`BinaryString.compareTo`), Spark's `parse_json` by UTF-16 code units. The two
orders differ whenever a supplementary character (an emoji, U+1F600) meets a
character from U+E000 upwards (fullwidth punctuation and currency signs, CJK
compatibility ideographs, the private use area): UTF-16 puts the emoji's
surrogate pair (`D83D`) first, UTF-8 puts it last (`F0 9F 98 80` after `EF BF
A5`). So an object Paimon itself wrote in byte order was rejected as unsorted;
an object in the other order would have passed the check and then been merged
out of sequence by `mergeRowTypes`, which walks two sorted lists with the same
comparator. The read path already tolerates both orders (`getFieldByKey` tries
both, #9253); inference did not.
The fix stops trusting the writer's order: `schemaOf` sorts each object's
fields by UTF-8 bytes, and `mergeRowTypes` merges with the same comparator, so
inference produces the same schema for either writer. Objects with only ASCII
or BMP keys are unaffected.
### Tests
The new tests reproduce the bug: on `master` they fail with the
`RuntimeException` above, and pass with this change.
-
`InferVariantShreddingSchemaTest#testInferSchemaWithKeysInEitherWriterOrder`:
the same object once in byte order (as Paimon writes it) and once with its
entries swapped into code-unit order (as Spark writes it), each merged with a
second object; both infer the same schema with every key exactly once.
-
`InferVariantShreddingWriteTest#testInferSchemaWithKeysOrderedByUtf8Bytes`:
writes such rows with inference on, checks the physical Parquet schema, the
reconstructed Variant and typed extraction of `$['😀']` / `$['￥']`.
- `VariantTestBase`: inserts the objects through `parse_json` (Spark's
order) and extracts both keys; runs under all four Spark 4.x configurations, so
the inferred-shredding suites also cover Spark evaluating on a Variant Paimon
rebuilt in byte order. Verified on Spark 4.1.2: 132 tests pass.
### API and Format
No.
### Documentation
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
