thswlsqls opened a new issue, #10875: URL: https://github.com/apache/paimon/issues/10875
**Search before asking** - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. **Paimon version** master @ 5c82868d50 / 2.2-SNAPSHOT **Compute Engine** Spark 4.1 (variant pushdown); the defect itself is in paimon-common. **Minimal reproduce step** On Spark 4.1 with default settings (`spark.sql.variant.pushVariantIntoScan` defaults to true), run `SELECT try_variant_get(v, '$.m', 'map<string,int>')` on a Paimon table whose `v` holds `{"m":{"a":1}}`. It returns NULL; with `spark.sql.variant.pushVariantIntoScan=false` it returns `{a -> 1}`. Verified by code trace and paimon-common unit tests, not by an end-to-end Spark run. **What doesn't meet your expectations?** A map with a STRING key should be extracted regardless of the key's nullability. **Anything else?** `SparkTypeUtils` always converts map keys to NOT NULL, so the pushed-down target is `MAP<STRING NOT NULL, INT>`. `VariantGet.cast` and `BaseVariantReader.create` check the key with `equals(DataTypes.STRING())`, which compares nullability, and treat it as an invalid cast. The checks came from #6652, which ported Spark's `MapType(_: StringType, _, _)` pattern. Only `try_variant_get`/`try_cast` with `MAP<STRING, ...>` targets (also nested in arrays) are affected; `variant_get` with a map target is not pushed down. **Are you willing to submit a PR?** - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
