LuciferYang opened a new issue, #9615: URL: https://github.com/apache/paimon/issues/9615
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Spark 4.1. The variant extraction push-down that produces a struct read target is bound in `paimon-spark-4.1` (`PaimonSupportsPushDownVariantExtractions`), and the failure itself is in paimon-common. ### Minimal reproduce step Read a struct out of a variant column where the file's shredding schema does not cover every field of that struct. Shredding is inferred per data file (`variant.shredding.inferenceMode` defaults to `PER_FILE`), so which fields are shredded depends on the data in each file: ```sql SELECT try_variant_get(v, '$', 'struct<a int, b string>') FROM t; ``` If the file shredded `a` but left `b` in the untyped value, planning the read throws a `NullPointerException`. At the reader level: ```java RowType shreddedType = RowType.of(new DataType[] {DataTypes.INT()}, new String[] {"a"}); VariantSchema schema = buildVariantSchema(variantShreddingSchema(shreddedType)); RowType targetType = RowType.of( new DataType[] {DataTypes.INT(), DataTypes.STRING()}, new String[] {"a", "b"}); // NullPointerException BaseVariantReader.create(schema, targetType, new VariantCastArgs(true, ZoneOffset.UTC), false); ``` `BaseVariantReader.RowReader`'s constructor fills its field-index array by unboxing a map lookup straight into an `int`: ```java fieldInputIndices[i] = schema.objectSchemaMap != null ? schema.objectSchemaMap.get(targetFields.get(i).name()) : -1; ``` `objectSchemaMap` only holds the fields the file actually shredded, so the lookup returns null for `b` and unboxing throws. The path into it is `VariantShreddingReadPlanFactory` (line 331) -> `PaimonShreddingUtils.getFieldsToExtract` -> `buildFieldsToExtract` -> `BaseVariantReader.create`, all during read planning, so the whole scan fails rather than one row. ### What doesn't meet your expectations? A struct field that is not shredded should be read from the untyped value, which is what the rest of the class is already built for. The field's own comment says so: ```java // For each field in `targetType`, store the index of the field with the same name in object // `typed_value`, or -1 if it doesn't exist in object `typed_value`. private final int[] fieldInputIndices; ``` and `readFromTyped` has the branch that does it: ```java } else if (unshreddedObject != null) { ... GenericVariant unshreddedField = unshreddedObject.getFieldByKey(fieldName); ``` Because the constructor could never produce -1, that branch and the `needUnshreddedObject` flag that guards it were both unreachable. The same convention is stated in `VariantSchema` ("if a given field is not in the schema, its value must be set to -1 to indicate that it is invalid") and honoured by the other lookups in the package, which take the result as an `Integer` and null-check it. ### Anything else? Nothing else in that directory unboxes a `Map.get` result into a primitive, so this looks like a single site rather than a pattern. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
