ReemaAlzaid opened a new pull request, #11615: URL: https://github.com/apache/incubator-gluten/pull/11615
## What changes are proposed in this pull request? This PR fixes `input_file_name()` / `input_file_block_*` behavior on Iceberg with Velox and addresses a related JNI crash path. Issue: https://github.com/apache/incubator-gluten/issues/11513 ### Main fixes - **JNI stability fix (Velox native path)** - `cpp/core/jni/JniCommon.h` - Pre-resolve reservation listener JNI methods in `SparkAllocationListener` constructor. - Add null-class guard in `getMethodIdOrError` to avoid unsafe JNI method lookup failures. - **Iceberg split metadata propagation fix** - `cpp/velox/compute/WholeStageResultIterator.cc` - Pass metadata map to `HiveIcebergSplit` (instead of empty metadata), so file metadata is preserved. - **Iceberg metadata wiring** - `gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala` - `gluten-iceberg/src/main/java/org/apache/gluten/substrait/rel/IcebergLocalFilesBuilder.java` - `gluten-iceberg/src/main/java/org/apache/gluten/substrait/rel/IcebergLocalFilesNode.java` - Generate and propagate metadata columns for: - `input_file_name` - `input_file_block_start` - `input_file_block_length` - **Scan transformer + rule generalization** - `gluten-substrait/src/main/scala/org/apache/gluten/execution/BatchScanExecTransformer.scala` - `gluten-iceberg/src/main/scala/org/apache/gluten/execution/IcebergScanTransformer.scala` - `gluten-kafka/src/main/scala/org/apache/gluten/execution/MicroBatchScanExecTransformer.scala` - `gluten-paimon/src-paimon/main/scala/org/apache/gluten/execution/PaimonScanTransformer.scala` - Add/implement `withOutput(...)` for `BatchScanExecTransformerBase` family. - Update `PushDownInputFileExpression` to work against `BatchScanExecTransformerBase` and deduplicate `rewriteExpr`. ## How was this patch tested? - Build/compile validation: - `mvn -o -rf :gluten-substrait -DskipTests compile` - `mvn -o -Piceberg,paimon,kafka -pl gluten-substrait,gluten-iceberg,gluten-paimon,gluten-kafka -DskipTests compile` - Repro scenario: - Ran Iceberg `input_file_name()` reproduction with Gluten Velox bundle. - Verified crash path moved from native segfault to stable execution. - Verified metadata propagation fix removes empty/null file metadata behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
