sunchao commented on code in PR #5552:
URL: https://github.com/apache/datafusion-comet/pull/5552#discussion_r3887265110
##########
native/core/src/execution/utils.rs:
##########
@@ -19,28 +19,45 @@
use crate::execution::operators::ExecutionError;
use arrow::{
array::ArrayData,
+ datatypes::Field,
+ error::ArrowError,
ffi::{FFI_ArrowArray, FFI_ArrowSchema},
};
+fn ffi_schema_for_field(field: &Field) -> Result<FFI_ArrowSchema, ArrowError> {
+ if field.name().contains('\0') {
+ // ArrowSchema names are NUL-terminated C strings. Spark owns the
logical output name, so
+ // substitute only the exported name while retaining the Field's type
and metadata.
+ let field = field
+ .clone()
+ .with_name(field.name().replace('\0', "\u{fffd}"));
+ FFI_ArrowSchema::try_from(&field)
+ } else {
+ FFI_ArrowSchema::try_from(field)
Review Comment:
[P2] Reconcile union nullability before broadcast coalescing
Exporting the actual Field exposes different top-level nullability across
`CometUnionExec` branches, but the union forwards those batches unchanged to
`Utils.coalesceBroadcastBatches`. Its `VectorSchemaRootAppender` requires
matching nullability and throws `IllegalArgumentException: Vector schema roots
have different schemas` for an ordinary integer payload that is nonnullable in
one branch and nullable in the other.
Reproduced with AQE disabled and a Parquet `t(_1 INT, _2 INT)` containing
`(1,10), (2,20), (3,30)`:
```sql
SELECT /*+ BROADCAST(b) */ p._1, b.v
FROM t p JOIN (
SELECT _1 AS k, 99 AS v FROM t
UNION ALL
SELECT _1 AS k, _2 + 1 AS v FROM t
) b ON p._1 = b.k;
```
On Spark 4.1.3/JDK 17, this fails at this head and returns all six expected
rows at the exact base `98cd8c967`; the Spark-only control also passes. The old
DataType export gave both payloads identical top-level flags. Please reconcile
compatible nullability differences during broadcast coalescing (or leave these
batches uncoalesced), retain complete Field export, and add this regression
case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
