ReemaAlzaid commented on PR #12471: URL: https://github.com/apache/gluten/pull/12471#issuecomment-5361302246
> > RowVectorStream::next() was rebuilding device batches into RowVectors > > shouldn't happen if `CudfToVelox` is the last operator to the GPU pipeline. This seems more like the bug from the planner phase. Can you please look into it? I looked into it and as u said here a GPU pipeline always ends with `CudfToVelox`, so device batches can't leak from its output. The issue is the broadcast build side, which bypasses that path entirely In q16, the `NOT IN` becomes a null aware anti join and Spark forces it to broadcast even with `autoBroadcastJoinThreshold=-1`. The consuming stage is planned for CPU, but `ColumnarBuildSideRelation#deserialized` creates its Runtime from the session config, so it picks `VeloxGpuColumnarBatchSerializer`, which uploads the broadcast batches to the device into a stage that expects host batches. It fails deterministically on the first batch of every task. The fix (#12838): broadcast bytes are host resident by nature the only question is where the single host to device upload happens. The deserializer can't know whether the consuming stage is GPU or CPU, so it should not be the one deciding; it now always hands over host batches (via a COLUMNAR_CUDF_ENABLED=false override scoped to its own Runtime the session config is untouched). The residency decision then happens at the consumer, the only place that knows the stage's contract: a GPU stage uploads through `CudfVectorStream` as its first operator, a CPU stage uses the batches directly. GPU execution is unchanged same operators, same single upload, just performed where it can't go wrong. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
