malinjawi commented on issue #12504:
URL: https://github.com/apache/gluten/issues/12504#issuecomment-5035220409
#12592 fixes the segfault, but the benchmark still cannot replay a BHJ: the
build side input iterator is `Iterator.empty` on the offload path
(`VeloxSerializedBroadcastRDD.scala:47`,
`VeloxBroadcastBuildSideRDD.scala:50`), so the dumped `data_*.parquet` for that
index has zero rows.
Proposed approach, all native — no new JNI, no Scala changes:
1. At `VeloxRuntime::parsePlan` time the table is already in
`HashTableCache`: `getIterators` populates it
(`GlutenWholeStageColumnarRDD.scala:77`) before `genFirstStageIterator` reaches
the native kernel.
2. So `VeloxWholeStageDumper` can walk the parsed plan for
`JoinRel.hashtableid`, look each up in `HashTableCache::instance()`, serialize
with the existing `HashTableSerializer`, and write
`hashtable_{stageId}_{partitionId}_{vId}_{id}.bin` next to the plan.
`ignoreNullKeys` falls out of the `HashTable<true>`/`<false>` dynamic_cast;
`joinHasNullKeys` is `entry->hasNullKeys`.
3. `GenericBenchmark` gains `--hash_table`, calls the existing
`gluten::deserializeHashTable(...)`, and registers via
`HashTableCache::instance()->add(...)` — mirroring `deserializeHashTableDirect`.
4. `getJoin()` consults `HashTableCache` when there is no JVM, so the
converter sets `useHashTableCache=true` and the probe runs against the real
table.
`HashTableCache::add` sets `builderTaskId = "external_gluten"` and
`buildComplete = true`, so `HashBuild::setupCachedHashTable` short-circuits to
`noMoreInput()` and never reads the build side — the empty parquet stays
harmless. An externally added entry also skips the `queryCtx` release callback
in `HashTableCache::get`, so it survives across benchmark iterations.
Happy to implement if this looks reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
