dwsmith1983 commented on PR #5615: URL: https://github.com/apache/datafusion-comet/pull/5615#issuecomment-5553189362
Thanks for running it. All of it is on the branch as five commits ending at 5fe0aa8ac; the description now says which stage shapes each piece covers. The fingerprint: `CometExecRDD` carries a `planFingerprint: Long` computed once on the driver with Spark's XXH64 over the same bytes, `PlanKey.hashCode` folds it and `equals` keeps `Arrays.equals`. On the 1000-column plan the cached path goes from 77.6us to 61.3us per task, single thread on JDK 17, with the per-task `PlanKey` construction dropping from 19.3us to nothing and the driver paying 1.76us once per stage. The base plan cache hit is asserted end to end in `PlanDataInjectorShuffleLifecycleSuite` on a map-only stage (four tasks add one base entry, re-executing the DataFrame adds nothing), alongside a fused-shuffle test that fills the shuffle store and leaves the base cache alone. One wrinkle from writing it: a fresh `spark.read.parquet` re-plans with new plan ids, so the hit check re-executes the same DataFrame. The rest, in your order: the Iceberg `commonCache` is gone, nothing bypasses the memo any more; the 3-arg `injectPlanData` overload and the null branch are deleted and the tests go through the two real entry points; `QueryContextInterner` no longer says re-parsed; `releaseAll` clears both stores from both managers' `stop`, with the recreated-context tests asserting both empty; the SPI has `type Prepared <: AnyRef`, both built-ins and the test stub declare it, and the one cast sits at the memo boundary; the memo key is prefixed with the injector class, with a test of two injectors sharing a key; `maxCachedShuffles` is its own constant; `cachedOrCompute` documents the synchronizedMap contract; the proto now ships `optional int32 source_key_hash` and `getKey` rebuilds the string from the source already in the message; `(this eq that)` and the throwaway manager are gone. The Delta contrib PR will need `override type Prepared` in its injector when it rebases onto this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
