QCLyu commented on issue #11605: URL: https://github.com/apache/gluten/issues/11605#issuecomment-5359474090
Hi @zhztheplayer Below is the result of PoC. I don’t think we should revive the [#11769](https://github.com/apache/gluten/pull/11769)-style stats-block work as the default next step without a target workload that looks like the win cases above. Prefer to pause this path (or keep the ticket open only for a follow-up if someone has a production query with high exact skip). Open to running TPC-DS Q94 with the same logging if you want that confirmation before closing. Please let me know if you need to discuss anything. Scope. Treat [#11657](https://github.com/apache/gluten/pull/11657) (ValueStream DF after deserialize) as baseline. Estimate the incremental win from skipping entire shuffle blocks before decompress/deserialize using join-key min/max + the pushed HashProbe filter (Filter::testInt64Range). No shuffle format change; temporary env-gated writer/filter logging only. What we measured 1. Selective BHJ (synthetic): ordered contiguous keys, scattered build keys + randomized probe, and sparse build keys. 2. Logged per-block int64 min/max and filter kind/value set. 3. Offline: coarse (block range vs filter global [min,max]) and exact (Velox-like testInt64Range with the value set — what a real reader would call). <img width="782" height="206" alt="Image" src="https://github.com/user-attachments/assets/bfa005a3-e26a-42cd-8d88-a6ebac0b6224"; /> Takeaway. Payoff of writer min/max + block skip is highly filter- and layout-dependent: * Strong when build keys are localized (range) or sparse enough that testInt64Range rejects most blocks. * Weak/none for common selective BHJs where build keys are dense across the probe domain (scattered Bitmask → 0%). * Orthogonal to #11657, which already filters after deserialize. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
