viirya commented on PR #58050: URL: https://github.com/apache/spark/pull/58050#issuecomment-5338750843
Good question. I looked into where that ~3% comes from: it's the cost of Parquet evaluating the pushed predicate against each row group's statistics, not extra data being read -- the read schema and the number of row groups read are identical with the flag on and off (no residual columns get pulled in), and record-level filtering is off under the vectorized reader, so there's no per-row cost. It's the same kind of fixed cost any pushed-but-non-pruning filter has; a plain (non-variant) column shows a comparable skip-none overhead in my measurements. Our predicate is a bit heavier than a plain one -- `or(leaf, isNotNull(residual)...)` is a few disjuncts across a few columns rather than a single comparison -- but those residual guards are what make the skip sound (they're the fix for the original data-loss hole), so there isn't a part of it I can drop without giving that up. On the default: I'm inclined to keep it on, since the cost only lands on queries that both have a shredded Variant field and a literal predicate on it while the layout doesn't allow skipping, and it's small and within run-to-run noise. But I don't feel strongly -- if you or @dongjoon-hyun would rather ship it off and let sorted-layout workloads opt in, I'm fine flipping it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
