raghuvanshraj commented on PR #26282: URL: https://github.com/apache/datafusion/pull/26282#issuecomment-6851529961
I built this branch as is and tried it with a `variant_get`-shaped UDF: `placement()` returning `MoveTowardsLeafNodes`, and `required_fields` listing `metadata`, `value`, `typed_value.<key>.value` and `typed_value.<key>.typed_value`. It is a thin wrapper over the arrow-rs `variant_get` kernel, not the datafusion-variant one, which has neither method yet. On a 100k-row, 265 MB file shredded on 20 keys, a single-key lookup went from 126.4 MB read to 13.2 KB, and the same held under `avg`, `group by`, `order by ... limit` and a filter, since `ExtractLeafExpressions` already pulls the call into the scan. Row counts matched with and without the declaration. @cetra3 @adriangb are you planning to rebase and carve out part 1 of the split, or would it help if we picked it up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
