andygrove commented on PR #5543: URL: https://github.com/apache/datafusion-comet/pull/5543#issuecomment-5800556954
Thanks @sunchao, all three were real and are fixed in a710d5fd7. The collated prefix pruning: `buildFilter` now leaves out any predicate holding a `StartsWith` over a string that isn't the default collation, and keeps equality and range pruning on those columns, since those compare with the collation itself. The new test caches your U+0130 values under UTF8_LCASE and gets 0 of 5 rows back without the fix. The pinned allocation is fixed without adding a copy. `Projection.load` reads each window's uncompressed-length prefix and puts the ones Arrow stored raw in a second allocation, so decompressing frees the one holding the compressed windows. Empty windows take Arrow's shared empty buffer instead of a slice of either. The new test builds your Boolean-and-string shape at 4,096 rows and checks what each bitmap is backed by: before the fix, a 512-byte bitmap kept a 1 MiB allocation alive. On the value tests, both now collect their reference rows before the relation is cached. The codec round-trip test had the same flaw, since `checkSparkAnswer` reads the one cached payload on both sides, so it does the same now. With the writer planted to store 777777 as the first `id` of each batch, all three fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
