sunchao opened a new pull request, #7721:
URL: https://github.com/apache/datafusion-comet/pull/7721

   ## Which issue does this PR close?
   
   Part of #6834. This follows up on the cache pruning work in #5543 under 
#5487.
   
   ## Rationale for this change
   
   With Comet's cache serializer, a query can lose rows when a string prefix 
ends inside a multibyte character. For example, `startswith(s, CAST(X'C3' AS 
STRING))` matches `é` because Spark compares the prefix byte `C3` with the 
first byte of `C3 A9`. All the cached strings can be valid UTF-8. [Spark row 
comparison](https://github.com/apache/spark/blob/v4.1.3/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L762-L771)
   
   The cache pruner takes one **character** from each batch bound for this 
prefix. If the lower bound is `é`, that gives `C3 A9`, which sorts after `C3`. 
The scan skips the entire batch before the row filter can evaluate it. [Spark 
cache pruning 
expression](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala#L330-L332)
   
   ## What changes are included in this PR?
   
   Extend the existing guard for collated prefixes in 
`ArrowCachedBatchSerializer.buildFilter`. When a predicate contains 
`StartsWith` with a malformed UTF-8 string literal, leave that predicate to the 
row filter. The validity check runs once when building the batch filter. Other 
conjuncts can still prune batches.
   
   ## How are these changes tested?
   
   Add `startswith(s, 'é') AND startswith(s, CAST(X'C3' AS STRING))` to 
`CometInMemoryCachePruningSuite`. The suite compares the returned row IDs with 
Spark's answer collected before caching. It also checks that the Comet scan 
decodes exactly the four matching rows out of 64, exercising pruning by the 
complete prefix alongside the partial one. It covers native Arrow, Spark 
columnar, and row inputs.
   
   `git diff --check` passed. The regression runs in hosted CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to