voonhous opened a new pull request, #20759:
URL: https://github.com/apache/hudi/pull/20759

   ### Describe the issue this Pull Request addresses
   
   Closes #20567
   
   A bloom filter expression index over a DATE result never matched: the writer 
adds `toString` of the external value (`2023-09-22`), while the reader probed 
with the literal's epoch days (`19622`). #20540 stopped the wrong results by 
not probing such terms, so a DATE filter skipped nothing.
   
   ### Summary and Changelog
   
   A bloom filter expression index over a DATE result prunes files again.
   
   - `ExpressionIndexSupport`: a DATE literal now gives the writer's strings as 
probe keys. Both the `java.sql.Date` and the `LocalDate` form are probed, since 
the writer prints one or the other depending on 
`spark.sql.datetime.java8API.enabled` at write time, and they differ for edge 
years. A file passes if it may hold any key, so the extra key only keeps files.
   - TIMESTAMP, floating point and decimal terms still give no keys (reasons in 
the scaladoc). The writer's key format is unchanged, so existing indexes keep 
working.
   - `TestExpressionIndex`: the bloom case that pinned no pruning now expects 1 
file instead of 2, plus an equality case; the index is written with the java8 
API off and on.
   
   ### Impact
   
   DATE equality and IN filters can use a bloom filter expression index.
   
   ### Risk Level
   
   low. Only DATE terms gain probe keys; the test fails without the change.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to