yihua opened a new pull request, #901:
URL: https://github.com/apache/hudi-rs/pull/901
## Description
closes #899
`SchemableFilter::apply_comparison` evaluated `IN` and `NOT IN` as one
equality kernel pass per value, so the cost of a key-list filter grew with the
list. For string columns (`Utf8`, `LargeUtf8`, `Utf8View`) the values now go
into a hash set once and each row is probed once; null rows stay null, as the
kernels left them. Other column types keep the comparison chain.
Reading 100 record keys out of a 10M-row table through `Table::read` with an
`IN` filter took about twice as long as reading one key; with this change the
two cost the same.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`test_in_on_string_columns_matches_the_comparison_chain` checks `IN` and
`NOT IN` over the three string types with a null row against the expected
masks, and that a non-string column still takes the chain. The existing filter
and read suites pass unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
