yunfengzhou-hub opened a new pull request, #8696: URL: https://github.com/apache/paimon/pull/8696
### Purpose When a SQL query contains a large `IN` clause (e.g. `IN (v1, v2, ..., vN)` with thousands of values), the expression planner naturally expands it into a deeply-nested OR expression tree. Paimon's `PredicateConverter` and `PredicateBuilder` previously processed this tree in ways that could trigger `StackOverflowError`: 1. **`PredicateConverter`**: The recursive `children.get(0).accept(this)` / `children.get(1).accept(this)` calls could overflow the JVM stack when the OR tree depth exceeded the stack limit (~1000+ levels). 2. **`PredicateBuilder.or()` / `and()`**: The previous left-fold `reduce` produced a left-leaning tree of depth N (where N is the number of predicates), which could cause `StackOverflowError` during subsequent recursive traversal (e.g. in `applyFilters`, predicate serialization, etc.). **Fix:** - **`PredicateConverter`**: Added an iterative `flattenAndConvert()` method that uses a stack-based traversal to flatten nested AND/OR expression trees into a flat list of predicates, eliminating recursive `accept()` calls. - **`PredicateBuilder`**: Replaced the left-fold `reduce` in `or()` and `and()` with a new `buildBinaryTree()` method that recursively splits the predicate list at the midpoint, producing a balanced binary tree with depth O(log N) instead of O(N). For example, 10 000 predicates produce a tree of depth ~14 instead of ~10 000. ### Tests - `PredicateBuilderTest` — verifies binary tree structure (balanced split, depth O(log N), correct children count) - `PredicateConverterTest` — verifies iterative flattening of nested OR/AND trees - `FlinkTableSourceTest` — verifies end-to-end `applyFilters` with 10 000-value nested OR tree (nested, right-fold, balanced, and flat multi-children shapes) - `PredicateTest` — updated `testPredicateToString` expectations for balanced tree format - `PredicateJsonSerdeTest` — updated JSON serialization expectations for `in()`, `notIn()`, and complex `and()` to balanced tree format - `OrcFilterConverterTest` — updated OR/AND filter expectations for balanced tree format - `ParquetFiltersTest` — updated OR/AND filter expectations for balanced tree format -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
