abstractdog opened a new pull request, #6954: URL: https://github.com/apache/hive/pull/6954
### What changes were proposed in this pull request? In `ParquetFilterPredicateConverter.buildFilterPredicateFromPredicateLeaf`, a leaf whose runtime value (`DynamicValue`) is not available now returns `null`, meaning the leaf filters nothing. Before, it threw an exception that failed the whole conversion. The two ways an unresolved value shows up are both handled: - **`BETWEEN` / `IN`:** `PredicateLeafImpl.getLiteralList()` already catches `NoDynamicValuesException` and returns an empty list. The converter now treats an empty list as "no filter" and no longer passes it to `FilterPredicateLeafBuilder.buildPredicate`, which threw `Not able to build 'between' operation filter with [] which needs two literals`. - **Single-literal operators:** `PredicateLeafImpl.getLiteral()` lets `NoDynamicValuesException` propagate. The converter now catches it and returns `null`. Returning `null` is how the converter already handles a leaf that can't filter, such as a column missing from the file schema or a `CONSTANT` node. ### Why are the changes needed? A single unresolved leaf made `toFilterPredicate` return `null` for the whole search argument. The reader then lost the row-group filtering of the static leaves in the same conjunction, and every occurrence logged an ERROR with a full stack trace. One TPC-DS 10 TB run logged 45,927 of these from the LLAP IO threads reading semijoin-filtered Parquet scans. Dropping only the unresolved leaf is safe. `SearchArgumentImpl.build()` pushes `NOT` down to the leaves and converts the expression to CNF, so the leaf can only sit under an `AND`, under an `OR`, or directly under a `NOT`. Removing it from an `AND` only makes the filter weaker. An `OR` containing it, or a `NOT` over it, still produces no filter. Row groups are never pruned by mistake. ### Does this PR introduce _any_ user-facing change? No. Parquet scans keep the row-group filtering of their other conjuncts, and the `fail to build predicate filter leaf with errors` ERROR lines for unresolved runtime values are gone. ### How was this patch tested? New test: `TestParquetFilterPredicate#testUnresolvedRuntimeValueDropsOnlyItsLeaf` - `a < 5 AND b = <unavailable> AND c BETWEEN <unavailable>` now converts to `lt(a, 5)`. It was `null`. - `a < 5 OR c BETWEEN <unavailable>` still converts to `null`. The unit-test `hive.log` (`ql/target/tmp/log/hive.log`) from each run is attached to [HIVE-30182](https://issues.apache.org/jira/browse/HIVE-30182): | Log | Code tested | Result | |---|---|---| | [pr6882-hive-with-patch.log](https://issues.apache.org/jira/secure/attachment/13086395/pr6882-hive-with-patch.log) | this patch; `Tes`TestParquetRowGroupFilter`, `TestConvertAstToSearchArg` | 47 tests, all pass. The new test logs no ERROR, only the existing DEBUG line`SearchArgumentImpl: Error while retrieving literalList` (lines 2185, 2240). The four `not supported for DATE/DECIMAL` ERRORs come from existing tests and are unrelated. | | [pr6882-hive-without-patch.log](https://issues.apache.org/jira/secure/attachment/13086397/pr6882-hive-without-patch.log) | old conver `expected:<[lt(a, 5)]> but was:<[null]>`. Line 1: `ERROR ... fail to build predicate filter leaf with errors ...NoDynamicValuesException: not available`, from the `b = <unavailable>` leaf. The conversion stops there and the `BETWEEN` leaf is never reached. | | [pr6882-hive-without-patch-between.log](https://issues.apache.org/jira/secure/attachment/13086396/pr6882-hive-without-patch-between.lest with the `BETWEEN` leaf moved before the `b` leaf (local change only, not committed) | Same failure. Line 56: `ERROR ... fail tobuild predicate filter leaf with errors ... Not able to build 'between' operation filter with [] which needs two literals`. This is the semijoin min/max case behind most of the TPC-DS log lines. | To reproduce (JDK 21): ``` mvn -pl ql test -Dtest=TestParquetFilterPredicate,TestParquetRowGroupFilter,TestConvertAstToSearchArg ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
