kane0409 opened a new issue, #9440: URL: https://github.com/apache/paimon/issues/9440
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version Paimon 1.4, using the Hive 3.1 connector. ### Compute Engine - Hive 3.1.2 - Tez - Java 8 ### Minimal reproduce step A Hive SQL query joining multiple Paimon tables occasionally fails while Tez is initializing input splits. The failure happens before any data is read: ```text ROOT_INPUT_INIT_FAILURE Vertex Input: nws_fleet_van_proof_parcel_detail_p initializer failed java.lang.IndexOutOfBoundsException: Index: 0, Size: 0 at java.util.ArrayList.get(ArrayList.java:435) at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convertLeaf( SearchArgumentToPredicateConverter.java:163) at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convertTree( SearchArgumentToPredicateConverter.java:124) at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convert( SearchArgumentToPredicateConverter.java:91) at org.apache.paimon.hive.utils.HiveUtils.createPredicate(HiveUtils.java:96) at org.apache.paimon.hive.utils.HiveSplitGenerator.generateSplits( HiveSplitGenerator.java:68) at org.apache.paimon.hive.mapred.PaimonInputFormat.getSplits( PaimonInputFormat.java:45) ``` The failing code is in the BETWEEN branch: ```java List<Object> literalList = leaf.getLiteralList(); return builder.between( idx, toLiteral(columnType, literalList.get(0)), toLiteral(columnType, literalList.get(1))); ``` Hive Tez dynamic semijoin reduction can inject a runtime predicate similar to: ```text proof_id BETWEEN DynamicValue(min) AND DynamicValue(max) ``` When Paimon generates splits, these dynamic values may not be available yet. In this case, PredicateLeaf.getLiteralList() returns an empty list, but Paimon accesses elements 0 and 1 without checking the list size. The issue is intermittent because whether Hive generates the runtime filter depends on the execution plan, table statistics, data size, and runtime timing. ### Minimal reproduce steps 1. Enable the Tez dynamic semijoin optimization: ```sql SET hive.execution.engine=tez; SET hive.tez.dynamic.semijoin.reduction=true; ``` 2. Run a join where Hive pushes a dynamic min/max predicate into a Paimon table scan: ```sql SELECT COUNT(*) FROM paimon_db.proof_table p JOIN paimon_db.proof_detail_table d ON p.id = d.proof_id WHERE p.p_date >= date_sub(current_date, 7) AND d.p_date >= date_sub(current_date, 8) AND d.relation_category IN (1, 3); ``` 3. When the dynamic values are unavailable during split initialization, the Paimon input initializer fails with IndexOutOfBoundsException. Disabling dynamic semijoin reduction prevents the failure: ```sql SET hive.tez.dynamic.semijoin.reduction=false; ``` ### What doesn't meet your expectations? ### Expected behavior An unavailable or incomplete dynamic predicate should not cause the query to fail. Because predicate pushdown is a best-effort optimization, Paimon should skip this predicate and allow Hive to apply the normal join/filter logic. For example: ```java List<Object> literalList = leaf.getLiteralList(); if (literalList == null || literalList.size() < 2) { throw new UnsupportedOperationException( "BETWEEN literals are not available"); } ``` convert() already catches UnsupportedOperationException, so the unavailable predicate can be ignored safely. ### Actual behavior Paimon directly calls literalList.get(0) and literalList.get(1), causing the entire Tez DAG to fail during root input initialization. ### Anything else? The Hive project documents that dynamic BETWEEN values may be unavailable while predicate pushdown is being evaluated: - https://issues.apache.org/jira/browse/HIVE-18827 - https://issues.apache.org/jira/browse/HIVE-23295 Paimon source: - https://github.com/apache/paimon/blob/release-1.4/paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/SearchArgumentToPredicateConverter.java#L159-L164 The current workaround is: ```sql SET hive.tez.dynamic.semijoin.reduction=false; ``` However, this disables a useful Hive runtime optimization and may increase the amount of data read and shuffled. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
