hubgeter opened a new pull request, #70674: URL: https://github.com/apache/doris/pull/70674
### Background Iceberg uses pushed predicates to prune manifests and data files before Doris evaluates rows. The FE converts scan predicates to neutral connector expressions, whose current representation unwraps CAST expressions. Predicates kept in Doris are evaluated after scanning the selected files. ### What problem does this PR solve? Issue Number: None Related PR: #69551 (added the original disabled reproduction), #70511 (related column-to-string CAST protection) PR Type: Bugfix Problem Summary: A DECIMAL(38,0) value of `9007199254740993` matches `CAST(value AS DOUBLE) = CAST(9007199254740992 AS DOUBLE)`, but removing the column CAST lets Iceberg compare the original decimal value and prune a matching file. Declare CAST predicate pushdown unsupported for Iceberg so the existing scan gate preserves these predicates in Doris before CAST information is lost. Most of this change moves the disabled downstream reproduction to engine tests covering planned expressions, actual Iceberg metadata, and row/file retention; independent plain conjuncts remain pushable. ### Details - **Frequency:** Data dependent: a remaining CAST changes comparison semantics, as adjacent high-precision decimals can become equal after conversion to DOUBLE. - **Root cause:** Iceberg opts into CAST predicate pushdown while the neutral converter unwraps the CAST. Source-side file pruning then uses a stronger predicate, and later Doris filtering cannot recover an unscanned file. - **Why existing tests missed it:** The original precision-loss reproduction in #69551 was disabled. It also started after CAST information had already disappeared; the replacement tests exercise the engine's filter gate with real planned column CASTs and actual Iceberg metadata. - **Fix:** Return `false` from Iceberg's existing CAST pushdown capability. The existing gate keeps the entire CAST-bearing conjunct local, preserves independent AND pushdown and complete OR/NOT semantics, and suppresses source LIMIT when local predicates remain. Update the SPI implementation notes and coverage inventory. - **Other paths and branches:** The capability API is unchanged. #70511 protects column-to-string CASTs and does not cover DECIMAL-to-DOUBLE comparisons. No release-branch backport is included; live SQL and native BE execution remain unverified. - **Behavior change:** Iceberg scans may read more files because all remaining CAST-bearing predicates, including otherwise safe ones, stay in Doris. CASTs already removed by the optimizer and independent predicates without CAST remain eligible for pushdown. ### Testing Validated locally on master base `e87bc301743969fdc123b6117c8077771ffc84f7`, with JDK 17 and Thrift 0.24.0: - Before the fix, explicitly running the original disabled SDK reproduction failed both matching-row and matching-file retention assertions. The new engine suite ran 8 cases with 7 failures and 1 pass. - After the fix, 188 selected tests passed with 0 failures, 0 errors and 0 skipped: 144 Iceberg connector tests, 6 SPI contract tests and 38 FE Core tests. All 8 new engine cases passed. - The cases cover positive/negative precision boundaries, reversed comparisons, AND/OR/NOT, preservation of local predicates, source LIMIT handling, and ordinary predicate pushdown. - FE Checkstyle, connector import boundaries, the metadata funnel gate and `git diff --check` passed. Executed through `run-fe-ut.sh`; a local `CUSTOM_MVN` wrapper selected `:fe-core -am`, enabled Checkstyle, selected the matching Thrift compiler, and disabled the Maven build cache. Test selection: ```bash bash run-fe-ut.sh --run 'PluginDrivenScanNodeIcebergCastTest,PluginDrivenScanNodeCastPredicateTest,PluginDrivenScanNodeLimitStripTest,ExprToConnectorExpressionConverterTest,IcebergPredicateConverter*Test,IcebergConnectorMetadataTest,ConnectorMetadataSurfaceTest,ConnectorPluginSurfaceTest' ``` The 72-module reactor compiled and validated successfully. This was a selected unit-test run; the full FE suite, live cluster SQL/REST reads, BE unit tests, full FE packaging and remote CI were not run. ### Release note Fix missing rows in Iceberg queries whose CAST predicates could incorrectly prune matching files. Remaining CAST predicates are evaluated in Doris and may require scanning more files. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. Existing SPI notes and the test coverage inventory are updated. - Pre-submission status: Partially verified: selected local unit tests and style checks passed; live SQL, BE execution and remote CI remain unverified. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, tests, documentation updates and this PR description. - Human review: Pending author and maintainer review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
