hubgeter opened a new pull request, #70674:
URL: https://github.com/apache/doris/pull/70674

   ### Background
   
   Iceberg uses pushed predicates to prune manifests and data files before 
Doris evaluates rows. The FE converts scan predicates to neutral connector 
expressions, whose current representation unwraps CAST expressions. Predicates 
kept in Doris are evaluated after scanning the selected files.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #69551 (added the original disabled reproduction), #70511 
(related column-to-string CAST protection)
   PR Type: Bugfix
   
   Problem Summary:
   
   A DECIMAL(38,0) value of `9007199254740993` matches `CAST(value AS DOUBLE) = 
CAST(9007199254740992 AS DOUBLE)`, but removing the column CAST lets Iceberg 
compare the original decimal value and prune a matching file. Declare CAST 
predicate pushdown unsupported for Iceberg so the existing scan gate preserves 
these predicates in Doris before CAST information is lost. Most of this change 
moves the disabled downstream reproduction to engine tests covering planned 
expressions, actual Iceberg metadata, and row/file retention; independent plain 
conjuncts remain pushable.
   
   ### Details
   
   - **Frequency:** Data dependent: a remaining CAST changes comparison 
semantics, as adjacent high-precision decimals can become equal after 
conversion to DOUBLE.
   - **Root cause:** Iceberg opts into CAST predicate pushdown while the 
neutral converter unwraps the CAST. Source-side file pruning then uses a 
stronger predicate, and later Doris filtering cannot recover an unscanned file.
   - **Why existing tests missed it:** The original precision-loss reproduction 
in #69551 was disabled. It also started after CAST information had already 
disappeared; the replacement tests exercise the engine's filter gate with real 
planned column CASTs and actual Iceberg metadata.
   - **Fix:** Return `false` from Iceberg's existing CAST pushdown capability. 
The existing gate keeps the entire CAST-bearing conjunct local, preserves 
independent AND pushdown and complete OR/NOT semantics, and suppresses source 
LIMIT when local predicates remain. Update the SPI implementation notes and 
coverage inventory.
   - **Other paths and branches:** The capability API is unchanged. #70511 
protects column-to-string CASTs and does not cover DECIMAL-to-DOUBLE 
comparisons. No release-branch backport is included; live SQL and native BE 
execution remain unverified.
   - **Behavior change:** Iceberg scans may read more files because all 
remaining CAST-bearing predicates, including otherwise safe ones, stay in 
Doris. CASTs already removed by the optimizer and independent predicates 
without CAST remain eligible for pushdown.
   
   ### Testing
   
   Validated locally on master base `e87bc301743969fdc123b6117c8077771ffc84f7`, 
with JDK 17 and Thrift 0.24.0:
   
   - Before the fix, explicitly running the original disabled SDK reproduction 
failed both matching-row and matching-file retention assertions. The new engine 
suite ran 8 cases with 7 failures and 1 pass.
   - After the fix, 188 selected tests passed with 0 failures, 0 errors and 0 
skipped: 144 Iceberg connector tests, 6 SPI contract tests and 38 FE Core 
tests. All 8 new engine cases passed.
   - The cases cover positive/negative precision boundaries, reversed 
comparisons, AND/OR/NOT, preservation of local predicates, source LIMIT 
handling, and ordinary predicate pushdown.
   - FE Checkstyle, connector import boundaries, the metadata funnel gate and 
`git diff --check` passed.
   
   Executed through `run-fe-ut.sh`; a local `CUSTOM_MVN` wrapper selected 
`:fe-core -am`, enabled Checkstyle, selected the matching Thrift compiler, and 
disabled the Maven build cache. Test selection:
   
   ```bash
   bash run-fe-ut.sh --run 
'PluginDrivenScanNodeIcebergCastTest,PluginDrivenScanNodeCastPredicateTest,PluginDrivenScanNodeLimitStripTest,ExprToConnectorExpressionConverterTest,IcebergPredicateConverter*Test,IcebergConnectorMetadataTest,ConnectorMetadataSurfaceTest,ConnectorPluginSurfaceTest'
   ```
   
   The 72-module reactor compiled and validated successfully. This was a 
selected unit-test run; the full FE suite, live cluster SQL/REST reads, BE unit 
tests, full FE packaging and remote CI were not run.
   
   ### Release note
   
   Fix missing rows in Iceberg queries whose CAST predicates could incorrectly 
prune matching files. Remaining CAST predicates are evaluated in Doris and may 
require scanning more files.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No. Existing SPI notes and the test coverage 
inventory are updated.
   - Pre-submission status: Partially verified: selected local unit tests and 
style checks passed; live SQL, BE execution and remote CI remain unverified.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6).
   - Generated: Code, tests, documentation updates and this PR description.
   - Human review: Pending author and maintainer review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to