gner56 opened a new issue, #70503: URL: https://github.com/apache/doris/issues/70503
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Reproduced on 4.1.0. The same code path is present on master (checked at e125ea8586). ### What's Wrong? A predicate that compares an integer column as a string, such as `CAST(int_col AS VARCHAR) >= '100'`, returns wrong results on Iceberg tables. Rows that satisfy the string comparison are silently missing, and no error is raised. The CAST is stripped before the predicate is pushed down to Iceberg, and the string literal is parsed as the column's type. Iceberg receives `int_col >= 100` and prunes data files on that numeric comparison. String and numeric order disagree (as strings `'15' >= '100'`, since `'5' > '0'`; as numbers `15 < 100`), so a file holding only matching rows like `15` is skipped. Doris still evaluates the original predicate on the rows it reads, so only whole pruned files are lost. On master the path is: `PluginDrivenScanNode.buildRemainingFilter` keeps CAST-bearing conjuncts, because `IcebergConnectorMetadata.supportsCastPredicatePushdown` returns `true`. `ExprToConnectorExpressionConverter.convert` unwraps every `CastExpr` (`return convert(expr.getChild(0))`), so the comparison reaches the connector as `IcebergPredicateConverter.extractIcebergLiteral` parses the string literal for column (`Integer.parseInt(string)`), producing `int_col >= 100` for `TableScan.filter`. The javadoc on `IcebergConnectorMetadata.supportsCastPredicatePushdown` describes this risk ("a comparison whose literal iceberg binds differently than Doris coerced it can skip f matching rows") and notes that no defect had been observed. This is a concrete case. On 4.1 the same happens in `IcebergUtils.convertDorisExprToSlotRef`, which unwraps any `CastEx `SlotRef`. ### What You Expected? `CAST(int_col AS VARCHAR) >= '100' AND CAST(int_col AS VARCHAR) < '200'` returns every row whose string form sorts into that range, as it does on a Doris internal table. ### How to Reproduce? ```sql CREATE CATALOG ice PROPERTIES ( 'type' = 'iceberg', 'iceberg.catalog.type' = 'hadoop', 'warehouse' = 'file:///tmp/iceberg_warehouse' ); CREATE DATABASE ice.db; CREATE TABLE ice.db.t (id INT); -- One data file per INSERT: file 1 holds only id 15, file 2 only id 150. INSERT INTO ice.db.t VALUES (15); INSERT INTO ice.db.t VALUES (150); SELECT id FROM ice.db.t WHERE CAST(id AS VARCHAR) >= '100' AND CAST(id AS VARCHAR) < '200' ORDER BY id; -- Expected: 15, 150 -- Actual: 150 EXPLAIN VERBOSE SELECT id FROM ice.db.t WHERE CAST(id AS VARCHAR) >= '100' AND CAST(id AS VARCHAR) < '200'; ``` The plan shows the CAST kept in Doris's own predicates but stripped in the Iceberg pushdown, with one of the two files pruned: ``` predicates: (CAST(id[#0] AS varchar(65533)) >= '100'), (CAST(id[#0] AS varchar(65533)) < '200') inputSplitNum=1, totalFileSize=982, scanRanges=1 icebergPredicatePushdown= ref(name="id") >= 100 ref(name="id") < 200 ``` For comparison, all of these return both rows: the same query without the filter; the same comparison when it is not pushed down, e.g. `CONCAT(CAST(id AS VARCHAR), '') >= '100' AND CONCAT(CAST(id AS VARCHAR), '') < ' the same data and filter on an internal table. With a local-file warehouse the table's `data/` directory may need creating by hand before the first INSERT; with S3 or HDFS it doesn't. Anything Else? Proposed fix: in `PluginDrivenScanNode.buildRemainingFilter`, keep a conjunct local contains a CAST from a non-string type to a string type, alongside the existing `containsTimestampTzCast` check. Such casts change ordering, so the stripped compar used for pruning. Casts that preserve ordering (e.g. INT to BIGINT) would keep being pushed down. I'm happy to submit a PR, including a regression test with the repro above. ### Anything Else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
