voonhous opened a new issue, #20568:
URL: https://github.com/apache/hudi/issues/20568

   ### Describe the problem
   
   Data skipping on an `IN` list of more than 10 values over a DATE or 
TIMESTAMP column fails to compile against the stats table. Spark's `OptimizeIn` 
turns such a list into an `InSet` 
(`spark.sql.optimizer.inSetConversionThreshold`), whose values are Catalyst 
internal ones. `Verdict` makes literals from them with `Literal(value)` 
(`Verdict.scala:43`, `:74`), so a DATE value becomes an INT and a TIMESTAMP 
value a BIGINT, which do not compare with the DATE / TIMESTAMP stats columns:
   
   ```
   AnalysisException: [DATATYPE_MISMATCH.BINARY_OP_DIFF_TYPES] Cannot resolve 
"(d_minValue <= 19611)" due to data type mismatch: the left and right operands 
of the binary operator have incompatible types ("DATE" and "INT").
   ```
   
   With `hoodie.fileIndex.dataSkippingFailureMode=strict` the query fails. With 
the default `fallback`, it logs the error and loses all pruning. 
`golden-index-filters.txt` pins the output: `d INSET (0:date)` compiles to 
`((d_minValue <= 0:int) AND (d_maxValue >= 0:int))`.
   
   ### Repro
   
   ```sql
   SET hoodie.metadata.index.column.stats.enable=true;
   
   CREATE TABLE t (ts STRING, d DATE, inst TIMESTAMP, uuid STRING, rider 
STRING, fare DOUBLE, city STRING) USING HUDI
   PARTITIONED BY (city)
   TBLPROPERTIES (primaryKey = 'uuid', preCombineField = 'ts');
   
   INSERT INTO t VALUES
   ('2023-09-19 23:40:49',DATE'2023-09-19',TIMESTAMP'2023-09-19 
23:40:49','u1','rider-A',19.10,'san_francisco'),
   ('2023-09-21 16:34:26',DATE'2023-09-21',TIMESTAMP'2023-09-21 
16:34:26','u2','rider-E',93.50,'san_francisco'),
   ('2023-09-23 09:42:17',DATE'2023-09-23',TIMESTAMP'2023-09-23 
09:42:17','u3','rider-F',34.15,'sao_paulo'),
   ('2023-09-19 02:38:07',DATE'2023-09-19',TIMESTAMP'2023-09-19 
02:38:07','u4','rider-I',41.06,'chennai');
   
   SELECT rider FROM t WHERE d IN (DATE'2023-09-11', DATE'2023-09-12', 
DATE'2023-09-13', DATE'2023-09-14', DATE'2023-09-15', DATE'2023-09-16', 
DATE'2023-09-17', DATE'2023-09-18', DATE'2023-09-19', DATE'2023-09-20', 
DATE'2023-09-21');
   ```
   
   Master `7888bf909694`, Spark 3.5, `hoodie.enable.data.skipping=true`:
   
   | filter | `fallback` | `strict` |
   | --- | --- | --- |
   | `d IN` 11 dates | correct rows, 3 of 3 files scanned | `AnalysisException` 
|
   | `inst IN` 11 timestamps | correct rows, 3 of 3 files scanned | 
`AnalysisException` |
   | `d IN` 2 dates (control, stays `In`) | 1 file | 1 file |
   
   ### Suggested fix
   
   Build the literals with the term's type, for ex `Literal.create(value, 
term.dataType)`, keeping the UTF8String handling.
   
   Related: #20540 keeps `InSet` out of the expression index because of this. 
#20537 removes that gate, so it needs this fix first.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to