[I] Wrong query results for filters that involve partition columns and data file columns [datafusion]

[via GitHub]

adriangb opened a new issue, #15912:
URL: https://github.com/apache/datafusion/issues/15912

   ### Describe the bug
   
   Filters such as `partition_col = col_from_file` are never applied if 
`datafusion.execution.parquet.pushdown_filters = true`
   
   ### To Reproduce
   
   With `datafusion-cli`:
   
   ```sql
   COPY  (
     SELECT arrow_cast('a', 'Utf8') AS val
   )  TO 'test_files/scratch/test/part=a/123.parquet'
   STORED AS PARQUET;
   COPY  (
     SELECT arrow_cast('b', 'Utf8') AS val
   )  TO 'test_files/scratch/test/part=b/123.parquet'
   STORED AS PARQUET;
   COPY  (
     SELECT arrow_cast('xyz', 'Utf8') AS val
   )  TO 'test_files/scratch/test/part=c/123.parquet'
   STORED AS PARQUET;
   
   set datafusion.execution.parquet.pushdown_filters = true;
   
   CREATE EXTERNAL TABLE test(part text, val text)
   STORED AS PARQUET
   PARTITIONED BY (part)
   LOCATION 'test_files/scratch/test/';
   
   SELECT * FROM test;
   
   explain analyze
   select * from test
   where part != val;
   ```
   
   ```
   > select * from test
   where part != val;
   +-----+------+
   | val | part |
   +-----+------+
   | a   | a    |
   | xyz | c    |
   | b   | b    |
   +-----+------+
   3 row(s) fetched. 
   ```
   
   Which is clearly wrong.
   
   ### Expected behavior
   
   ```
   > select * from test
   where part != val;
   +-----+------+
   | val | part |
   +-----+------+
   | xyz | c    |
   +-----+------+
   ```
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org