mhatipog commented on issue #27455:
URL: https://github.com/apache/datafusion/issues/27455#issuecomment-6921404161
The same happens with float partition columns, where several valid spellings
of one value are common: `-0` vs `-0.0` vs `0` vs `0.0`, `1e2` vs `100.0`, and
`NaN` vs `nan`. In addition, `-Infinity` never matches, because the literal's
`to_string()` is `-inf`. DataFusion 55.0.0:
```python
import os, tempfile, pyarrow as pa, pyarrow.parquet as pq, datafusion
root = tempfile.mkdtemp()
for i, s in enumerate(["-0", "0.0", "1e2", "100.0", "-Infinity", "NaN"]):
os.makedirs(f"{root}/d={s}"); pq.write_table(pa.table({"id":
pa.array([i], pa.int64())}), f"{root}/d={s}/p.parquet")
ctx = datafusion.SessionContext()
ctx.register_parquet("t", root, table_partition_cols=[("d", pa.float64())],
file_extension=".parquet")
ctx.sql("SELECT id, d FROM t ORDER BY id").to_pylist()
# d parsed as -0.0, 0.0, 100.0, 100.0, -inf, nan
ctx.sql("SELECT id FROM t WHERE d = 0.0").to_pylist()
# [1] expected [0, 1]
ctx.sql("SELECT id FROM t WHERE d = 100.0").to_pylist()
# [3] expected [2, 3]
ctx.sql("SELECT id FROM t WHERE d = CAST('-Infinity' AS
DOUBLE)").to_pylist() # [] expected [4]
ctx.sql("SELECT id FROM t WHERE d >= 0.0 AND d <= 0.0").to_pylist()
# [1] (simplified to d = 0.0)
ctx.sql("SELECT id FROM t WHERE d IN (0.0, 100.0)").to_pylist()
# [0, 1, 2, 3] correct
```
In a wider run on Linux with 16 spellings, `d = CAST('NaN' AS DOUBLE)` and
`d = CAST('Infinity' AS DOUBLE)` (which miss the `nan` and `inf` directories)
and `d BETWEEN -0.0 AND -0.0` fail the same way. DuckDB, Polars,
pyarrow.dataset and Spark return the full set for every predicate there. So a
fix that compares parsed values rather than strings (or that falls back to
listing when the literal has more than one spelling) would also need to cover
floats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]