ianthetechie opened a new issue, #27253:
URL: https://github.com/apache/datafusion/issues/27253
### Describe the bug
Valid queries against a parquet table with certain structural
characteristics now fail with an execution error: `Field common not found in
struct.`
As noted in the issue title, my _guess_ is that this is due to
overaggressive pruning of nested fields during pushdown, but that needs some
further verification.
I have worked up a minimal reproducer (below) which you can replicate with
the `datafusion-cli`.
### To Reproduce
```sql
-- Set up: an external parquet table with a few rows.
-- Apologies for the somewhat oblique construction, but certain DataFusion
versions
-- rejected a simpler version with map(['en'], [english]).
COPY (
SELECT named_struct(
'primary', local_name,
'common', map([language], [english])
) AS names
FROM (VALUES ('Local', 'en', 'English'), (NULL, 'en', 'Fallback'), ('',
'en', ''))
AS input(local_name, language, english)
) TO '/tmp/datafusion-reproducer.parquet' STORED AS PARQUET;
CREATE EXTERNAL TABLE nested_names
STORED AS PARQUET
LOCATION '/tmp/datafusion-reproducer.parquet';
-- These are defaults for a while AFAIK but including for reproducibility.
SET datafusion.execution.parquet.pushdown_filters = true;
SET datafusion.optimizer.enable_leaf_expression_pushdown = true;
-- Expected:
-- +-----------------------------+
-- | nested_names.names[primary] |
-- +-----------------------------+
-- | Local |
-- | NULL |
-- +-----------------------------+
--
-- Actual:
-- Parquet error: External: External error: Error evaluating filter predicate
-- caused by
-- Execution error: Field common not found in struct
SELECT names.primary
FROM nested_names
WHERE coalesce(names.primary, names.common['en']) <> '';
SET datafusion.execution.parquet.pushdown_filters = false;
SET datafusion.optimizer.enable_leaf_expression_pushdown = false;
-- With BOTH of these settings disabled, it succeeds and returns Local and
NULL.
-- It still fails if either one is true.
SELECT names.primary
FROM nested_names
WHERE coalesce(names.primary, names.common['en']) <> '';
-- Hacky SQL workaround: succeeds even with pushdown enabled.
SET datafusion.execution.parquet.pushdown_filters = true;
SET datafusion.optimizer.enable_leaf_expression_pushdown = true;
SELECT names.primary
FROM nested_names
WHERE coalesce(names.primary, map_extract(names.common, 'en')[1]) <> '';
```
### Expected behavior
The original query should succeed as written with syntax like
`names.common['en']`, without needing to disable predicate pushdown or
SQL-level hacks.
### Additional context
I haven't had a chance to do a full git bisect, but installing a few
releases, I've narrowed it quite a bit. Release-wise, the bug first appeared in
54.0.0; 53.1.0 was fine, and it is still broken on `main` at the time of this
writing.
### Willingness to contribute
I would like to contribute a fix for this bug, but request guidance from the
DataFusion community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
