raghuvanshraj commented on issue #27089:
URL: https://github.com/apache/datafusion/issues/27089#issuecomment-6949236514
@alamb it is the second one, the per-file cast null-fills
`typed_value.status` for `b.parquet`. Schema merging is not the problem. It
does not pick the first file, it unions the two `typed_value` structs, so the
table has `typed_value.id` and `typed_value.status` whichever file is listed
first. `mixed_events/` is the directory from the repro above.
`mixed_events_rev/` holds the same two files with `a.parquet` renamed so it
sorts after `b.parquet`:
```shell
$ ls mixed_events mixed_events_rev
mixed_events:
a.parquet b.parquet
mixed_events_rev:
b.parquet z.parquet
```
```sql
SELECT arrow_typeof(event) FROM 'mixed_events/' LIMIT 1;
SELECT arrow_typeof(event) FROM 'mixed_events_rev/' LIMIT 1;
```
```text
Struct("metadata": non-null BinaryView, "value": BinaryView, "typed_value":
Struct("id": non-null Struct("value": BinaryView, "typed_value": Int64),
"status": Struct("value": BinaryView, "typed_value": Int64)))
Struct("metadata": non-null BinaryView, "value": BinaryView, "typed_value":
Struct("id": non-null Struct("value": BinaryView, "typed_value": Int64),
"status": Struct("value": BinaryView, "typed_value": Int64)))
```
Rows 4-6 are null in both directories. That union is a fine table shape for
Variant, the shredded file already reads through it.
The wrong rows come from the per-file cast. For `b.parquet` the default
adapter sees `event` with a different struct type and wraps it in a `CastExpr`
([schema_rewriter.rs#L774-L778](https://github.com/apache/datafusion/blob/3e55e62388ad60c2302660fe5d728e346ab6309f/datafusion/physical-expr-adapter/src/schema_rewriter.rs#L774-L778)),
and the struct cast fills the missing `typed_value.status` child with nulls
([nested_struct.rs#L107-L111](https://github.com/apache/datafusion/blob/3e55e62388ad60c2302660fe5d728e346ab6309f/datafusion/common/src/nested_struct.rs#L107-L111)).
Under the shredding spec a field whose `value` and `typed_value` are both null
means "this field does not exist"
([VariantShredding.md#L195](https://github.com/apache/parquet-format/blob/5cc18cae643835b19f869d3cffb085cd3c04eb15/VariantShredding.md#L195)),
so `status` is never looked up in the residual `value` bytes where `b.parquet`
actually has it.
`VariantExprAdapterFactory` replaces that cast with a reshred of the file's
layout into the table's
([expr_adapter.rs#L211](https://github.com/datafusion-contrib/datafusion-variant/blob/8141d3f/src/expr_adapter.rs#L211)),
which is why `CREATE EXTERNAL TABLE` returns the right rows. The url table
builds its `ListingTable` without any adapter factory
([dynamic_file.rs#L85-L95](https://github.com/apache/datafusion/blob/3e55e62388ad60c2302660fe5d728e346ab6309f/datafusion/core/src/datasource/dynamic_file.rs#L85-L95)),
so it gets the default one and the null fill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]