adriangb commented on issue #15338:
URL: https://github.com/apache/datafusion/issues/15338#issuecomment-5196650328
This appears fixed on `main` (verified against `datafusion-cli 54.1.0`).
Name-based nested struct casting now handles field reordering, so the specific
complaint here (that inserting `someNewField` *before* `fields` fails while
appending it at the end work) no longer reproduces.
Repro, using a Parquet file whose physical struct order differs from the
declared table order, which is the path the delta-rs failure took:
```sql
COPY (
SELECT 'ws_2' AS uid,
{'properties': {'someNewField': 'test-value',
'fields': [{'messageId': 'veniam sed et elit'}]}}
AS event
) TO '/tmp/df15338/a.parquet' STORED AS PARQUET;
-- declared order is {fields, someNewField}; the file's is {someNewField,
fields}
CREATE EXTERNAL TABLE a (
uid VARCHAR,
event STRUCT<"properties" STRUCT<"fields" STRUCT<"messageId" VARCHAR>[],
"someNewField" VARCHAR>>
) STORED AS PARQUET LOCATION '/tmp/df15338/a.parquet';
SELECT * FROM a;
-- | ws_2 | {properties: {fields: [{messageId: veniam sed et elit}],
someNewField: test-value}} |
```
(The quoted identifiers matter — unquoted ones are lowercased and then
genuinely won't match by name.)
Schema evolution across two files in one directory works too, with the older
file's missing field null-filled:
```
| ws_1 | {properties: {fields: [{messageId: first write}], someNewField:
NULL}} |
| ws_2 | {properties: {fields: [{messageId: second write}], someNewField:
test-value}} |
```
The `item` vs `element` list-item name difference visible in the original
error is also handled: `cast_column` matches names only inside `Struct`, and
rebuilds list types from the target's item field.
Closing as fixed. @ion-elgreco @dabljues @bodubas if you still hit this
through delta-rs please let us know and we can re-open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
