mxm commented on issue #13406:
URL: https://github.com/apache/iceberg/issues/13406#issuecomment-3095715002
> One of the trickiest parts is handling lists of structs, especially when
those structs contain optional fields. So that the first struct/object in the
list does not have all elements..
Are you referring to something like the following?
```
Type: list(struct(int, string))
Example: [{1, null}, {2, "data"}, ...]
```
>We ended up writing our own schema parser that dynamically generates
schemas from incoming events and merges them with the existing table schema.
That sounds like you are replacing functionality of the Dynamic Sink. The
first part, schema generation, should be handled by you, but merging with the
existing table schema should be handled by the Dynamic Sink.
Could you perhaps give an example where the Dynamic Sink fails to evolve the
table schema?
>It also checks the field/row order and reorders the data to match the
expected schema.
Field order should be handled automatically by the Dynamic Sink. We match
fields by name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
