Fokko commented on issue #2792:
URL:
https://github.com/apache/iceberg-python/issues/2792#issuecomment-3612979029
Oof, this is a good point @geruh. I would also be leaning towards B. If we
want to fit this perfectly into PyIceberg today, then we need another decoder.
Right now, we have the BinaryDecorder for Avro, but we could also add a
JsonDecoder that takes the REST payload and reads it into the Record. You would
need a schema that matches the response, and that would fix it (except the maps
:/)
> Partition is unbound ["test"] instead of a Record
This should be pretty straightforward and we should be able to convert a
string into a record pretty easily.
> Maps are {"keys": [...], "values": [...]} instead of {key: value}
I think this one can be a bit more tricky, and a Pydantic pre-validator
might be able to solve that (we do similar stuff for the Iceberg Metadata JSON).
> Bounds are primitives (bytes, hex)
Do we need these in the first place? This should be evaluated by the
catalog. I would also be fine dropping these first, since they are probably
pretty big in memory as well.
> content is position-deletes string instead of enum int
Same as the maps, a pre-validator should do the trick here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
