rdblue commented on issue #170: Add support for Iceberg MR / InputFormat and OutputFormat APIs URL: https://github.com/apache/incubator-iceberg/issues/170#issuecomment-519659314 @guilload, good points about the reader functions. This is an area where we haven't invested more time because we are currently only supporting a small set of in-memory representations. What I would like is to eventually make a way to plug in representations for struct, map, and list and have a generic way to support those across data formats. Consider struct, for example. Iceberg uses an interface, `StructLike`, to work with struct objects. So if we write a `ValueReader` for Parquet and a `ValueReader` for Avro that supports a `StructLike`, we could then add an interface like this for plugging in a custom type: ``` interface DataModel<S extends StructLike, M extends Map, L extends List> { S newStruct(); M newMap(); L newList(); } ``` Some of the value readers are already generic and can also convert from an intermediate representation to a final representation. If we supported intermediate and final representations, then your final type wouldn't even need to implement `StructLike`. You could use a wrapper to translate from `StructLike` to your record's API and then return your record.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
