tustvold commented on issue #5427: URL: https://github.com/apache/arrow-rs/issues/5427#issuecomment-1964858358
> Right, this is why I was thinking it would actually help maintenance if we separated the IO parts (object_store) from the parquet encoding/decoding parts. That way it would be clear where the responsibilities lay I think it is important that a parquet implementation provides first party support for object stores, and this is ultimately the use-case object_store was developed for. Much like arrow-cpp has ArrowFilesystem, and Hadoop has HadoopFilesystem, almost all data analytics ecosystems provide some first-party IO abstraction. That's not to say we shouldn't provide extension points for people to do different things, but it makes sense, at least to me, that a parquet crate provides a first-party way to read parquet data stored in object storage -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
