drin commented on issue #40583: URL: https://github.com/apache/arrow/issues/40583#issuecomment-2008256136
> File formats are intended to be an orthogonal detail to I/O. I won't belabor the point because I understand you to mean this for the Arrow library in particular. I was just mentioning that file format is not orthogonal if a storage system wants to improve performance. Parquet is an example where it is not orthogonal, the point of RowGroups is to allow portions of a file to be accessible independent of the larger file. But this was meant to be a broad comment, not on how the dataset API should be designed; I totally agree that we want to remove the skyhook file format if it's hindering Arrow's development. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
