emkornfield commented on PR #9678: URL: https://github.com/apache/arrow-rs/pull/9678#issuecomment-4431260910
> TLDR is I think this is a great idea. I also think it woudl be ok to merge this into arrow-rs even if there is not a broader consensus on the mailing list that we should do it in the format itself My thinking is that some usecases are basically using parquet with the same read/writer and compatibility with older java based implementations is not important. This is the same thing for some of the newer encodings too @etseidl @alamb This is true, but if you follow this logic to is conclusion, then each implementation could come up with new encodings themselves without broader consensus in the ecosystem? Is that the direction the arrow-rs implementation aims to take? IIUC this would allow writing files that would fail parsing from other readers due to the field currently being required? If so, while it seems like there is general consensus on the parquet mailing list to transition to this, doing before it is adopted in parquet-format seems like a small risk of fragmenting the ecosystem? CC @wgtmac -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
