matthewmturner commented on issue #5731: URL: https://github.com/apache/arrow-rs/issues/5731#issuecomment-2123041499
@erratic-pattern thank you for the feedback. i understand and agree with your point - documenting explicitly what goes in `app_metadata` in particular. That being said, I think we need to distinguish what should be done _now_ to get the different implementations in a state where they are compatible with each other (or explicitly deciding we are okay with a transitory state of being incompatible) and what the next steps are to update the spec to have better defined behavior with regards to `Any` / bytes encoding - which will require work from the whole arrow community. While we could hold off on making changes until there is agreement on the spec changes I think that leaves us in a quite bad spot as the current implementations are not compatible. My preference is to interpret the spec literally (i.e. `Any` is allowed where it is specified) and have all implementations align on this (even if it is not perfect). Then, update the spec to be explicit and the implementations can be updated to stay in conformance with that. I dont think there should be room for implementations to make any assumptions about how to handle the spec (maybe thats unreasonable, im not sure). The alternate route of course being that we leave things as they are in the current state, which is incompatible, and then work with the community to determine what changes are needed before making any updates. It's not clear to me that this could be done in a short period of time which is why my preference is to get conformance first and then update spec. But, we probably need opinions from other implementations as well - maybe its doable. CC @lidavidm and @zeroshade for their views -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
