On Mon, Jul 29, 2019 at 10:25 AM Antoine Pitrou <anto...@python.org> wrote: > > > Le 26/07/2019 à 07:27, Micah Kornfield a écrit : > > I started PR https://github.com/apache/arrow/pull/4951 to capture the > > proposed change. > > > > I would like to verify that with this change all alignment issues are > > solved. > > > > It seems like there is mostly consensus on having another minor release to > > get this resolved. > > You mean 0.15.0, right? > > > I think the open questions I have: > > 1. Do we want to provide backwards compatibility for writers? > > IMHO, yes. How many APIs would we have to add a flag to? >
To reduce the burden of maintaining such flags I would suggest introducing an IpcOptions struct in C++ to APIs that can produce messages. There are some other kinds of IPC configuration that might be useful in the future, such as switching between 8-byte (current) and 64-byte padding. > Or do we need a more general "provide_compatibility_with_version" option > on writers, for future format changes? > > > 2. Similar to the file format, should there be some leading "magic" > > indicator in the stream format to indicate a version, so we can do > > something a little less ugly if we encounter more issues? > > At least a version number. Perhaps also a magic number, I don't know. > We have a version number in the metadata already https://github.com/apache/arrow/blob/master/format/Schema.fbs#L22. I don't think we need to add other signaling beyond the 0xFFFFFFFF (stream continues / non-null message) / 0x00000000 (stream end / null message) marker at the beginning of the IPC payload Unless you envision a situation where the metadata cannot be obtained. I would hope that this can never occur. > Regards > > Antoine.
