Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release
+1 (binding) In "[VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release" on Mon, 29 Jun 2020 16:42:45 -0500, Wes McKinney wrote: > Hi, > > As discussed on the mailing list [1], in order to demarcate the > pre-1.0.0 and post-1.0.0 worlds, and to allow the > forward-compatibility-protection changes we are making to actually > work (i.e. so that libraries can recognize that they have received > data with a feature that they do not support), I have proposed to > increment the MetadataVersion from V4 to V5. Additionally, if the > union validity bitmap changes are accepted, the MetadataVersion could > be used to control whether unions are permitted to be serialized or > not (with V4 -- used by v0.8.0 to v0.17.1, unions would not be > permitted). > > Since there have been no backward incompatible changes to the Arrow > format since 0.8.0, this would be no different, and (aside from the > union issue) libraries supporting V5 are expected to accept BOTH V4 > and V5 so that backward compatibility is not broken, and any > serialized data from prior versions of the Arrow libraries (0.8.0 > onward) will continue to be readable. > > Implementations are recommended, but not required, to provide an > optional "V4 compatibility mode" for forward compatibility > (serializing data from >= 1.0.0 that needs to be readable by older > libraries, e.g. Spark deployments stuck on an older Java-Arrow > version). In this compatibility mode, non-forward-compatible features > added in 1.0.0 and beyond would not be permitted. > > A PR with the changes to Schema.fbs (possibly subject to some > clarifying changes to the comments) is at [2]. > > Once the PR is merged, it will be necessary for implementations to be > updated and tested as appropriate at minimum to validate that backward > compatibility is preserved (i.e. V4 IPC payloads are still readable -- > we have some in apache/arrow-testing and can add more as needed). > > The vote will be open for at least 72 hours. > > [ ] +1 Accept addition of MetadataVersion::V5 along with its general > implications above > [ ] +0 > [ ] -1 Do not accept because... > > [1]: > https://lists.apache.org/thread.html/r856822cc366d944b3ecdf32c2ea9b1ad8fc9d12507baa2f2840a64b6%40%3Cdev.arrow.apache.org%3E > [2]: https://github.com/apache/arrow/pull/7566
Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release
+1 (binding) On Tue, Jun 30, 2020 at 2:53 AM Antoine Pitrou wrote: > +1 (binding) > > > Le 29/06/2020 à 23:42, Wes McKinney a écrit : > > Hi, > > > > As discussed on the mailing list [1], in order to demarcate the > > pre-1.0.0 and post-1.0.0 worlds, and to allow the > > forward-compatibility-protection changes we are making to actually > > work (i.e. so that libraries can recognize that they have received > > data with a feature that they do not support), I have proposed to > > increment the MetadataVersion from V4 to V5. Additionally, if the > > union validity bitmap changes are accepted, the MetadataVersion could > > be used to control whether unions are permitted to be serialized or > > not (with V4 -- used by v0.8.0 to v0.17.1, unions would not be > > permitted). > > > > Since there have been no backward incompatible changes to the Arrow > > format since 0.8.0, this would be no different, and (aside from the > > union issue) libraries supporting V5 are expected to accept BOTH V4 > > and V5 so that backward compatibility is not broken, and any > > serialized data from prior versions of the Arrow libraries (0.8.0 > > onward) will continue to be readable. > > > > Implementations are recommended, but not required, to provide an > > optional "V4 compatibility mode" for forward compatibility > > (serializing data from >= 1.0.0 that needs to be readable by older > > libraries, e.g. Spark deployments stuck on an older Java-Arrow > > version). In this compatibility mode, non-forward-compatible features > > added in 1.0.0 and beyond would not be permitted. > > > > A PR with the changes to Schema.fbs (possibly subject to some > > clarifying changes to the comments) is at [2]. > > > > Once the PR is merged, it will be necessary for implementations to be > > updated and tested as appropriate at minimum to validate that backward > > compatibility is preserved (i.e. V4 IPC payloads are still readable -- > > we have some in apache/arrow-testing and can add more as needed). > > > > The vote will be open for at least 72 hours. > > > > [ ] +1 Accept addition of MetadataVersion::V5 along with its general > > implications above > > [ ] +0 > > [ ] -1 Do not accept because... > > > > [1]: > https://lists.apache.org/thread.html/r856822cc366d944b3ecdf32c2ea9b1ad8fc9d12507baa2f2840a64b6%40%3Cdev.arrow.apache.org%3E > > [2]: https://github.com/apache/arrow/pull/7566 > > >
Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release
+1 (binding) Le 29/06/2020 à 23:42, Wes McKinney a écrit : > Hi, > > As discussed on the mailing list [1], in order to demarcate the > pre-1.0.0 and post-1.0.0 worlds, and to allow the > forward-compatibility-protection changes we are making to actually > work (i.e. so that libraries can recognize that they have received > data with a feature that they do not support), I have proposed to > increment the MetadataVersion from V4 to V5. Additionally, if the > union validity bitmap changes are accepted, the MetadataVersion could > be used to control whether unions are permitted to be serialized or > not (with V4 -- used by v0.8.0 to v0.17.1, unions would not be > permitted). > > Since there have been no backward incompatible changes to the Arrow > format since 0.8.0, this would be no different, and (aside from the > union issue) libraries supporting V5 are expected to accept BOTH V4 > and V5 so that backward compatibility is not broken, and any > serialized data from prior versions of the Arrow libraries (0.8.0 > onward) will continue to be readable. > > Implementations are recommended, but not required, to provide an > optional "V4 compatibility mode" for forward compatibility > (serializing data from >= 1.0.0 that needs to be readable by older > libraries, e.g. Spark deployments stuck on an older Java-Arrow > version). In this compatibility mode, non-forward-compatible features > added in 1.0.0 and beyond would not be permitted. > > A PR with the changes to Schema.fbs (possibly subject to some > clarifying changes to the comments) is at [2]. > > Once the PR is merged, it will be necessary for implementations to be > updated and tested as appropriate at minimum to validate that backward > compatibility is preserved (i.e. V4 IPC payloads are still readable -- > we have some in apache/arrow-testing and can add more as needed). > > The vote will be open for at least 72 hours. > > [ ] +1 Accept addition of MetadataVersion::V5 along with its general > implications above > [ ] +0 > [ ] -1 Do not accept because... > > [1]: > https://lists.apache.org/thread.html/r856822cc366d944b3ecdf32c2ea9b1ad8fc9d12507baa2f2840a64b6%40%3Cdev.arrow.apache.org%3E > [2]: https://github.com/apache/arrow/pull/7566 >
Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release
+1 (binding) On Tue, Jun 30, 2020, at 11:11 AM, Neville Dipale wrote: > +1 (non-binding) > > On Tue, 30 Jun 2020 at 06:29, Ben Kietzman wrote: > > > +1 (non binding) > > > > On Tue, Jun 30, 2020, 00:25 Wes McKinney wrote: > > > > > +1 (binding) > > > > > > On Mon, Jun 29, 2020 at 10:49 PM Micah Kornfield > > > wrote: > > > > > > > > +1 (binding) > > > > > > > > On Mon, Jun 29, 2020 at 2:43 PM Wes McKinney > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > As discussed on the mailing list [1], in order to demarcate the > > > > > pre-1.0.0 and post-1.0.0 worlds, and to allow the > > > > > forward-compatibility-protection changes we are making to actually > > > > > work (i.e. so that libraries can recognize that they have received > > > > > data with a feature that they do not support), I have proposed to > > > > > increment the MetadataVersion from V4 to V5. Additionally, if the > > > > > union validity bitmap changes are accepted, the MetadataVersion could > > > > > be used to control whether unions are permitted to be serialized or > > > > > not (with V4 -- used by v0.8.0 to v0.17.1, unions would not be > > > > > permitted). > > > > > > > > > > Since there have been no backward incompatible changes to the Arrow > > > > > format since 0.8.0, this would be no different, and (aside from the > > > > > union issue) libraries supporting V5 are expected to accept BOTH V4 > > > > > and V5 so that backward compatibility is not broken, and any > > > > > serialized data from prior versions of the Arrow libraries (0.8.0 > > > > > onward) will continue to be readable. > > > > > > > > > > Implementations are recommended, but not required, to provide an > > > > > optional "V4 compatibility mode" for forward compatibility > > > > > (serializing data from >= 1.0.0 that needs to be readable by older > > > > > libraries, e.g. Spark deployments stuck on an older Java-Arrow > > > > > version). In this compatibility mode, non-forward-compatible features > > > > > added in 1.0.0 and beyond would not be permitted. > > > > > > > > > > A PR with the changes to Schema.fbs (possibly subject to some > > > > > clarifying changes to the comments) is at [2]. > > > > > > > > > > Once the PR is merged, it will be necessary for implementations to be > > > > > updated and tested as appropriate at minimum to validate that > > backward > > > > > compatibility is preserved (i.e. V4 IPC payloads are still readable > > -- > > > > > we have some in apache/arrow-testing and can add more as needed). > > > > > > > > > > The vote will be open for at least 72 hours. > > > > > > > > > > [ ] +1 Accept addition of MetadataVersion::V5 along with its general > > > > > implications above > > > > > [ ] +0 > > > > > [ ] -1 Do not accept because... > > > > > > > > > > [1]: > > > > > > > > > > https://lists.apache.org/thread.html/r856822cc366d944b3ecdf32c2ea9b1ad8fc9d12507baa2f2840a64b6%40%3Cdev.arrow.apache.org%3E > > > > > [2]: https://github.com/apache/arrow/pull/7566 > > > > > > > > > > >
Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release
+1 (non-binding) On Tue, 30 Jun 2020 at 06:29, Ben Kietzman wrote: > +1 (non binding) > > On Tue, Jun 30, 2020, 00:25 Wes McKinney wrote: > > > +1 (binding) > > > > On Mon, Jun 29, 2020 at 10:49 PM Micah Kornfield > > wrote: > > > > > > +1 (binding) > > > > > > On Mon, Jun 29, 2020 at 2:43 PM Wes McKinney > > wrote: > > > > > > > Hi, > > > > > > > > As discussed on the mailing list [1], in order to demarcate the > > > > pre-1.0.0 and post-1.0.0 worlds, and to allow the > > > > forward-compatibility-protection changes we are making to actually > > > > work (i.e. so that libraries can recognize that they have received > > > > data with a feature that they do not support), I have proposed to > > > > increment the MetadataVersion from V4 to V5. Additionally, if the > > > > union validity bitmap changes are accepted, the MetadataVersion could > > > > be used to control whether unions are permitted to be serialized or > > > > not (with V4 -- used by v0.8.0 to v0.17.1, unions would not be > > > > permitted). > > > > > > > > Since there have been no backward incompatible changes to the Arrow > > > > format since 0.8.0, this would be no different, and (aside from the > > > > union issue) libraries supporting V5 are expected to accept BOTH V4 > > > > and V5 so that backward compatibility is not broken, and any > > > > serialized data from prior versions of the Arrow libraries (0.8.0 > > > > onward) will continue to be readable. > > > > > > > > Implementations are recommended, but not required, to provide an > > > > optional "V4 compatibility mode" for forward compatibility > > > > (serializing data from >= 1.0.0 that needs to be readable by older > > > > libraries, e.g. Spark deployments stuck on an older Java-Arrow > > > > version). In this compatibility mode, non-forward-compatible features > > > > added in 1.0.0 and beyond would not be permitted. > > > > > > > > A PR with the changes to Schema.fbs (possibly subject to some > > > > clarifying changes to the comments) is at [2]. > > > > > > > > Once the PR is merged, it will be necessary for implementations to be > > > > updated and tested as appropriate at minimum to validate that > backward > > > > compatibility is preserved (i.e. V4 IPC payloads are still readable > -- > > > > we have some in apache/arrow-testing and can add more as needed). > > > > > > > > The vote will be open for at least 72 hours. > > > > > > > > [ ] +1 Accept addition of MetadataVersion::V5 along with its general > > > > implications above > > > > [ ] +0 > > > > [ ] -1 Do not accept because... > > > > > > > > [1]: > > > > > > > https://lists.apache.org/thread.html/r856822cc366d944b3ecdf32c2ea9b1ad8fc9d12507baa2f2840a64b6%40%3Cdev.arrow.apache.org%3E > > > > [2]: https://github.com/apache/arrow/pull/7566 > > > > > > >
Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release
+1 (non binding) On Tue, Jun 30, 2020, 00:25 Wes McKinney wrote: > +1 (binding) > > On Mon, Jun 29, 2020 at 10:49 PM Micah Kornfield > wrote: > > > > +1 (binding) > > > > On Mon, Jun 29, 2020 at 2:43 PM Wes McKinney > wrote: > > > > > Hi, > > > > > > As discussed on the mailing list [1], in order to demarcate the > > > pre-1.0.0 and post-1.0.0 worlds, and to allow the > > > forward-compatibility-protection changes we are making to actually > > > work (i.e. so that libraries can recognize that they have received > > > data with a feature that they do not support), I have proposed to > > > increment the MetadataVersion from V4 to V5. Additionally, if the > > > union validity bitmap changes are accepted, the MetadataVersion could > > > be used to control whether unions are permitted to be serialized or > > > not (with V4 -- used by v0.8.0 to v0.17.1, unions would not be > > > permitted). > > > > > > Since there have been no backward incompatible changes to the Arrow > > > format since 0.8.0, this would be no different, and (aside from the > > > union issue) libraries supporting V5 are expected to accept BOTH V4 > > > and V5 so that backward compatibility is not broken, and any > > > serialized data from prior versions of the Arrow libraries (0.8.0 > > > onward) will continue to be readable. > > > > > > Implementations are recommended, but not required, to provide an > > > optional "V4 compatibility mode" for forward compatibility > > > (serializing data from >= 1.0.0 that needs to be readable by older > > > libraries, e.g. Spark deployments stuck on an older Java-Arrow > > > version). In this compatibility mode, non-forward-compatible features > > > added in 1.0.0 and beyond would not be permitted. > > > > > > A PR with the changes to Schema.fbs (possibly subject to some > > > clarifying changes to the comments) is at [2]. > > > > > > Once the PR is merged, it will be necessary for implementations to be > > > updated and tested as appropriate at minimum to validate that backward > > > compatibility is preserved (i.e. V4 IPC payloads are still readable -- > > > we have some in apache/arrow-testing and can add more as needed). > > > > > > The vote will be open for at least 72 hours. > > > > > > [ ] +1 Accept addition of MetadataVersion::V5 along with its general > > > implications above > > > [ ] +0 > > > [ ] -1 Do not accept because... > > > > > > [1]: > > > > https://lists.apache.org/thread.html/r856822cc366d944b3ecdf32c2ea9b1ad8fc9d12507baa2f2840a64b6%40%3Cdev.arrow.apache.org%3E > > > [2]: https://github.com/apache/arrow/pull/7566 > > > >
Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release
+1 (binding) On Mon, Jun 29, 2020 at 10:49 PM Micah Kornfield wrote: > > +1 (binding) > > On Mon, Jun 29, 2020 at 2:43 PM Wes McKinney wrote: > > > Hi, > > > > As discussed on the mailing list [1], in order to demarcate the > > pre-1.0.0 and post-1.0.0 worlds, and to allow the > > forward-compatibility-protection changes we are making to actually > > work (i.e. so that libraries can recognize that they have received > > data with a feature that they do not support), I have proposed to > > increment the MetadataVersion from V4 to V5. Additionally, if the > > union validity bitmap changes are accepted, the MetadataVersion could > > be used to control whether unions are permitted to be serialized or > > not (with V4 -- used by v0.8.0 to v0.17.1, unions would not be > > permitted). > > > > Since there have been no backward incompatible changes to the Arrow > > format since 0.8.0, this would be no different, and (aside from the > > union issue) libraries supporting V5 are expected to accept BOTH V4 > > and V5 so that backward compatibility is not broken, and any > > serialized data from prior versions of the Arrow libraries (0.8.0 > > onward) will continue to be readable. > > > > Implementations are recommended, but not required, to provide an > > optional "V4 compatibility mode" for forward compatibility > > (serializing data from >= 1.0.0 that needs to be readable by older > > libraries, e.g. Spark deployments stuck on an older Java-Arrow > > version). In this compatibility mode, non-forward-compatible features > > added in 1.0.0 and beyond would not be permitted. > > > > A PR with the changes to Schema.fbs (possibly subject to some > > clarifying changes to the comments) is at [2]. > > > > Once the PR is merged, it will be necessary for implementations to be > > updated and tested as appropriate at minimum to validate that backward > > compatibility is preserved (i.e. V4 IPC payloads are still readable -- > > we have some in apache/arrow-testing and can add more as needed). > > > > The vote will be open for at least 72 hours. > > > > [ ] +1 Accept addition of MetadataVersion::V5 along with its general > > implications above > > [ ] +0 > > [ ] -1 Do not accept because... > > > > [1]: > > https://lists.apache.org/thread.html/r856822cc366d944b3ecdf32c2ea9b1ad8fc9d12507baa2f2840a64b6%40%3Cdev.arrow.apache.org%3E > > [2]: https://github.com/apache/arrow/pull/7566 > >
Re: [VOTE] Increment MetadataVersion in Schema.fbs from V4 to V5 for 1.0.0 release
+1 (binding) On Mon, Jun 29, 2020 at 2:43 PM Wes McKinney wrote: > Hi, > > As discussed on the mailing list [1], in order to demarcate the > pre-1.0.0 and post-1.0.0 worlds, and to allow the > forward-compatibility-protection changes we are making to actually > work (i.e. so that libraries can recognize that they have received > data with a feature that they do not support), I have proposed to > increment the MetadataVersion from V4 to V5. Additionally, if the > union validity bitmap changes are accepted, the MetadataVersion could > be used to control whether unions are permitted to be serialized or > not (with V4 -- used by v0.8.0 to v0.17.1, unions would not be > permitted). > > Since there have been no backward incompatible changes to the Arrow > format since 0.8.0, this would be no different, and (aside from the > union issue) libraries supporting V5 are expected to accept BOTH V4 > and V5 so that backward compatibility is not broken, and any > serialized data from prior versions of the Arrow libraries (0.8.0 > onward) will continue to be readable. > > Implementations are recommended, but not required, to provide an > optional "V4 compatibility mode" for forward compatibility > (serializing data from >= 1.0.0 that needs to be readable by older > libraries, e.g. Spark deployments stuck on an older Java-Arrow > version). In this compatibility mode, non-forward-compatible features > added in 1.0.0 and beyond would not be permitted. > > A PR with the changes to Schema.fbs (possibly subject to some > clarifying changes to the comments) is at [2]. > > Once the PR is merged, it will be necessary for implementations to be > updated and tested as appropriate at minimum to validate that backward > compatibility is preserved (i.e. V4 IPC payloads are still readable -- > we have some in apache/arrow-testing and can add more as needed). > > The vote will be open for at least 72 hours. > > [ ] +1 Accept addition of MetadataVersion::V5 along with its general > implications above > [ ] +0 > [ ] -1 Do not accept because... > > [1]: > https://lists.apache.org/thread.html/r856822cc366d944b3ecdf32c2ea9b1ad8fc9d12507baa2f2840a64b6%40%3Cdev.arrow.apache.org%3E > [2]: https://github.com/apache/arrow/pull/7566 >
