I'll do the JS updates. Is it safe to validate against the Arrow C++
integration tests?
On 8/22/19 7:28 PM, Micah Kornfield wrote:
I created https://issues.apache.org/jira/browse/ARROW-6313 as a tracking
issue with sub-issues on the development work. So far no-one has claimed
Java and Javascript tasks.
Would it make sense to have a separate dev branch for this work?
Thanks,
Micah
On Thu, Aug 22, 2019 at 3:24 PM Wes McKinney <wesmck...@gmail.com> wrote:
The vote carries with 4 binding +1 votes and 1 non-binding +1
I'll merge the specification patch later today and we can begin
working on implementations so we can get this done for 0.15.0
On Tue, Aug 20, 2019 at 12:30 PM Bryan Cutler <cutl...@gmail.com> wrote:
+1 (non-binding)
On Tue, Aug 20, 2019, 7:43 AM Antoine Pitrou <solip...@pitrou.net>
wrote:
Sorry, had forgotten to send my vote on this.
+1 from me.
Regards
Antoine.
On Wed, 14 Aug 2019 17:42:33 -0500
Wes McKinney <wesmck...@gmail.com> wrote:
hi all,
As we've been discussing [1], there is a need to introduce 4 bytes of
padding into the preamble of the "encapsulated IPC message" format to
ensure that the Flatbuffers metadata payload begins on an 8-byte
aligned memory offset. The alternative to this would be for Arrow
implementations where alignment is important (e.g. C or C++) to copy
the metadata (which is not always small) into memory when it is
unaligned.
Micah has proposed to address this by adding a
4-byte "continuation" value at the beginning of the payload
having the value 0xFFFFFFFF. The reason to do it this way is that
old clients will see an invalid length (what is currently the
first 4 bytes of the message -- a 32-bit little endian signed
integer indicating the metadata length) rather than potentially
crashing on a valid length. We also propose to expand the "end of
stream" marker used in the stream and file format from 4 to 8
bytes. This has the additional effect of aligning the file footer
defined in File.fbs.
This would be a backwards incompatible protocol change, so older
Arrow
libraries would not be able to read these new messages. Maintaining
forward compatibility (reading data produced by older libraries)
would
be possible as we can reason that a value other than the continuation
value was produced by an older library (and then validate the
Flatbuffer message of course). Arrow implementations could offer a
backward compatibility mode for the sake of old readers if they
desire
(this may also assist with testing).
Additionally with this vote, we want to formally approve the change
to
the Arrow "file" format to always write the (new 8-byte)
end-of-stream
marker, which enables code that processes Arrow streams to safely
read
the file's internal messages as though they were a normal stream.
The PR making these changes to the IPC documentation is here
https://github.com/apache/arrow/pull/4951
Please vote to accept these changes. This vote will be open for at
least 72 hours
[ ] +1 Adopt these Arrow protocol changes
[ ] +0
[ ] -1 I disagree because...
Here is my vote: +1
Thanks,
Wes
[1]:
https://lists.apache.org/thread.html/8440be572c49b7b2ffb76b63e6d935ada9efd9c1c2021369b6d27786@%3Cdev.arrow.apache.org%3E
