[
https://issues.apache.org/jira/browse/ARROW-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928851#comment-16928851
]
Paul Taylor edited comment on ARROW-6370 at 9/12/19 7:53 PM:
-------------------------------------------------------------
[~saschahofmann]
bq. From my understanding of Arrow it should be a platform-independent format,
meaning that if I am sending an arrow table from Python to JS it should turn
out the same, right?
Yes, and that's what's happening here. But you're sending 8-byte integers to a
platform which has historically only supported 4-byte integers, which is why
you see each 8-byte integer as a pair of 4-byte integers.
I recommend reading [this post|https://v8.dev/features/bigint] on BigInts in
the v8 blog.
BigInts (and their related typed arrays) are relatively new additions to JS,
and aren't supported in all engines yet.
We have done our best to support getting and setting BigInt values when running
in VM that does support them, but for now we still have to support platforms
without BigInt. That's why the values Array for Int64Vector is a stride-2
Int32Array instead of a BigInt64Array.
was (Author: paul.e.taylor):
[~saschahofmann]
bq. From my understanding of Arrow it should be a platform-independent format,
meaning that if I am sending an arrow table from Python to JS it should turn
out the same, right?
Yes, and that's what's happening here. But you're sending 8-byte integers to a
platform which has historically only supported 4-byte integers, which is why
you see each 8-byte integer as a pair of 4-byte integers.
I recommend reading [this post|https://v8.dev/features/bigint] on BigInts in
the v8 blog.
BigInts (and their related typed arrays) are relatively new additions to JS,
and aren't supported in all engines yet.
We have done our best to support getting and setting BigInt values when running
in VM that do support them, but for now we still have to support platforms
without BigInt. That's why the values Array for Int64Vector is a stride-2
Int32Array instead of a BigInt64Array.
> [JS] Table.from adds 0 on int columns
> -------------------------------------
>
> Key: ARROW-6370
> URL: https://issues.apache.org/jira/browse/ARROW-6370
> Project: Apache Arrow
> Issue Type: Bug
> Components: JavaScript
> Affects Versions: 0.14.1
> Reporter: Sascha Hofmann
> Priority: Major
>
> I am generating an arrow table in pyarrow and send it via gRPC like this:
> {code:java}
> sink = pa.BufferOutputStream()
> writer = pa.RecordBatchStreamWriter(sink, batch.schema)
> writer.write_batch(batch)
> writer.close()
> yield ds.Response(
> status=200,
> loading=False,
> response=[sink.getvalue().to_pybytes()]
> )
> {code}
> On the javascript end, I parse it like that:
> {code:java}
> Table.from(response.getResponseList()[0])
> {code}
> That works but when I look at the actual table, int columns have a 0 for
> every other row. String columns seem to be parsed just fine.
> The Python byte array created from to_pybytes() has the same length as
> received in javascript. I am also able to recreate the original table for the
> byte array in Python.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
