[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513132#comment-17513132 ] Kazuaki Ishizaki commented on ARROW-15778: -- [~bryanc] Thank you for your suggestion. Sure, I will create a simple PR as is, and will open an follow-up issue to add a proper test. > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512169#comment-17512169 ] Bryan Cutler commented on ARROW-15778: -- [~kiszk] since your patch looks to be pretty simple and low risk, what do you think about making a PR for it as is? If it passes current tests, we know it doesn't mess up anything for little endian. Then we could have a separate task to add a proper test with big endian machine, if that requires more effort? > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17502198#comment-17502198 ] Ravi Gummadi commented on ARROW-15778: -- Thanks [~kiszk] > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17502054#comment-17502054 ] Kazuaki Ishizaki commented on ARROW-15778: -- This fix is already created [here|https://github.com/kiszk/arrow/commit/db06d043b07c63069169b5a88a4393aa229c095c]. We confirmed this issue was solved by this fix. The challenge is to create a test case. I realized that the latest arrow requires more native libraries for the build. It takes some time. So, I will create a test case with an older version arrow. > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497949#comment-17497949 ] Kazuaki Ishizaki commented on ARROW-15778: -- [~apitrou] Thank you. In another issue, I am suspecting the endianness in schema. I will look at this. > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497474#comment-17497474 ] David Li commented on ARROW-15778: -- Ah, I misunderstood then, thanks for clarifying. > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497472#comment-17497472 ] Antoine Pitrou commented on ARROW-15778: I'm not sure any form of negotiation is needed? The way it works at the IPC level is that the writer emits data in whichever endianness it chooses (also setting the corresponding metadata field to the appropriate value) and the reader decides to byte-swap data is required. So it would work similarly at the Flight level. > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497463#comment-17497463 ] David Li commented on ARROW-15778: -- Flight doesn't do any endianness detection/negotiation anyways (it expects producer/consumer to set appropriate options) though we should eventually fix that. > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497459#comment-17497459 ] Antoine Pitrou commented on ARROW-15778: Also cc [~lidavidm] since this affects Flight. > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (ARROW-15778) [Java] Endianness field not emitted in IPC stream
[ https://issues.apache.org/jira/browse/ARROW-15778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497456#comment-17497456 ] Antoine Pitrou commented on ARROW-15778: The offending code seems to be there: https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/types/pojo/Schema.java#L202-L213 This seems reasonably easy to fix (perhaps a one-line fix, though a test should ideally be added as well). [~emkornfield] [~kiszk] > [Java] Endianness field not emitted in IPC stream > - > > Key: ARROW-15778 > URL: https://issues.apache.org/jira/browse/ARROW-15778 > Project: Apache Arrow > Issue Type: Bug > Components: Java >Reporter: Antoine Pitrou >Priority: Major > Fix For: 8.0.0 > > > It seems the Java IPC writer implementation does not emit the Endianness > information at all (making it Little by default). This complicates > interoperability with the C++ IPC reader, which does read this information > and acts on it to decide whether it needs to byteswap the incoming data. -- This message was sent by Atlassian Jira (v8.20.1#820001)
