Re: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Wes McKinney 
We do not have any tests that consider endianness. This would be a
welcome contribution, particularly for the IPC / memory sharing side
of things.

On Mon, Aug 1, 2016 at 10:56 AM, Sanjay Rao <getsanjay...@live.com> wrote:
> Hi Wes,
> Thanks for your reply, so can I say that C++ version of Arrow works on Big 
> Endian system(although not tested officially) ?
> Also would need your help in getting me details on how to run and one sample 
> program using APIs.
> Thanks a lot,Sanjay
>
>> From: wesmck...@gmail.com
>> Date: Mon, 1 Aug 2016 10:32:40 -0700
>> Subject: Re: Is LittleEndian system mandatory for using Apache Arrow ?
>> To: dev@arrow.apache.org
>> CC: emkornfi...@gmail.com
>>
>> I replied on the other thread (missed this one).
>>
>> I believe that since the C++ codebase uses native endianness, we would
>> need to add byte-swapping logic in the IPC code path to support big
>> endian systems.
>>
>> On Mon, Aug 1, 2016 at 9:31 AM, Sanjay Rao <getsanjay...@live.com> wrote:
>> > Hi Micah,
>> > Thanks, I am trying to write Java examples with Java libraries of Apache 
>> > Arrow on Big Endian system, I already got JARs built, this is part of a 
>> > research I am doing.
>> > I ran "mvn test" on Big Endian system and it failed in TestEndianess, 
>> > kindly let me know if at least Java version of Apache Arrow will run on 
>> > Big Endian system.
>> > Thanks,Sanjay
>> >
>> >> From: emkornfi...@gmail.com
>> >> Date: Mon, 1 Aug 2016 09:23:51 -0700
>> >> Subject: Re: Is LittleEndian system mandatory for using Apache Arrow ?
>> >> To: dev@arrow.apache.org
>> >>
>> >> Hi Sanjay,
>> >> There was some discussion on the mailing list a little while ago about 
>> >> this
>> >> [1].  We discussed making endianness part of the IPC metadata.  I don't
>> >> think we've gotten far enough in the implementation to take action on the
>> >> discussion though.
>> >>
>> >> Right now I think the C++ code at least is endian agnostic, but it hasn't
>> >> been tested against a big-endian system as far as I know.  As referenced 
>> >> in
>> >> the prior thread (and I don't think its been changed), there is an 
>> >> explicit
>> >> check for endianness when constructing vectors in Java
>> >>
>> >> Cheers,
>> >> Micah
>> >>
>> >>
>> >> https://mail-archives.apache.org/mod_mbox/arrow-dev/201604.mbox/%3CCAK7Z5T_0T9hOa=mbJ=V30jc_Em169e=qjpyqnsblrakb8dm...@mail.gmail.com%3E
>> >>
>> >> On Mon, Aug 1, 2016 at 8:57 AM, Sanjay Rao <getsanjay...@live.com> wrote:
>> >>
>> >> > Hi,
>> >> > I read that Integer representation is assumed to LittleEndian in Arrow,
>> >> > does this mean we cannot use Arrow in Big Endian Systems like SPARC ?
>> >> >
>> >> > Thanks,Sanjay
>> >
>

RE: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Sanjay Rao 
Hi Wes,
Thanks for your reply, so can I say that C++ version of Arrow works on Big 
Endian system(although not tested officially) ?
Also would need your help in getting me details on how to run and one sample 
program using APIs.
Thanks a lot,Sanjay

> From: wesmck...@gmail.com
> Date: Mon, 1 Aug 2016 10:32:40 -0700
> Subject: Re: Is LittleEndian system mandatory for using Apache Arrow ?
> To: dev@arrow.apache.org
> CC: emkornfi...@gmail.com
> 
> I replied on the other thread (missed this one).
> 
> I believe that since the C++ codebase uses native endianness, we would
> need to add byte-swapping logic in the IPC code path to support big
> endian systems.
> 
> On Mon, Aug 1, 2016 at 9:31 AM, Sanjay Rao <getsanjay...@live.com> wrote:
> > Hi Micah,
> > Thanks, I am trying to write Java examples with Java libraries of Apache 
> > Arrow on Big Endian system, I already got JARs built, this is part of a 
> > research I am doing.
> > I ran "mvn test" on Big Endian system and it failed in TestEndianess, 
> > kindly let me know if at least Java version of Apache Arrow will run on Big 
> > Endian system.
> > Thanks,Sanjay
> >
> >> From: emkornfi...@gmail.com
> >> Date: Mon, 1 Aug 2016 09:23:51 -0700
> >> Subject: Re: Is LittleEndian system mandatory for using Apache Arrow ?
> >> To: dev@arrow.apache.org
> >>
> >> Hi Sanjay,
> >> There was some discussion on the mailing list a little while ago about this
> >> [1].  We discussed making endianness part of the IPC metadata.  I don't
> >> think we've gotten far enough in the implementation to take action on the
> >> discussion though.
> >>
> >> Right now I think the C++ code at least is endian agnostic, but it hasn't
> >> been tested against a big-endian system as far as I know.  As referenced in
> >> the prior thread (and I don't think its been changed), there is an explicit
> >> check for endianness when constructing vectors in Java
> >>
> >> Cheers,
> >> Micah
> >>
> >>
> >> https://mail-archives.apache.org/mod_mbox/arrow-dev/201604.mbox/%3CCAK7Z5T_0T9hOa=mbJ=V30jc_Em169e=qjpyqnsblrakb8dm...@mail.gmail.com%3E
> >>
> >> On Mon, Aug 1, 2016 at 8:57 AM, Sanjay Rao <getsanjay...@live.com> wrote:
> >>
> >> > Hi,
> >> > I read that Integer representation is assumed to LittleEndian in Arrow,
> >> > does this mean we cannot use Arrow in Big Endian Systems like SPARC ?
> >> >
> >> > Thanks,Sanjay
> >

Re: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Wes McKinney 
I replied on the other thread (missed this one).

I believe that since the C++ codebase uses native endianness, we would
need to add byte-swapping logic in the IPC code path to support big
endian systems.

On Mon, Aug 1, 2016 at 9:31 AM, Sanjay Rao <getsanjay...@live.com> wrote:
> Hi Micah,
> Thanks, I am trying to write Java examples with Java libraries of Apache 
> Arrow on Big Endian system, I already got JARs built, this is part of a 
> research I am doing.
> I ran "mvn test" on Big Endian system and it failed in TestEndianess, kindly 
> let me know if at least Java version of Apache Arrow will run on Big Endian 
> system.
> Thanks,Sanjay
>
>> From: emkornfi...@gmail.com
>> Date: Mon, 1 Aug 2016 09:23:51 -0700
>> Subject: Re: Is LittleEndian system mandatory for using Apache Arrow ?
>> To: dev@arrow.apache.org
>>
>> Hi Sanjay,
>> There was some discussion on the mailing list a little while ago about this
>> [1].  We discussed making endianness part of the IPC metadata.  I don't
>> think we've gotten far enough in the implementation to take action on the
>> discussion though.
>>
>> Right now I think the C++ code at least is endian agnostic, but it hasn't
>> been tested against a big-endian system as far as I know.  As referenced in
>> the prior thread (and I don't think its been changed), there is an explicit
>> check for endianness when constructing vectors in Java
>>
>> Cheers,
>> Micah
>>
>>
>> https://mail-archives.apache.org/mod_mbox/arrow-dev/201604.mbox/%3CCAK7Z5T_0T9hOa=mbJ=V30jc_Em169e=qjpyqnsblrakb8dm...@mail.gmail.com%3E
>>
>> On Mon, Aug 1, 2016 at 8:57 AM, Sanjay Rao <getsanjay...@live.com> wrote:
>>
>> > Hi,
>> > I read that Integer representation is assumed to LittleEndian in Arrow,
>> > does this mean we cannot use Arrow in Big Endian Systems like SPARC ?
>> >
>> > Thanks,Sanjay
>

RE: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Sanjay Rao 
Hi Micah,
Thanks, I am trying to write Java examples with Java libraries of Apache Arrow 
on Big Endian system, I already got JARs built, this is part of a research I am 
doing.
I ran "mvn test" on Big Endian system and it failed in TestEndianess, kindly 
let me know if at least Java version of Apache Arrow will run on Big Endian 
system.
Thanks,Sanjay 

> From: emkornfi...@gmail.com
> Date: Mon, 1 Aug 2016 09:23:51 -0700
> Subject: Re: Is LittleEndian system mandatory for using Apache Arrow ?
> To: dev@arrow.apache.org
> 
> Hi Sanjay,
> There was some discussion on the mailing list a little while ago about this
> [1].  We discussed making endianness part of the IPC metadata.  I don't
> think we've gotten far enough in the implementation to take action on the
> discussion though.
> 
> Right now I think the C++ code at least is endian agnostic, but it hasn't
> been tested against a big-endian system as far as I know.  As referenced in
> the prior thread (and I don't think its been changed), there is an explicit
> check for endianness when constructing vectors in Java
> 
> Cheers,
> Micah
> 
> 
> https://mail-archives.apache.org/mod_mbox/arrow-dev/201604.mbox/%3CCAK7Z5T_0T9hOa=mbJ=V30jc_Em169e=qjpyqnsblrakb8dm...@mail.gmail.com%3E
> 
> On Mon, Aug 1, 2016 at 8:57 AM, Sanjay Rao <getsanjay...@live.com> wrote:
> 
> > Hi,
> > I read that Integer representation is assumed to LittleEndian in Arrow,
> > does this mean we cannot use Arrow in Big Endian Systems like SPARC ?
> >
> > Thanks,Sanjay

Re: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Micah Kornfield 
Hi Sanjay,
There was some discussion on the mailing list a little while ago about this
[1].  We discussed making endianness part of the IPC metadata.  I don't
think we've gotten far enough in the implementation to take action on the
discussion though.

Right now I think the C++ code at least is endian agnostic, but it hasn't
been tested against a big-endian system as far as I know.  As referenced in
the prior thread (and I don't think its been changed), there is an explicit
check for endianness when constructing vectors in Java

Cheers,
Micah


https://mail-archives.apache.org/mod_mbox/arrow-dev/201604.mbox/%3CCAK7Z5T_0T9hOa=mbJ=V30jc_Em169e=qjpyqnsblrakb8dm...@mail.gmail.com%3E

On Mon, Aug 1, 2016 at 8:57 AM, Sanjay Rao  wrote:

> Hi,
> I read that Integer representation is assumed to LittleEndian in Arrow,
> does this mean we cannot use Arrow in Big Endian Systems like SPARC ?
>
> Thanks,Sanjay