Re: HDFS ORC to Arrow Dataset

2021-09-09 Thread Manoj Kumar
On further analysis :- ==114164== Process terminating with default action of signal 6 (SIGABRT) ==114164==at 0x4AD118B: raise (raise.c:51) ==114164==by 0x4AB092D: abort (abort.c:100) ==114164==by 0x598D768: os::abort(bool) (in

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread QP Hou
Congrats Nic! On Thu, Sep 9, 2021 at 8:47 AM Neal Richardson wrote: > > On behalf of the Apache Arrow PMC, I'm happy to announce that Nic Crane > has accepted an invitation to become a committer on Apache Arrow. > > Welcome and thank you for your contributions! > > Neal

Re: [Question] Allocations along 64 byte cache lines

2021-09-09 Thread Jed Brown
Jorge Cardoso Leitão writes: > Yes, I expect aligned SIMD loads to be faster. > > My understanding is that we do not need an alignment requirement for this, > though: split the buffer in 3, [unaligned][aligned][unaligned], use aligned > loads for the middle and un-aligned (or not even SIMD) for

Re: [Question] Allocations along 64 byte cache lines

2021-09-09 Thread Jorge Cardoso Leitão
Thanks Yibo, Yes, I expect aligned SIMD loads to be faster. My understanding is that we do not need an alignment requirement for this, though: split the buffer in 3, [unaligned][aligned][unaligned], use aligned loads for the middle and un-aligned (or not even SIMD) for the prefix and suffix.

Re: [DISCUSS][Julia] How to restart at apache/arrow-julia?

2021-09-09 Thread Sutou Kouhei
Hi, Thanks for all comments about release schedule. Let's use release-on-demand approach based on arrow-datafusion's flow for the Julia Arrow implementation. Do we have more items to be discussed? Can we start voting? Thanks, -- kou In "Re: [DISCUSS][Julia] How to restart at

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Phillip Cloud
Congrats and welcome! On Thu, Sep 9, 2021, 16:16 Matthew Topol wrote: > Congrats! > > -Original Message- > From: Krisztián Szűcs > Sent: Thursday, September 9, 2021 3:47 PM > To: dev@arrow.apache.org > Subject: Re: [ANNOUNCE] New Arrow committer: Nic Crane > > Congrats Nic! > > On

RE: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Matthew Topol
Congrats! -Original Message- From: Krisztián Szűcs Sent: Thursday, September 9, 2021 3:47 PM To: dev@arrow.apache.org Subject: Re: [ANNOUNCE] New Arrow committer: Nic Crane Congrats Nic! On 2021. Sep 9., Thu at 21:31, Daniël Heres wrote: > Congrats, Nic! > > On Thu, Sep 9, 2021,

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Krisztián Szűcs
Congrats Nic! On 2021. Sep 9., Thu at 21:31, Daniël Heres wrote: > Congrats, Nic! > > On Thu, Sep 9, 2021, 21:20 Jorge Cardoso Leitão > wrote: > > > Congrats!! =) > > > > On Thu, Sep 9, 2021, 20:12 Micah Kornfield > wrote: > > > > > Congrats! > > > > > > On Thursday, September 9, 2021, Weston

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Daniël Heres
Congrats, Nic! On Thu, Sep 9, 2021, 21:20 Jorge Cardoso Leitão wrote: > Congrats!! =) > > On Thu, Sep 9, 2021, 20:12 Micah Kornfield wrote: > > > Congrats! > > > > On Thursday, September 9, 2021, Weston Pace > wrote: > > > > > Congratulations Nic! > > > > > > On Thu, Sep 9, 2021 at 7:43 AM

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Jorge Cardoso Leitão
Congrats!! =) On Thu, Sep 9, 2021, 20:12 Micah Kornfield wrote: > Congrats! > > On Thursday, September 9, 2021, Weston Pace wrote: > > > Congratulations Nic! > > > > On Thu, Sep 9, 2021 at 7:43 AM Antoine Pitrou > wrote: > > > > > > > > > Welcome on board Nic! > > > > > > > > > On Thu, 9 Sep

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Micah Kornfield
Congrats! On Thursday, September 9, 2021, Weston Pace wrote: > Congratulations Nic! > > On Thu, Sep 9, 2021 at 7:43 AM Antoine Pitrou wrote: > > > > > > Welcome on board Nic! > > > > > > On Thu, 9 Sep 2021 11:47:16 -0400 > > Neal Richardson wrote: > > > On behalf of the Apache Arrow PMC, I'm

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Weston Pace
Congratulations Nic! On Thu, Sep 9, 2021 at 7:43 AM Antoine Pitrou wrote: > > > Welcome on board Nic! > > > On Thu, 9 Sep 2021 11:47:16 -0400 > Neal Richardson wrote: > > On behalf of the Apache Arrow PMC, I'm happy to announce that Nic Crane > > has accepted an invitation to become a committer

Re: Arrow in HPC

2021-09-09 Thread David Li
I would support doing the work necessary to get UCX (or really any other transport) supported, even if it is a lot of work. (I'm hoping this clears the path to supporting a Flight-to-browser transport as well; a few projects seem to have rolled their own approaches but I think Flight itself

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Antoine Pitrou
Welcome on board Nic! On Thu, 9 Sep 2021 11:47:16 -0400 Neal Richardson wrote: > On behalf of the Apache Arrow PMC, I'm happy to announce that Nic Crane > has accepted an invitation to become a committer on Apache Arrow. > > Welcome and thank you for your contributions! > > Neal >

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Mauricio Vargas
CONGRATULATIONS !! THIS IS GREAT NEWS On Thu, Sep 9, 2021 at 12:40 PM Niranda Perera wrote: > Congrats Nic! :-) > > On Thu, Sep 9, 2021 at 12:17 PM Percy Camilo Triveño Aucahuasi < > percy.camilo...@gmail.com> wrote: > > > Happy for you Nic, congrats! > > > > On Thu, Sep 9, 2021 at 11:06 AM

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Niranda Perera
Congrats Nic! :-) On Thu, Sep 9, 2021 at 12:17 PM Percy Camilo Triveño Aucahuasi < percy.camilo...@gmail.com> wrote: > Happy for you Nic, congrats! > > On Thu, Sep 9, 2021 at 11:06 AM David Li wrote: > > > Congratulations Nic! > > > > -David > > > > On Thu, Sep 9, 2021, at 11:51, Keith Kraus

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Percy Camilo Triveño Aucahuasi
Happy for you Nic, congrats! On Thu, Sep 9, 2021 at 11:06 AM David Li wrote: > Congratulations Nic! > > -David > > On Thu, Sep 9, 2021, at 11:51, Keith Kraus wrote: > > Congrats Nic! > > > > On Thu, Sep 9, 2021 at 11:47 AM Neal Richardson < > neal.p.richard...@gmail.com> > > wrote: > > > > > On

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread David Li
Congratulations Nic! -David On Thu, Sep 9, 2021, at 11:51, Keith Kraus wrote: > Congrats Nic! > > On Thu, Sep 9, 2021 at 11:47 AM Neal Richardson > wrote: > > > On behalf of the Apache Arrow PMC, I'm happy to announce that Nic Crane > > has accepted an invitation to become a committer on

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Rok Mihevc
Congratulations Nic! :) Rok On Thu, Sep 9, 2021 at 5:51 PM Keith Kraus wrote: > > Congrats Nic! > > On Thu, Sep 9, 2021 at 11:47 AM Neal Richardson > wrote: > > > On behalf of the Apache Arrow PMC, I'm happy to announce that Nic Crane > > has accepted an invitation to become a committer on

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Keith Kraus
Congrats Nic! On Thu, Sep 9, 2021 at 11:47 AM Neal Richardson wrote: > On behalf of the Apache Arrow PMC, I'm happy to announce that Nic Crane > has accepted an invitation to become a committer on Apache Arrow. > > Welcome and thank you for your contributions! > > Neal >

Re: [ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Eduardo Ponce
Congratulations Nic! ~Eduardo On Thu, Sep 9, 2021 at 11:47 AM Neal Richardson wrote: > On behalf of the Apache Arrow PMC, I'm happy to announce that Nic Crane > has accepted an invitation to become a committer on Apache Arrow. > > Welcome and thank you for your contributions! > > Neal >

[ANNOUNCE] New Arrow committer: Nic Crane

2021-09-09 Thread Neal Richardson
On behalf of the Apache Arrow PMC, I'm happy to announce that Nic Crane has accepted an invitation to become a committer on Apache Arrow. Welcome and thank you for your contributions! Neal

Re: Arrow in HPC

2021-09-09 Thread Jed Brown
Yibo Cai writes: > HPC infrastructure normally leverages RDMA for fast data transfer among > storage nodes and compute nodes. Computation tasks are dispatched to > compute nodes with best fit resources. > > Concretely, we are investigating porting UCX as Flight transport layer. > UCX is a

Re: Arrow in HPC

2021-09-09 Thread Felipe Aramburu
UCX allow the user to use various transportation protocols without having to be aware of which protocol is being used under the hood. The image at the top of this page gives a good explanation of how this happens https://openucx.org/introduction/ . So the application can be agnostic if the

Re: Arrow in HPC

2021-09-09 Thread Keith Kraus
There's nothing stopping us from transmitting HTTP/2 or another binary protocol over UCX. You can think of UCX as a transport layer abstraction library which allows transparently taking advantage of things like RDMA over InfiniBand / RoCE, inter-process shared memory, TCP sockets, etc. The other

Re: [DISCUSS][Julia] How to restart at apache/arrow-julia?

2021-09-09 Thread Andrew Lamb
I also think release on demand is a good strategy. The primary reasons to do an arrow-rs release every 2 weeks were: 1. To have predictable cadence into downstream projects (e.g. datafusion and others) 2. Amortize the overhead associated with each release (the process is non trivial and the

Re: Arrow in HPC

2021-09-09 Thread Antoine Pitrou
Le 09/09/2021 à 12:34, Yibo Cai a écrit : Hi, We have some rough ideas of applying Flight in HPC (High Performance Computation). Would like to hear comments. HPC infrastructure normally leverages RDMA for fast data transfer among storage nodes and compute nodes. Computation tasks are

Arrow in HPC

2021-09-09 Thread Yibo Cai
Hi, We have some rough ideas of applying Flight in HPC (High Performance Computation). Would like to hear comments. HPC infrastructure normally leverages RDMA for fast data transfer among storage nodes and compute nodes. Computation tasks are dispatched to compute nodes with best fit