Hi, Correct me if I'm wrong, but it doesn't seem to me that allocating size of 4KB multiple would equivalently mean that the starting address of data is also a multiple of 4KB.
For example, in the Java implementation when an ArrowBuf is allocated we later slice it to allocate both a data buffer and validity buffer. Until now, the allocation of data buffer slices the ArrowBuf from 0 to dataBufferSize. We would like to support slicing from bitsUntilPageAlignment to dataBufferSize + bitsUntilPageAlignment. That way the dataBufferStartAddress() would be a multiple of 4 KB. That's what we would like to achieve. Would it help if I opened a PR with our workaround for page-alignment in Java? On Sat, Mar 30, 2019 at 10:07 PM Wes McKinney <wesmck...@gmail.com> wrote: > Thank you, this is helpful. > > On the page alignment issue, is it enough to ensure that each buffer > (according to the IPC protocol) has a size with a certain multiple? > This might be something we can make configurable on the IPC write path > on the C++ side. There is nothing fundamentally illegal in the > protocol about writing extra padding. Currently generally only pad to > 8-byte or 64-byte boundaries. > > - Wes > > On Sat, Mar 30, 2019 at 10:20 AM Dimitris Lekkas <dlekk...@gmail.com> > wrote: > > > > Hi, > > > > Basically, fletcher has emphasized on reading an Arrow schema and > > generating a > > hardware interface along with a template for the functional part of the > > accelerator. > > That way the FPGAs are aware for arrow-backed data. > > > > On the other hand, we do not work on enabling FPGAs to interpret arrow > > formats but > > we rather want to feed them with just the "pure" data (I am referring to > > arrow columns). > > Relying on a generated hardware-interface is a different approach because > > on our case > > we do not want FPGAs to be aware that Arrow exists. We do not generate > any > > code > > related to the specialized hardware. > > > > For the above reason, we wanted columns to be page-aligned in order to > > enable > > fast DMA transfers on them. > > > > Regards > > > > > > On Fri, Mar 29, 2019 at 6:59 PM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > hi Dimitris, > > > > > > Could you comment on how your work is similar to this earlier project > > > > > > https://github.com/johanpel/fletcher > > > > > > There's a risk of some fragmentation of efforts if we end up with > > > multiple third parties doing this work in isolation; I hope that some > > > collaborations are able to take place inside Apache Arrow > > > > > > - Wes > > > > > > On Fri, Mar 29, 2019 at 4:27 AM Dimitris Lekkas <dlekk...@gmail.com> > > > wrote: > > > > > > > > Hello, > > > > > > > > 1) Wes the PR is already submitted and awaiting review. > > > > 2) Antoine, you will have feedback from us on the issue shortly. > > > > 3) Melik, we plan to contribute back to the Arrow community. > > > Particularly, > > > > we are working on tweaking plasma to support allocations enabling > fast > > > > interaction with FPGAs. I will make a proposal for changes > pertaining to > > > > plasma, so you will hear from me soon. Additionally, we have enabled > > > > page alignment of data buffers in Java implementation so if that > sounds > > > > interesting to you we could discuss on that as well. > > > > > > > > Regards, > > > > > > > > Dimitris > > > > > > > > On Fri, Mar 29, 2019 at 6:31 AM Melik-Adamyan, Areg < > > > > areg.melik-adam...@intel.com> wrote: > > > > > > > > > Hi Chris, > > > > > > > > > > Do you have plans to contribute the infrastructure part back to the > > > > > community so the others can build hybrid pipelines? > > > > > > > > > > -----Original Message----- > > > > > From: Wes McKinney [mailto:wesmck...@gmail.com] > > > > > Sent: Thursday, March 28, 2019 10:51 AM > > > > > To: dev@arrow.apache.org > > > > > Cc: ch...@inaccel.com > > > > > Subject: Re: FPGA support for Apache Arrow > > > > > > > > > > hi Chris -- in addition to commenting on ARROW-2447, do you want to > > > submit > > > > > a PR to add yourself to Powered By? > > > > > > > > > > https://github.com/apache/arrow/blob/master/site/powered_by.md > > > > > > > > > > best > > > > > Wes > > > > > > > > > > On Thu, Mar 28, 2019 at 6:01 AM Antoine Pitrou <anto...@python.org > > > > > wrote: > > > > > > > > > > > > > > > > > > Hello Chris, > > > > > > > > > > > > Le 28/03/2019 à 11:56, ch...@inaccel.com a écrit : > > > > > > > > > > > > > > I would like to let you know that we have released our new > version > > > > > > > <https://docs.inaccel.com/latest/manager/examples/> of our > > > > > > > framework that allows FPGAs to talk to any other framework > under > > > Arrow. > > > > > > > > > > > > > > That means that any Arrow-based framework can utilize now > FPGAs in > > > > > > > order to speedup their applications (e.g. ML, analytics, > > > compression, > > > > > etc.). > > > > > > > > > > > > Since you are doing heterogenous computing with Arrow, it would > be > > > > > > extra nice if you / your engineers could take some time to give > > > > > > feedback on this issue: > > > > > > https://issues.apache.org/jira/browse/ARROW-2447 > > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine. > > > > > > > > >
