The PySpark community is aware of arrow, but certainly more reaching out to the Spark SQL devs could really be beneficial to get us all on the same page :)
On Thu, Dec 15, 2016 at 10:49 AM Julian Hyde <[email protected]> wrote: > I think someone should reach out to Matei and Shoumik, and see if they > would like to collaborate. Wes, would you like to do that? > > > > Also, reach out to the Spark community. Are they aware of Arrow? Are they > planning to use it, or are they developing an alternative? > > > > Julian > > > > > > > > > On Dec 13, 2016, at 10:23 AM, Wes McKinney <[email protected]> wrote: > > > > > > Looks like there will be a talk about Weld at Strata next Spring: > > > > > > > http://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/57646 > > > > > > It would be nice to see if there's any work that we can be doing in > > > Arrow to try to collaborate on such closely-related efforts. > > > > > > - Wes > > > > > > On Sat, Nov 26, 2016 at 3:53 PM, Wes McKinney <[email protected]> > wrote: > > >> I've always contended that building a sort of "runtime" for in-memory > > >> column expressions was a natural next step after hardening Arrow as a > > >> data structure and memory exchange mechanism. I'm hopeful that we'll > > >> see some cross-pollination between projects like Weld and Arrow to > > >> produce open source solutions that help drive more collaboration and > > >> consolidation around shared low-level needs for building analytical > > >> systems. We still have significant work to do to solidify Arrow as a > > >> standard -- anyone all reading can do to help work against > > >> fragmentation in data structures / memory representations would be > > >> very much appreciated! > > >> > > >> - Wes > > >> > > >> On Wed, Nov 23, 2016 at 11:35 AM, Donald Foss <[email protected]> > wrote: > > >>> I had that in my queue, and your watching and thoughts about it mean I > need > > >>> to watch it today ;) > > >>> > > >>> Thanks for the link to where you've played with abstraction before. I'm > > >>> going to fork a copy and do some playing myself. I'm very interested > in how > > >>> universal it can be, and a library to go from expression to universal > > >>> intermediate, them output the code in the language of one's choice. The > > >>> optimizations could be performed in the intermediate layer, the output > > >>> later, or both if there are specific code optimizations available after > > >>> translation. > > >>> > > >>> -Donald > > >>> > > >>> On Nov 22, 2016 2:31 PM, Julien Le Dem > > >>> wrote: > > > >
