Matei Zaharia just spoke at the AMPlab seminar [1], and showed a couple of slides about Weld. In the video of the day [2], his talk starts at 4:05:00, and he starts talking about Weld at 4:28:30.
The essence is an intermediate language for row-level expressions, with the ability to do limited iteration, with the goal of making it easier to pass data between UDFs written in different languages. Sounds familiar? I would presume that an implementation of the language would be strongly tied to a memory format. Or maybe it allows multiple possible implementations, one of which would be Arrow in Java. The slide listed Pandas as one of the supported front ends, so I wondered if Wes knew something about the project. I have been thinking of doing something similar in the Calcite / Drill / Arrow world. In Calcite we have RexNodes as an expression language, and we have a Java code generator that can target data represented as Java arrays, and another variant that can target data represented as Java structs. Drill of course has a code generator that can target data in Arrow. I have been thinking for a while of abstracting the code generators so that the person implementing, say, the Filter+Project for “select x + y … where x > 5” doesn’t have to get their hands dirty with code generation. There are a lot of optimizations to be done, e.g. remembering that you’ve already made sure that x is not null. Julian [1] https://amplab.cs.berkeley.edu/endofproject/ <https://amplab.cs.berkeley.edu/endofproject/> [2] https://youtu.be/KAacs9jYPHU <https://youtu.be/KAacs9jYPHU> > On Nov 19, 2016, at 4:31 AM, Donald Foss <donald.f...@gmail.com> wrote: > > Did you find that at https://cs.stanford.edu/~matei/? > <https://cs.stanford.edu/~matei/?> That’s the only thing I can find via > Google about it. Do you have more detail or a link to the paper itself? I > get the feeling that it is not yet fully complete despite 21 November > camera-ready CIDR 2017 deadline. > > For those who aren’t familiar with CIDR, it is a conference that occurs every > other year. This year’s agenda/program may be found at > http://cidrdb.org/cidr2017/program.html > <http://cidrdb.org/cidr2017/program.html>. CIDR is not an acronym for > network subnet masks—the first thing I thought of, Classless Inter Domain > Routing, but Conference on Innovative Data Systems Research, which focuses > primarily on systems. I hate to admit this, but I’m unfamiliar with the > conference, however that appears that it is because I’ve been out of academia > for far too long, and this conference seems to be the presentation of quite a > few interesting papers. Just judging by title, a poor, yet humorous judge > indeed, I like: > - “Dependency-Driven Analytics: A Compass for Uncharted Data Oceans” (Donald > - Why just data lakes when you can have data oceans?) > - “My Weak Consistency is Strong” (Donald - Great title, reminds me of Star > Wars and the “Force”) > - “SPOOF: Sum-Product Optimization and Operator Fusion for Large-Scale > Machine Learning” (Donald - Another brilliant backronym.) > > The Weld paper is the last paper to be presented on 10 January 2017 between > 2:30 and 4:05 (UTC-8). > > On a side note, looking down that page a little, I love the title of the last > paper in 2016, Yggdrasil: An Optimized System for Training Deep Decision > Trees at Scale > <https://cs.stanford.edu/~matei/papers/2016/nips_yggdrasil.pdf>. When I see > Yggdrasil, the first thing I think of is a really big tree and Norse > mythology. It’s a great name. I’m going to read some of his other papers > this weekend. > > Donald Foss > donald.f...@gmail.com > ------ __o > ----_`\<,_ > ---(_)/ (_) > > The information in this email is confidential and may be legally privileged. > It is intended solely for the addressee. Access to this e-mail by anyone else > is unauthorized. > >> On Nov 18, 2016, at 4:42 PM, Julian Hyde <jh...@apache.org> wrote: >> >> Anyone know anything about Matei Zaharia’s Weld project? >> >> • S. Palkar, J. Thomas, A. Shanbhag, H. Pirk, M. Schwarzkopf, S. >> Amarasinghe and M. Zaharia. Weld: A Common Runtime for High Performance Data >> Analytics, to appear at CIDR 2017. >> >> It seems to have similar goals to Arrow. >> >> Julian >> >
