Would love to have a discussion of what are the next set of technical things to
work on. To begin, here is a summary of what has gone into the PRs recently.
#1: A large amount of focus has been on the query execution and optimization
methods. The key PRs here are PR #113, Issue #69/PR172. These added, as noted
in the PR discussion, an aggressive semi-join technique. A paper describing
this work will appear in VLDB 2017 at the end of the summer. The paper is up at
http://www.vldb.org/pvldb/vol10/p889-zhu.pdf. There are continual changes to
the optimizer that largely Jianqiao fixes. AFAIK, there are no major features
that are currently planned on this component. Please chime in if this is not
#2: There is work on adding support for union/intersection operator lead by our
newest committer, Tianrun Li. This originated from him putting GRAIL on top of
Quickstep. GRAIL is a way to map BSP-style graph analytics on a relational
core, and was published (as a separate project) two years ago. See
#3: The big active feature right now is on the distributed version, lead by
Zuyu. (Zuyu please step in to correct the notes below as necessary.) In various
discussions and PRs the plan he has put forward is to:
Work on the partitioning in Quickstep as that is critical in the distributed
version. There was robust discussion on this around Issue 73/PR 217
Moving data around nodes. The idea here is to use the design that we have with
blocks having unique id, and letting a node pull a block from a remote node
when it can’t find a block. This would use the GPRC mechanism that we have. I
think it would be good to have a discussion of this design and get input from
everyone who can help. This is clearly a big feature. Zuyu: Do you want to
summarize your design in more detail when you have worked it out?
#4: Most of the Wisconsin folks are off for the summer, so I think things are
going to slow down a bit on the development pace.
As a note, we have been slack in having discussions of design on the dev list.
I apologize for my part on this front. Having a discussion on the dev list is a
great idea, and healthy for all in the long-run. So please feel free to chime
in with your views and feel free to make a design proposal for specific parts
that you think need work on.
Happy Memorial Day everyone!