Hi Bharath, Thanks for your inquiry. A small team is committed to Pig on Spark project and is actively working toward to the direction, while the pace of progression is slower than wished. This is mostly due to the constrained resources and sort of diversion of the initial contributors. We certainly welcome any sort of feedback especially contributions from the community.
We have a detailed design doc that's ready to shared in the community so that prospective contributors can take as a reference. While our initial objective is to achieve functional completeness, we are committed to enhancement and optimizations to make Pig on Spark run better and faster. As we are closer to the initial milestone, we will work on the user doc for consumption. Let me know if you have any questions. Thanks, Xuefu On Mon, Jun 8, 2015 at 6:09 PM, Bharath Ravi Kumar <[email protected]> wrote: > Hi, > > I'm looking for clarity on the future of the spark execution engine for > pig. While I've noticed activity on the spark branch on the pig git repo, > it hasn't been merged to trunk since the initial spork announcement. > Besides, it's not clear if it will continue to be maintained and enhanced > to exploit spark's capabilities to a greater extent (e.g. caching of RDD's > etc). I also see very little (up to date) user documentation on the setup. > I hence seek clarity on the future of the backend. Thanks. > > -Bharath >
