I've added a wiki page for a "Pig on Storm Proposal" at https://cwiki.apache.org/confluence/display/PIG/Pig+on+Storm+Proposal
I've included a primer on Storm (and Trident) as well as some of the challenges I foresee. Please read though my proposal and let me know what your thoughts are. On Wed, Jul 24, 2013 at 2:36 PM, Alan Gates <ga...@hortonworks.com> wrote: > This sounds exciting. The next question is how do you plan to do it? > Would a physical plan be translated to a Storm job (or jobs)? Would it > need a different physical plan? Or would you just have the connection at > the language layer and all the planning separate? Do you envision needing > extensions/changes to the language to support Storm? Feel free to add a > page to Pig's wiki with your thoughts on an approach. > > Alan. > > On Jul 23, 2013, at 9:52 AM, Pradeep Gollakota wrote: > > > Hi Pig Developers, > > > > I wanted to reach out to you all and ask for you opinion on something. > > > > As a Pig user, I have come to love Pig as a framework. Pig provides a > great > > set of abstractions that make working with large datasets easy. Currently > > Pig is only backed by hadoop. However, with the new rise of Twitter Storm > > as a distributed real time processing engine, Pig users are missing out > on > > a great opportunity to be able to work with Pig in Storm. As a user of > Pig, > > Hadoop and Storm, and keeping with the Pig philosophy of "Pigs live > > anywhere," I'd like to get your thoughts on starting the implementation > of > > a Pig backend for Storm. > > > > Thanks > > Pradeep > >
