The issue was that trying to be non-backend specific forced us to write abstractions for all of Hadoop's features rather than use them directly. For example, we had to have an abstract notion of a file system rather than use HDFS, an abstract notion of input rather than use InputFormats. This caused efficiency issues for the runtime system and made it much harder for developers to understand what was going on in the code. Given that no other implementations existed, it did not make sense to pay this price.
Alan. On Dec 14, 2011, at 8:13 AM, Tharindu Mathew wrote: > Hi Gianmarco, > > Thanks for the response... > > I do understand the Pig Latin is not coupled with anything. Hence, the > question for use in another framework. > > I was referring to what you said secondly. It would be great to run Pig as > a Pig Latin Executor without Hadoop. > > Why was such an extension point a burden? It seems like the right thing to > do in terms of design. > > On Wed, Dec 14, 2011 at 7:29 PM, Gianmarco De Francisci Morales < > [email protected]> wrote: > >> Hi, >> Pig Latin (the language) is not coupled to Hadoop and can be used on any >> other framework. >> Pig (the system) however, is tightly coupled to Hadoop. >> Extension point existed but were removed because they were a major burden. >> That said, it is theoretically possible to build another Pig Latin executor >> that runs on a different framework. >> >> Cheers, >> -- >> Gianmarco >> >> >> >> On Wed, Dec 14, 2011 at 13:17, Tharindu Mathew <[email protected]> >> wrote: >> >>> I understand that Pig Latin is a data flow language. In that sense it >>> should be theoretically possible to execute Pig Latin in any framework >>> though currently and it is meant to be executed in a Hadoop enviornment. >>> How hard would it be to switch Pig Latin to run on a different framework? >>> Are there any extension points for this if at all or is Pig Latin is >>> tightly coupled to Hadoop? >>> >>> -- >>> Regards, >>> >>> Tharindu >>> >>> blog: http://mackiemathew.com/ >>> >> > > > > -- > Regards, > > Tharindu > > blog: http://mackiemathew.com/
