I agree with you and while some of that has been remedied, I wouldn't say things are perfect. Scripting ability while still limited has better streaming support so you can have relations streamed Into a custom script executing in either map or reduce phase depending upon where it is placed.
If you want to glue together a bunch of map-reduce programs and pig script then the best option is to Invoke pig from your java program that also manages your M/R chain. Hadoop workflow system (Oozie) Is coming along which should make this better. For gluing together multiple pig programs the best there is "exec script.pig" which can be called from inside your script. However it is not a very neat solution since you would want to pass a bunch of things to the invoked script and also check for certain conditions to exists. So again a java program or a perl/python/ruby script managing your chain is a better option. Regards -...@nkur On 2/22/10 10:08 PM, "Ted Dunning" <ted.dunn...@gmail.com> wrote: Actually, no. I meant other programs written in pure Java. It used to be that the very restricted scripting ability of Pig made processing chains composed of Pig and map-reduce programs very brittle. In fact, just gluing together multiple Pig programs used to be very ugly. On Mon, Feb 22, 2010 at 12:42 AM, Ankur C. Goel <gan...@yahoo-inc.com>wrote: > About interfacing with other systems I assume you have an RDBMS in mind. -- Ted Dunning, CTO DeepDyve
