Re: ERROR 6017: Execution failed, while processing

[Alan Gates]

IMO Pig would do best to stay a data flow language and not take on  
control flow.  (I'm not sure all committers agree with me on this.)   
There's no lack of scripting languages out there that can be used for  
that (as seen on PIG-928) or frameworks like Piglet or Oozie.  But we  
could still do C preprocessor style stuff.  We've taken the first step  
of parameter substitution.  If we took two more steps, %include and  
arguments for parameter substitution (that is the ability to say  
%declare a(x, y, k1, k2) join x by k1, y by k2) , we would avoid  
control flow while still adding a lot of benefit.  Full bore data  
pipelines will always need some kind of work flow system to manage  
their various Pig components.  But it would be nice if for medium  
sized jobs (say 500 lines of Pig Latin) Pig was still usable without  
the added complexity of workflow.  If we do this in steps, include  
now, arguments for %declare later, I think that's fine.  I'd just like  
to see a plan for where we're going with it.

Alan.

On Mar 15, 2010, at 5:28 PM, Dmitriy Ryaboy wrote:

Alan -- yeah, right now we use the rather brittle approach of naming
conventions to do this. Something more template/macro-like would be  
better.
Of course something like Piglet, or equivalents in other languages,  
can
obviate the need for these constructs, and I am not entirely sure  
functions,
loops, etc are something we want to get into reinventing. I guess the
question becomes whether we want Pig Latin to be a first-class  
language that
programmers write code in directly, or if we shift focus on building  
out the
tooling for generating Pig scripts, and Pig Latin becomes something  
you drop
into for one-offs.

-D

On Mon, Mar 15, 2010 at 4:02 PM, Alan Gates <ga...@yahoo-inc.com>  
wrote:

In your example below how would the results of these load functions  
be
accessed in your main script?

I certainly see the value of #include plus functions (or #define if  
you
prefer).  Without functions though you'll have namespace clashes (any
relation names used in the imported files will be visible to other  
imported
files and to the main script) and the user will have to know the  
name of
input and output relations for the imported files so he can use it
subsequently in his script.  For example if you had a pig script that
implemented a certain type of join:

RETURN = join INPUT1 by $0, INPUT2 by $0

Now the user has to know that INPUT1 and INPUT2 must be the names  
of his
input relations and that the output relation will be named RETURN.   
This is
also limited because we can't define which key(s) to do the join  
on.  To
make this useful we're going to want a macro or function ability so  
we can
pass in names of inputs and other parameters (like which keys to  
join on),
control the names of results, and have variable scoping.

That said, I'm all for it.  I think it would make Pig must more  
usable.

Alan.




On Mar 15, 2010, at 2:58 PM, Dmitriy Ryaboy wrote:

Alan, this would be quite useful, as essentially this would allow
developers
to create functions by writing them into separate pig scripts and
combining
them as necessary.

For example we have code that auto-generates load statements with  
fairly
complex schemas based on protocol buffers (see

http://www.slideshare.net/hadoopusergroup/twitter-protobufs-and-hadoop-hug-021709
).
It would be very handy to be able to say something like

#include common_jars.pig
#include load_tweets.pig
#include load_users.pig

#include filter_nonenglish_tweets.pig
#include geomap_users.pig

.. etc ..

-D

On Mon, Mar 15, 2010 at 2:23 PM, Alan Gates <ga...@yahoo-inc.com>  
wrote:


On Mar 12, 2010, at 10:36 AM, hc busy wrote:



Is there any work towards something like C languages '#include'  
in Pig?
My
large pig script is actually developed separately in several  
smaller pig
files. Individually the pig files do not run because they depend  
on
previous
scripts, but logically they are separate because each step does
something
different.

Currently the only thing existing along these lines is the exec  
command

in grunt.  I don't think we're opposed to a #include  
functionality, we
just
haven't done it.  However, given that Pig doesn't have function  
calls,
and
presumably each Pig Latin script is self contained, it isn't  
clear to me
how
useful it will be.

Alan.