hc busy
Tue, 16 Mar 2010 17:31:36 -0700
ahhh, that %declare a(...) is exactly what would help with the variable name problem. Because otherwise, it's like a register language where all function.pig files take parameters named a1,a2,a3,a4,a5,... and before #include'ing a pig file, the caller sets a1, a2, a3, a4,...; and function.pig will return values in a1, a2, a3 after it's done.
Personally I look at pig as a nice high-level language to type into (as opposed to some assembly language that we'll eventually compile sql into). In terms of optimization, I think if we stay functional, it is essentially a data flow language (except with the possibility of infinitely long pipes) If there will be no flow control in pig, then the other way to go about this is to introduce JIT type technology. Basically the external command streams in commands (like my typing into grunt), but the system continuously optimizes the execution and caching of intermediate results. Or else, hey, or else we could just introduce "GOTO" statements. ;-) On Tue, Mar 16, 2010 at 3:08 PM, Alan Gates <ga...@yahoo-inc.com> wrote: > IMO Pig would do best to stay a data flow language and not take on control > flow. (I'm not sure all committers agree with me on this.) There's no lack > of scripting languages out there that can be used for that (as seen on > PIG-928) or frameworks like Piglet or Oozie. But we could still do C > preprocessor style stuff. We've taken the first step of parameter > substitution. If we took two more steps, %include and arguments for > parameter substitution (that is the ability to say %declare a(x, y, k1, k2) > join x by k1, y by k2) , we would avoid control flow while still adding a > lot of benefit. Full bore data pipelines will always need some kind of work > flow system to manage their various Pig components. But it would be nice if > for medium sized jobs (say 500 lines of Pig Latin) Pig was still usable > without the added complexity of workflow. If we do this in steps, include > now, arguments for %declare later, I think that's fine. I'd just like to > see a plan for where we're going with it. > > Alan. > > > On Mar 15, 2010, at 5:28 PM, Dmitriy Ryaboy wrote: > > Alan -- yeah, right now we use the rather brittle approach of naming >> conventions to do this. Something more template/macro-like would be >> better. >> Of course something like Piglet, or equivalents in other languages, can >> obviate the need for these constructs, and I am not entirely sure >> functions, >> loops, etc are something we want to get into reinventing. I guess the >> question becomes whether we want Pig Latin to be a first-class language >> that >> programmers write code in directly, or if we shift focus on building out >> the >> tooling for generating Pig scripts, and Pig Latin becomes something you >> drop >> into for one-offs. >> >> -D >> >> On Mon, Mar 15, 2010 at 4:02 PM, Alan Gates <ga...@yahoo-inc.com> wrote: >> >> In your example below how would the results of these load functions be >>> accessed in your main script? >>> >>> I certainly see the value of #include plus functions (or #define if you >>> prefer). Without functions though you'll have namespace clashes (any >>> relation names used in the imported files will be visible to other >>> imported >>> files and to the main script) and the user will have to know the name of >>> input and output relations for the imported files so he can use it >>> subsequently in his script. For example if you had a pig script that >>> implemented a certain type of join: >>> >>> RETURN = join INPUT1 by $0, INPUT2 by $0 >>> >>> Now the user has to know that INPUT1 and INPUT2 must be the names of his >>> input relations and that the output relation will be named RETURN. This >>> is >>> also limited because we can't define which key(s) to do the join on. To >>> make this useful we're going to want a macro or function ability so we >>> can >>> pass in names of inputs and other parameters (like which keys to join >>> on), >>> control the names of results, and have variable scoping. >>> >>> That said, I'm all for it. I think it would make Pig must more usable. >>> >>> Alan. >>> >>> >>> >>> >>> On Mar 15, 2010, at 2:58 PM, Dmitriy Ryaboy wrote: >>> >>> Alan, this would be quite useful, as essentially this would allow >>> >>>> developers >>>> to create functions by writing them into separate pig scripts and >>>> combining >>>> them as necessary. >>>> >>>> For example we have code that auto-generates load statements with fairly >>>> complex schemas based on protocol buffers (see >>>> >>>> >>>> http://www.slideshare.net/hadoopusergroup/twitter-protobufs-and-hadoop-hug-021709 >>>> ). >>>> It would be very handy to be able to say something like >>>> >>>> #include common_jars.pig >>>> #include load_tweets.pig >>>> #include load_users.pig >>>> >>>> #include filter_nonenglish_tweets.pig >>>> #include geomap_users.pig >>>> >>>> .. etc .. >>>> >>>> -D >>>> >>>> On Mon, Mar 15, 2010 at 2:23 PM, Alan Gates <ga...@yahoo-inc.com> >>>> wrote: >>>> >>>> >>>> On Mar 12, 2010, at 10:36 AM, hc busy wrote: >>>>> >>>>> >>>>> >>>>> Is there any work towards something like C languages '#include' in >>>>>> Pig? >>>>>> My >>>>>> large pig script is actually developed separately in several smaller >>>>>> pig >>>>>> files. Individually the pig files do not run because they depend on >>>>>> previous >>>>>> scripts, but logically they are separate because each step does >>>>>> something >>>>>> different. >>>>>> >>>>>> Currently the only thing existing along these lines is the exec >>>>>> command >>>>>> >>>>>> in grunt. I don't think we're opposed to a #include functionality, >>>>> we >>>>> just >>>>> haven't done it. However, given that Pig doesn't have function calls, >>>>> and >>>>> presumably each Pig Latin script is self contained, it isn't clear to >>>>> me >>>>> how >>>>> useful it will be. >>>>> >>>>> Alan. >>>>> >>>>> >>>>> >>> >