I just tested a > 4k script. Pig 0.8.1 takes 3m59.592s and Pig 0.9.1 takes 0m56.553s to compile (use -c option in command line). Seems Pig 0.9 compiles much faster in large queries. I also did some tests on small queries before, Pig 0.9 is about 15% slower than Pig 0.8. The slow down is less significant since compilation time only takes a fraction of total runtime for those queries. If there are some cases Pig 0.9 lag behind a lot, please share with me.
Thanks, Daniel On Tue, Oct 25, 2011 at 9:20 AM, Thejas Nair <[email protected]> wrote: > This is not expected. Daniel had some experiments done with large queries > (> 2k lines) and the parsing was actually faster with them. > There was some small slow down for smaller queries. > If you can replicate with a script that you can share, that would be great. > > -Thejas > > > > On 10/21/11 3:37 PM, Jonathan Coveney wrote: > >> Pig is version 0.9. I have a script that under version 0.8, begins running >> quickly...but under version 0.9, it takes 10 minutes to parse (I did >> -debug >> ALL and it builds a big old AST). I am curious if there is anything that >> could cause this difference? It is a very long script (almost 2k lines!), >> but I'm wondering why the difference. >> >> I'm going to try and replicate with a script I can share, and then run it >> against trunk....just curious if anyone has seen this before. >> >> >
