Macros for code that appears more than once. Split script into multiple scripts and schedule with dependency in azkaban.
On Wednesday, March 5, 2014, Christopher Petrino <[email protected]> wrote: > Thank you Dan and Jacob. I am currently on 0.11.1 but open to upgrading. > Over the last few weeks I developed a Pig script that has become a little > over a 150 lines long and I was hoping I could find a way to modularize the > script. I was going to follow something like mentioned in this link: > > http://stackoverflow.com/questions/7557528/how-to-call-a-pig-script-within-another-pig-scriptbut > was curious what the community has been doing. Thank you for your > input! > > -Chris > > > On Wed, Mar 5, 2014 at 10:55 AM, Jacob Perkins > <[email protected]<javascript:;> > >wrote: > > > Christopher, > > > > You might consider breaking it into one or more reusable macros. What > > version of pig are you using? > > > > For complicated scripts, especially if you didn't write them, you might > > want to take a look at lipstick, https://github.com/Netflix/Lipstick > > It allows you to visualize the dag and clearly shows what logical > > operators map to map-reduce jobs. It could be a starting point for > managing > > complexity at least. > > > > --jacob > > @thedatachef > > > > > > On Mar 5, 2014, at 7:33 AM, Christopher Petrino > > <[email protected]<javascript:;>> > wrote: > > > > > Hi all, what is everyone's approach for managing a Pig scripts that has > > > become very long? What is your best way to break it up into smaller > > pieces? > > > > > -- Russell Jurney twitter.com/rjurney [email protected] datasyndrome.com
