Hi Bryan, Josh, IIRC this comes from the original dotfile jobplan implementation. I kept it for backward compatible. You can see that only the "jobplan" (e.g. the original/main plan) is stored in the Configuration.
+Gabriel i am not sure I remember the original intent to have the jobplan stored in the Configuration? On Wed, Jul 1, 2015 at 11:02 PM, Josh Wills <[email protected]> wrote: > +Christian > > I'm not sure what the intent was there-- Christian? > > J > > On Wed, Jul 1, 2015 at 12:29 PM, Bryan Baugher <[email protected]> wrote: > >> We recently ran into an issue where our code to serialize a pipeline's >> configuration was throwing an exception because one of the key/values in >> the config was too big (65k characters). We found this key/value was >> 'crunch.planner.dotfile' which is included in the pipelines config from >> Crunch. >> >> My question is why does Crunch provide this value into the config object? >> >> Crunch saves the dotfile string in the MRExecutor context[1] and I don't >> think any pipeline would need this at runtime. It also seems like there are >> no references to this config value anywhere within Crunch other then to >> write the value into the config object. >> >> [1] - >> https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/DotfileUtills.java#L139-L140 >> >> >
