We recently ran into an issue where our code to serialize a pipeline's configuration was throwing an exception because one of the key/values in the config was too big (65k characters). We found this key/value was 'crunch.planner.dotfile' which is included in the pipelines config from Crunch.
My question is why does Crunch provide this value into the config object? Crunch saves the dotfile string in the MRExecutor context[1] and I don't think any pipeline would need this at runtime. It also seems like there are no references to this config value anywhere within Crunch other then to write the value into the config object. [1] - https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-core/src/main/java/org/apache/crunch/impl/mr/plan/DotfileUtills.java#L139-L140
