Joel Baranick created GOBBLIN-227: ------------------------------------- Summary: JobLauncherUtils.cleanTaskStagingData fails for jobs with forks Key: GOBBLIN-227 URL: https://issues.apache.org/jira/browse/GOBBLIN-227 Project: Apache Gobblin Issue Type: Bug Reporter: Joel Baranick
*Precondition:* Using Hocon configuration and have two forks configured. *Summary:* When {{AbstractJobLauncher}} calls {{JobLauncherUtils.cleanTaskStagingData}} it tries to lookup {{writer.staging.dir}} in the configuration and fails. *Details:* Hocon configuration doesn't allow the following config: {code:none} writer.staging.dir=/foo writer.staging.dir.0=/foo writer.staging.dir.1=/foo {code} Initially {{writer.staging.dir}} is of type String, but when the Hocon parser encounters {{writer.staging.dir.0}}, it decides that {{writer.staging.dir}} is now of type Object, overwriting the prior value with {{_\{"0": "/foo"\}_}}. The effective Hocon configuration is: {code:javascript} { "writer": { "staging": { "dir": { "0": "/foo", "1": "/foo" } } } } {code} Fork specific configuration uses the same config keys as regular configuration except the fork number is appended like: {{.1}}. The code that looks up fork specific configuration doesn't automatically fallback to regular configuration. For example, if the code is trying to find {{writer.staging.dir.0}} and it isn't configured, the job will fail. Then means that all forks must configure fork specific versions of {{writer.staging.dir}}. When {{AbstractJobLauncher}} calls {{JobLauncherUtils.cleanTaskStagingData}} it cleans up the based on the current job's configuration. Because of this, {{fork.branches}} is always set to {{1}}. The call to {{WriterUtils.getWriterStagingDir(state, numBranches, branchId)}} is make with {{numBranches=1}} and {{branchId=0}}. This results in the method looking for {{writer.staging.dir}}. Unfortunately, when using Hocon configuration the value {{writer.staging.dir}} doesn't exist and the job fails. -- This message was sent by Atlassian JIRA (v6.4.14#64029)