Zihan Li created GOBBLIN-1561:
---------------------------------
Summary: Improve error message when flow compilation fails
Key: GOBBLIN-1561
URL: https://issues.apache.org/jira/browse/GOBBLIN-1561
Project: Apache Gobblin
Issue Type: Improvement
Reporter: Zihan Li
When flow compilation fails, it prints a large list of irrelevant configs that
are missing from different edges. The reason for this is that the pathfinding
does not know what the intended path was, so it doesn't know which edges were
actually missing required config. For example, this create flow command:
{code:java}
curli --dv-auth SELF --fabric ei-ltx1 "d2://sharedFlowconfigsV2" -X POST -H
'X-RestLi-Method: create' -H 'X-RestLi-Protocol-Version: 2.0.0' --data '{"id":
{"flowName": "test123", "flowGroup": "test123"}, "properties":
{"gobblin.flow.sourceIdentifier": "faro", "gobblin.flow.destinationIdentifier":
"faro", "user.to.proxy": "gobblintest",
"gobblin.flow.input.dataset.descriptor.path": "/tmp/gaas-cli-test/a",
"gobblin.flow.output.dataset.descriptor.path": "/tmp/gaas-cli-test/a",
"gobblin.flow.input.dataset.descriptor.partition.type": "none",
"gobblin.flow.output.dataset.descriptor.partition.type": "none",
"report.job.progress": "true", "flow.applyRetention": "true",
"flow.applyInputRetention": "true",
"gobblin.retention.newestK.versions.retained": "0", "dataset.datetimePattern":
"yyyy/MM/dd", "copy.date.pattern": "yyyy/MM/dd"}, "templateUris": "FS:///"}'
{code}
Fails because the "version.finder" property is unresolved for the
faro_faro_retention edge, but that info is buried in this large error message:
{code:java}
"Flow was not compiled successfully. Compilation errors encountered: [Error
compiling edge
faro_azureLnkdKtwoAdls2_inter-cluster-datetime-file-based-gen2-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 3: Could
not resolve substitution to a value: ${azure.container}, Error compiling edge
faro_azureLnkdAdls2_single-hop-selfserve-unixts-file-based-gen2-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 3: Could
not resolve substitution to a value: ${azure.container}, Error compiling edge
faro_azureScusAdls2-data_inter-cluster-file-based-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 10: Could
not resolve substitution to a value: ${destination.adl.token.type}, Error
compiling edge
faro_azureLnkdArchAdls2_inter-cluster-file-based-gen2-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 3: Could
not resolve substitution to a value: ${azure.container}, Error compiling edge
faro_azureLnkdAdls2_single-hop-selfserve-timeaware-file-based-gen2-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 3: Could
not resolve substitution to a value: ${azure.container}, Error compiling edge
faro_faro_retention:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 67: Could
not resolve substitution to a value: ${version.finder}, No path found from
source: faro and destination: faro, Error compiling edge
faro_azureLnkdArchAdls2_inter-cluster-datetime-file-based-gen2-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 3: Could
not resolve substitution to a value: ${azure.container}, Error compiling edge
faro_azureLnkdAdls2_single-hop-selfserve-glob-file-based-gen2-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 3: Could
not resolve substitution to a value: ${azure.container}, Error compiling edge
faro_azureLnkdKtwoAdls2_inter-cluster-file-based-gen2-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 3: Could
not resolve substitution to a value: ${azure.container}, Error compiling edge
faro_azureLnkdKtwoAdls2_inter-cluster-regex-file-based-gen2-copy-azure:
com.typesafe.config.ConfigException$UnresolvedSubstitution: Reader: 3: Could
not resolve substitution to a value: ${azure.container}]"{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
