Will-Lo commented on code in PR #3822:
URL: https://github.com/apache/gobblin/pull/3822#discussion_r1385478350
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/spec/JobExecutionPlan.java:
##########
@@ -112,10 +112,10 @@ private static JobSpec buildJobSpec(FlowSpec flowSpec,
Config jobConfig, Long fl
// job names are assumed to be unique within a dag.
int hash = flowInputPath.hashCode();
jobName = Joiner.on(JOB_NAME_COMPONENT_SEPARATION_CHAR).join(flowGroup,
flowName, jobName, edgeId, hash);
- // jobNames are commonly used as a directory name, which is limited to
255 characters
+ // jobNames are commonly used as a directory name, which is limited to
255 characters (account for potential prefixes added/file name lengths)
if (jobName.length() >= MAX_JOB_NAME_LENGTH) {
- // shorten job length to be 128 characters (flowGroup) + (hashed)
flowName, hashCode length
- jobName =
Joiner.on(JOB_NAME_COMPONENT_SEPARATION_CHAR).join(flowGroup,
flowName.hashCode(), hash);
+ // shorten job length but make it uniquely identifiable in multihop
flows or concurrent jobs, max length 139 characters (128 flow group + hash)
+ jobName =
Joiner.on(JOB_NAME_COMPONENT_SEPARATION_CHAR).join(flowGroup,
jobName.hashCode());
Review Comment:
I think it will only be done on long flow names so it should be fine, I
wanted to get a hash of the jobName and not reuse the prior hash so that it can
cover scenarios where there is the same input path in a multi hop flow with
multiple edges
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
