[
https://issues.apache.org/jira/browse/GOBBLIN-1952?focusedWorklogId=889332&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-889332
]
ASF GitHub Bot logged work on GOBBLIN-1952:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 07/Nov/23 19:53
Start Date: 07/Nov/23 19:53
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #3822:
URL: https://github.com/apache/gobblin/pull/3822#discussion_r1385477354
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/spec/JobExecutionPlan.java:
##########
@@ -112,10 +112,10 @@ private static JobSpec buildJobSpec(FlowSpec flowSpec,
Config jobConfig, Long fl
// job names are assumed to be unique within a dag.
int hash = flowInputPath.hashCode();
jobName = Joiner.on(JOB_NAME_COMPONENT_SEPARATION_CHAR).join(flowGroup,
flowName, jobName, edgeId, hash);
- // jobNames are commonly used as a directory name, which is limited to
255 characters
+ // jobNames are commonly used as a directory name, which is limited to
255 characters (account for potential prefixes added/file name lengths)
Review Comment:
1. No because then it would create a different work directory per execution
which would lead to jars not being cached etc.
2. Not sure but I believe it was decided a while back that it would be more
unique in the multihop/parallel job setting than edgeId or flowName, as you can
have one edge map to multiple jobs.
Issue Time Tracking
-------------------
Worklog Id: (was: 889332)
Time Spent: 1h 10m (was: 1h)
> GaaS JobNames with long lengths cause issues with HDFS folders
> --------------------------------------------------------------
>
> Key: GOBBLIN-1952
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1952
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: William Lo
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Gobblin-as-a-Service creates jobnames using flowgroups flownames, edges, and
> jobnames from the template. However, this tends to create a very long string
> which then causes issues in Gobblin job when creating files that use the
> jobname to create working directories or state stores. Although there has
> been previous code that shortens job name lengths, we want to further
> increase this by being more aggressive with the maximum length of the jobname
> to reduce the odds of exceeding 255 chars (max length of HDFS component)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
