[
https://issues.apache.org/jira/browse/GOBBLIN-2032?focusedWorklogId=912757&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-912757
]
ASF GitHub Bot logged work on GOBBLIN-2032:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 03/Apr/24 02:07
Start Date: 03/Apr/24 02:07
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #3911:
URL: https://github.com/apache/gobblin/pull/3911#discussion_r1548823526
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/GobblinTemporalApplicationMaster.java:
##########
@@ -79,13 +79,14 @@ public GobblinTemporalApplicationMaster(String
applicationName, String applicati
ConfigValueFactory.fromAnyRef(YarnHelixUtils.getContainerNum(containerId.toString()))),
Optional.<Path>absent());
- String containerLogDir =
config.getString(GobblinYarnConfigurationKeys.LOGS_SINK_ROOT_DIR_KEY);
- GobblinYarnLogSource gobblinYarnLogSource = new GobblinYarnLogSource();
- if (gobblinYarnLogSource.isLogSourcePresent()) {
- Path appWorkDir = PathUtils.combinePaths(containerLogDir,
GobblinClusterUtils.getAppWorkDirPath(this.clusterName, this.applicationId),
"AppMaster");
- logCopier = gobblinYarnLogSource.buildLogCopier(this.config,
containerId.toString(), this.fs, appWorkDir);
- this.applicationLauncher
- .addService(logCopier);
+ if (config.hasPath(GobblinYarnConfigurationKeys.LOGS_SINK_ROOT_DIR_KEY)) {
+ String containerLogDir =
config.getString(GobblinYarnConfigurationKeys.LOGS_SINK_ROOT_DIR_KEY);
+ GobblinYarnLogSource gobblinYarnLogSource = new GobblinYarnLogSource();
+ if (gobblinYarnLogSource.isLogSourcePresent()) {
+ Path appWorkDir = PathUtils.combinePaths(containerLogDir,
GobblinClusterUtils.getAppWorkDirPath(this.clusterName, this.applicationId),
"AppMaster");
Review Comment:
The task runner in temporal has the check for this property already built in
Issue Time Tracking
-------------------
Worklog Id: (was: 912757)
Time Spent: 40m (was: 0.5h)
> Support making log dir sink optional
> ------------------------------------
>
> Key: GOBBLIN-2032
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2032
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
> Reporter: William Lo
> Assignee: Hung Tran
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In Gobblin cluster, there is a configuration to set the log sink in order to
> create a log copier service which copies logs to HDFS from Yarn cluster.
> Since Yarn has a built in log aggregator, we should have a way to disable
> this log sink since it becomes redundant, so we should look into ways to make
> the log directory optional.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
