[jira] [Commented] (TEZ-4555) Fail fast in LocalClient if the dirs (log, local) haven't been created
[
https://issues.apache.org/jira/browse/TEZ-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840916#comment-17840916
]
László Bodor commented on TEZ-4555:
---
thanks [~ayushtkn] for the review!
> Fail fast in LocalClient if the dirs (log, local) haven't been created
> --
>
> Key: TEZ-4555
> URL: https://issues.apache.org/jira/browse/TEZ-4555
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 0.10.4
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://github.com/apache/tez/blob/f080031f5c72bc4bfd8090ccdc670bdc0f7fd090/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L332-L335
> {code}
> Path logDir = new Path(userDir, "localmode-log-dir");
> Path localDir = new Path(userDir, "localmode-local-dir");
> localFs.mkdirs(logDir);
> localFs.mkdirs(localDir);
> {code}
> in case of a non-writable local fs path (/base), this mkdirs silently returns
> with false, whereas I can see that it's not writable on my mac:
> {code}
> mkdir -p /base
> mkdir: /base: Read-only file system
> {code}
> leading to a confusing error message later:
> {code}
> 2024-04-24T02:03:52,101 ERROR [DAGAppMaster Thread] client.LocalClient: Error
> starting DAGAppMaster
> java.io.FileNotFoundException:
> /base/scratch/laszlobodor/_tez_session_dir/b76689bc-d25e-4d65-a339-44206ff57ce2/.tez/application_1713949431891_0001_wd/tez-conf.pb
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_292]
> at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_292]
> at java.io.FileInputStream.(FileInputStream.java:138)
> ~[?:1.8.0_292]
> at
> org.apache.tez.common.TezUtilsInternal.readUserSpecifiedTezConfiguration(TezUtilsInternal.java:84)
> ~[tez-common-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at
> org.apache.tez.client.LocalClient.createDAGAppMaster(LocalClient.java:394)
> ~[tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at org.apache.tez.client.LocalClient$1.run(LocalClient.java:357)
> [tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> {code}
> actually, the fix should be done in HIVE-28212, but we need to fail fast here
> and give a hint to the user about the folder
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (TEZ-4555) Fail fast in LocalClient if the dirs (log, local) haven't been created
[
https://issues.apache.org/jira/browse/TEZ-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840746#comment-17840746
]
Ayush Saxena commented on TEZ-4555:
---
Committed to master.
Thanx [~abstractdog] for the contribution!!!
> Fail fast in LocalClient if the dirs (log, local) haven't been created
> --
>
> Key: TEZ-4555
> URL: https://issues.apache.org/jira/browse/TEZ-4555
> Project: Apache Tez
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> https://github.com/apache/tez/blob/f080031f5c72bc4bfd8090ccdc670bdc0f7fd090/tez-dag/src/main/java/org/apache/tez/client/LocalClient.java#L332-L335
> {code}
> Path logDir = new Path(userDir, "localmode-log-dir");
> Path localDir = new Path(userDir, "localmode-local-dir");
> localFs.mkdirs(logDir);
> localFs.mkdirs(localDir);
> {code}
> in case of a non-writable local fs path (/base), this mkdirs silently returns
> with false, whereas I can see that it's not writable on my mac:
> {code}
> mkdir -p /base
> mkdir: /base: Read-only file system
> {code}
> leading to a confusing error message later:
> {code}
> 2024-04-24T02:03:52,101 ERROR [DAGAppMaster Thread] client.LocalClient: Error
> starting DAGAppMaster
> java.io.FileNotFoundException:
> /base/scratch/laszlobodor/_tez_session_dir/b76689bc-d25e-4d65-a339-44206ff57ce2/.tez/application_1713949431891_0001_wd/tez-conf.pb
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_292]
> at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_292]
> at java.io.FileInputStream.(FileInputStream.java:138)
> ~[?:1.8.0_292]
> at
> org.apache.tez.common.TezUtilsInternal.readUserSpecifiedTezConfiguration(TezUtilsInternal.java:84)
> ~[tez-common-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at
> org.apache.tez.client.LocalClient.createDAGAppMaster(LocalClient.java:394)
> ~[tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at org.apache.tez.client.LocalClient$1.run(LocalClient.java:357)
> [tez-dag-0.9.1.2024.0.19.0-3.jar:0.9.1.2024.0.19.0-3]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
> {code}
> actually, the fix should be done in HIVE-28212, but we need to fail fast here
> and give a hint to the user about the folder
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
