HeartSaVioR commented on issue #23764: [SPARK-26825][SS] Fix temp checkpoint creation in cluster mode when default filesystem is not local. URL: https://github.com/apache/spark/pull/23764#issuecomment-467667830 Hmm... I might be missing to take a deep look at origin issue and hence wrong. I agree with @jose-torres statement. We may need to re-analyze origin issue [SPARK-26825](https://issues.apache.org/jira/browse/SPARK-26825) again - especially why file/directory creation fails in cluster mode. If temporary directory is correctly given, as it is running on YARN container, I would suspect why it fails to make a change on temporary directory. ``` *Cluster mode:* java.io.tmpdir=/yarn/nm/usercache/root/appcache/application_1549064555573_0029/container_1549064555573_0029_01_000001/tmp/ createTempDir(namePrefix = s"temporary") => /yarn/nm/usercache/root/appcache/application_1549064555573_0029/container_1549064555573_0029_01_000001/tmp/temporary-47c13b28-14bd-4d1b-8acc-3e445948415e ``` ``` 19/02/01 12:53:14 ERROR streaming.StreamMetadata: Error writing stream metadata StreamMetadata(68f9fb30-5853-49b4-b192-f1e0483e0d95) to hdfs://ns1/data/yarn/nm/usercache/root/appcache/application_1548823131831_0160/container_1548823131831_0160_02_000001/tmp/temporary-3789423a-6ded-4084-aab3-3b6301c34e07/metadata org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x ``` ``` scala> val fs = checkpointPath.getFileSystem(spark.sessionState.newHadoopConf()) fs: org.apache.hadoop.fs.FileSystem = DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_632752661_1, ugi=root (auth:SIMPLE)]] scala> checkpointPath.makeQualified(fs.getUri, fs.getWorkingDirectory).toUri.toString res1: String = hdfs://ns1/yarn/nm/usercache/root/appcache/application_1549064555573_0029/container_1549064555573_0029_01_000001/tmp/temporary-47c13b28-14bd-4d1b-8acc-3e445948415e ``` Could you find any weird thing here?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
