HeartSaVioR edited a comment on issue #23764: [SPARK-26825][SS] Fix temp checkpoint creation in cluster mode when default filesystem is not local. URL: https://github.com/apache/spark/pull/23764#issuecomment-467667830 Hmm... I might be wrong, I had to take a deep look at origin issue. I agree with @jose-torres statement. We may need to re-analyze origin issue [SPARK-26825](https://issues.apache.org/jira/browse/SPARK-26825) again - especially why file/directory creation fails in cluster mode. If temporary directory is correctly given, as it is running on YARN container, I would suspect why it fails to make a change on temporary directory. ``` *Cluster mode:* java.io.tmpdir=/yarn/nm/usercache/root/appcache/application_1549064555573_0029/container_1549064555573_0029_01_000001/tmp/ createTempDir(namePrefix = s"temporary") => /yarn/nm/usercache/root/appcache/application_1549064555573_0029/container_1549064555573_0029_01_000001/tmp/temporary-47c13b28-14bd-4d1b-8acc-3e445948415e ``` ``` 19/02/01 12:53:14 ERROR streaming.StreamMetadata: Error writing stream metadata StreamMetadata(68f9fb30-5853-49b4-b192-f1e0483e0d95) to hdfs://ns1/data/yarn/nm/usercache/root/appcache/application_1548823131831_0160/container_1548823131831_0160_02_000001/tmp/temporary-3789423a-6ded-4084-aab3-3b6301c34e07/metadata org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x ``` ``` scala> val fs = checkpointPath.getFileSystem(spark.sessionState.newHadoopConf()) fs: org.apache.hadoop.fs.FileSystem = DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_632752661_1, ugi=root (auth:SIMPLE)]] scala> checkpointPath.makeQualified(fs.getUri, fs.getWorkingDirectory).toUri.toString res1: String = hdfs://ns1/yarn/nm/usercache/root/appcache/application_1549064555573_0029/container_1549064555573_0029_01_000001/tmp/temporary-47c13b28-14bd-4d1b-8acc-3e445948415e ``` Could we find any weird thing here?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
