Pralabh Kumar created SPARK-43235: ------------------------------------- Summary: ClientDistributedCacheManager doesn't set the LocalResourceVisibility.PRIVATE if isPublic throws exception Key: SPARK-43235 URL: https://issues.apache.org/jira/browse/SPARK-43235 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.4.0 Reporter: Pralabh Kumar
Hi Spark Team . Currently *ClientDistributedCacheManager* *getVisibility* methods checks whether resource visibility can be set to private or public. In order to set *LocalResourceVisibility.PUBLIC* ,isPublic checks permission of all the ancestors directories for the executable directory . It goes till the root folder to check permission of all the parents (ancestorsHaveExecutePermissions) checkPermissionOfOther calls FileStatus getFileStatus to check the permission . If the FileStatus getFileStatus throws exception Spark Submit fails . It didn't sets the permission to Private. if (isPublic(conf, uri, statCache)) { LocalResourceVisibility.PUBLIC } else { LocalResourceVisibility.PRIVATE } Generally if the user doesn't have permission to check for root folder (specifically in case of cloud file system(GCS) (for the buckets) , methods throws error IOException(Error accessing Bucket). *Ideally if there is an error in isPublic , which means Spark isn't able to determine the execution permission of all the parents directory , it should set the LocalResourceVisibility.PRIVATE. However, it currently throws an exception in isPublic and hence Spark Submit fails* -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org