Pralabh Kumar created SPARK-43235:
-------------------------------------
Summary: ClientDistributedCacheManager doesn't set the
LocalResourceVisibility.PRIVATE if isPublic throws exception
Key: SPARK-43235
URL: https://issues.apache.org/jira/browse/SPARK-43235
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.4.0
Reporter: Pralabh Kumar
Hi Spark Team .
Currently *ClientDistributedCacheManager* *getVisibility* methods checks
whether resource visibility can be set to private or public.
In order to set *LocalResourceVisibility.PUBLIC* ,isPublic checks permission
of all the ancestors directories for the executable directory . It goes till
the root folder to check permission of all the parents
(ancestorsHaveExecutePermissions)
checkPermissionOfOther calls FileStatus getFileStatus to check the permission .
If the FileStatus getFileStatus throws exception Spark Submit fails . It
didn't sets the permission to Private.
if (isPublic(conf, uri, statCache)) {
LocalResourceVisibility.PUBLIC
} else {
LocalResourceVisibility.PRIVATE
}
Generally if the user doesn't have permission to check for root folder
(specifically in case of cloud file system(GCS) (for the buckets) , methods
throws error IOException(Error accessing Bucket).
*Ideally if there is an error in isPublic , which means Spark isn't able to
determine the execution permission of all the parents directory , it should set
the LocalResourceVisibility.PRIVATE. However, it currently throws an exception
in isPublic and hence Spark Submit fails*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
