Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/21895
@jerryshao that can be done but I see mainly 2 problems IIUC:
- your suggestion about blacklisting has the same "caching" and "memory
leakage" problems of the solution proposed here, ie. if permissions on the file
are changed, we wouldn't be aware until STS is restarted and we need to store
in memory the set of the files in blacklist (they may be much less than the
total number of files, this is true, so probably this is not a big problem);
- with your suggestion, we will also try to delete the log file IIUC, so
we have to lower to debug also the log related to access denied when deleting
the files (I don't think this is a big issue, but this is something which has
to be taken into account, as user may miss other issues which are currently
evident with the normal logging level, eg. if the file is readonly for the
spark user).
What do you think? Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
