Wei Zheng commented on SPARK-21521:

We came across the same problem recently - SHS UI only shows the jobs for the 
user who started SHS service. Although that user is a super user (both in local 
FS and HDFS), it cannot read other users' job log files (due to rwxrwx---).

Special logic to tell whether a user is a super user is nice, but I don't know 
if that's doable, because that logic may be vendor specific. For those using 
HDFS maybe we can read dfs.permissions.supergroup from hdfs-site.xml and tell, 
but other system like MapR doesn't use hdfs-site.xml at all but has different 
configs. I don't know if that's the case for other vendors.

We currently work around this issue by changing LOG_FILE_PERMISSIONS from 770 
to 774. I'm not sure if that's a safe change though.

> History service requires user is in any group
> ---------------------------------------------
>                 Key: SPARK-21521
>                 URL: https://issues.apache.org/jira/browse/SPARK-21521
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Adrian Bridgett
>            Priority: Major
> (Regression cf. 2.0.2)
> We run spark as several users, these write to the history location where the 
> files are saved as those users with permissions of 770 (this is hardcoded in 
> EventLoggingListener.scala).
> The history service runs as root so that it has permissions on these files 
> (see https://spark.apache.org/docs/latest/security.html).
> This worked fine in v2.0.2, however in v2.2.0 the events are being skipped 
> unless I add the root user into each users group at which point they are seen.
> We currently have all acls configuration unset.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to