[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20891 @mgaido91 should probably close this PR for now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20891 I'm +1 on keeping the behavior similar to that of hadoop. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20891 @vanzin thanks for your comment. I think it is important, but if I am the only one thinking this in the community of course I can close this PR. Therefore, I'd like to know also others' opinion on this topic. @jerryshao the YARN RM UI shows the applications to all the users, as SHS is doing now, indeed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20891 @mgaido91 what is the status of Hadoop, for example like YARN RM UI, will it show apps which is run by other users, while this user doesn't have permission to see? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20891 @mgaido91 I really think it's wrong to try to draw a parallel to something like Oracle. Oracle is completely unlike Spark - it's a self-contained system where you don't have any outside visibility except through what Oracle gives you. Spark relies on a bunch of other systems to do things like run processes on a cluster, store data, etc. And the things you're trying to hide here are all visible in those different layers. Even with Oracle, you could check whether people are running certain tools on client machines and say "hey, user foo is connecting to Oracle". You may not know which DB they're connecting to, and you definitely won't know what it is that they're doing. But you also don't know that with Spark. To go through your examples: - user names *are not sensitive information*. You can see them in /etc/passwd. You can see them by listing files on your fs - *even if you don't have read permissions on the file itself*, or reading ACLs for those files. If you want two companies to not see each other, you deploy different clusters (or, in this case, different SHS reading from different event log directories, with different authentication for each). - The app name is arguable. But it's always been public in Spark, so people shouldn't be using that for anything sensitive. If they are, well, they already have a security problem right there, today, and your patch won't fix it, since that data has already leaked. And better hope that app name was not set in any command line, since those are visible to anyone who can log into the same machine. - Who's using the cluster. Again, not sensitive information. If you want to draw a parallel to something like Oracle, you should be looking at the thrift server. That one is supposed to be a multi-user service that shouldn't leak information to users other than the one that submitted a specific job. I have no idea whether that is the case today, but if it's not, it would be a completely different change from what you have here. If you still think this is important, at the very least this needs to be opt-in. But I'm still very skeptical about the need for this at all. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20891 > There is no other system which allows users without read permissions to list what other users are doing. I just ran ps on my machine and saw everything the root user was doing. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20891 @jerryshao @ajbozarth also @vanzin told me the same in the JIRA. Honestly, I think that it was an error to reject it in the past. There is no other system which allows users without read permissions to list what other users are doing. You can check the behavior of any DB for instance (Postgres, Oracle, ...). In this way, we are showing to all users which other users are on the system, when they are running applications and so on. Some example of information which users can know in this way, while they shouldn't: - the name of the other users on the system (if I am a company and I have 2 consultant companies working on the same cluster, I might not want each company to know that the other is working there too); - if the name of the applications are explicit, I can understand what another user is doing on the cluster, even though I do not have read permissions for their applications; again I might know that a competitor company is working on that cluster for doing a specific work - non-admin users can see how many and which users are currently using the cluster. All these things should not be disclosed to non-admin users. I think this is especially critical in situation where a company has a cluster but many other consulting companies are working on it. The one who owns the cluster is likely not willing to disclose to its consultant which other consultant are there and what they are doing. In this way, we are letting them know. Moreover, it is semantically wrong. You are saying that a user has no read permission for an application, but actually it can see that the application exists, its name, duration, submitting user, and so on. So you can see some details for something you have no read access to. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/20891 jerry is correct, the community rejected a similar pr in the past --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/20891 I think I fixed this issue before, but the community rejected such proposal. They thought that listing all the applications for any user is OK without checking the permission. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88560/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20891 **[Test build #88560 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88560/testReport)** for PR 20891 at commit [`126e6a8`](https://github.com/apache/spark/commit/126e6a8e7d333ecf99c26b374698d7cd0e1a9d19). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88561/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20891 **[Test build #88561 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88561/testReport)** for PR 20891 at commit [`cefd672`](https://github.com/apache/spark/commit/cefd672e79b508e995382ce146cd70a4d130af01). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1736/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20891 **[Test build #88561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88561/testReport)** for PR 20891 at commit [`cefd672`](https://github.com/apache/spark/commit/cefd672e79b508e995382ce146cd70a4d130af01). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1735/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20891 **[Test build #88560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88560/testReport)** for PR 20891 at commit [`126e6a8`](https://github.com/apache/spark/commit/126e6a8e7d333ecf99c26b374698d7cd0e1a9d19). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88545/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20891 **[Test build #88545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88545/testReport)** for PR 20891 at commit [`bc87945`](https://github.com/apache/spark/commit/bc879455d8c7057a181989461cae19e60c82966d). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20891 **[Test build #88545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88545/testReport)** for PR 20891 at commit [`bc87945`](https://github.com/apache/spark/commit/bc879455d8c7057a181989461cae19e60c82966d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20891: [SPARK-23782][CORE][UI] SHS should list only application...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20891 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1726/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
