[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 OK, let me try it, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17582 It would be good, but maybe the 2.1 backport will merge cleanly to 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 What about branch 2.0, do we also need to backport to it @vanzin ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17582 No luck with 2.1, please file a separate PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17582 LGTM. Merging to master / 2.2, will try 2.1 and 2.0 too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 OK, thanks @tgravescs . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/17582 As @vanzin said I think this is fine for now to get this fixed quickly, but filing a follow up jira makes sense.Actually this might be good to get into the 2.1.1 release if they are going to spin another rc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 Thanks @tgravescs for your comments. Do you think it is a good idea to read out ACLs when `mergeApplicationListing ` in [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L457) and keep in `applications`, so that we don't need to load SparkUI to check ACLs when downloading event logs? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/17582 changes lgtm. Did you file a jira to track changing to not use withSparkUI? If user is downloading because the file is huge and takes a long time to render or causes history server to have issue this would hurt that use case. We could wait and see if someone has that use case too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 Just update the description, please review again @vanzin , thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75998/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75998/testReport)** for PR 17582 at commit [`4b3781f`](https://github.com/apache/spark/commit/4b3781ff6dce571130538a3f29a7e386f3e3fb9b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17582 @jerryshao is the PR description still accurate? It seems you're not really implementing 2 anymore. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75998/testReport)** for PR 17582 at commit [`4b3781f`](https://github.com/apache/spark/commit/4b3781ff6dce571130538a3f29a7e386f3e3fb9b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17582 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75977/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75977/testReport)** for PR 17582 at commit [`4b3781f`](https://github.com/apache/spark/commit/4b3781ff6dce571130538a3f29a7e386f3e3fb9b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75977/testReport)** for PR 17582 at commit [`4b3781f`](https://github.com/apache/spark/commit/4b3781ff6dce571130538a3f29a7e386f3e3fb9b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75970/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75967/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75970 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75970/testReport)** for PR 17582 at commit [`4b3781f`](https://github.com/apache/spark/commit/4b3781ff6dce571130538a3f29a7e386f3e3fb9b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75967/testReport)** for PR 17582 at commit [`68c9d83`](https://github.com/apache/spark/commit/68c9d83a48751e57988f09a46c8e61a073c7d582). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75934/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75934/testReport)** for PR 17582 at commit [`a8fb1e0`](https://github.com/apache/spark/commit/a8fb1e0ad6573aa290b179cd9c1883abc62552be). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75934/testReport)** for PR 17582 at commit [`a8fb1e0`](https://github.com/apache/spark/commit/a8fb1e0ad6573aa290b179cd9c1883abc62552be). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 Thanks @tgravescs for your reply. > on the history server I would expect spark.acls.enable=false and spark.history.ui.acls.enable=true, I can see where that could be confusing, perhaps we should document this better. spark.acls.enable on the history UI really is protecting the root UI, not the app level ui's. We could explicitly turn this off. This could be happened when history server and spark application shares same configuration file. That's why in our internal test the behavior is not expected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/17582 so we should definitely fix the /api/v1/applications//logs to go through the acls. It looks like it should be protected in ApiRootResource.java. You have the app id so it needs to do something like the withSparkUI to get the acls included in that application. Like I mentioned the listing (/api/v1/applications) and /api/v1/applications/ (which is same info I believe as listing) were intentionally left open. I don't really see a reason to change that but if other people have a use case for it then perhaps we should make which pages are protected by acls configurable. on the history server I would expect spark.acls.enable=false and spark.history.ui.acls.enable=true, I can see where that could be confusing, perhaps we should document this better. spark.acls.enable on the history UI really is protecting the root UI, not the app level ui's. We could explicitly turn this off. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 @tgravescs , with the changes of history UI, REST API and web UI are now mixed. The base URL to list all the apps is through REST API. The key problem here is that in History Server we could have configured two ACLs (spark.acls.enable [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala#L299) and spark.history.ui.acls.enable [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala#L248)), this two ACLs checks different URLs, for example: ``` /api/v1/applications /api/v1/applications/ /api/v1/applications//logs ``` are controlled by `spark.acls.enable`. And web UI will use this API to list and show apps on the UI. If this is enabled, then only SHS owner, admin/view user groups could see the apps list on the SHS's base web UI. And all the other URLs (web UI and REST API) for application details are controlled by `spark.history.ui.acls.enable`. If we configured differently for this two ACLs, then we will get some unexpected behaviors. So what I here fixed is to unify the ACL and offer the right behavior. >No, there shouldn't be sensitive information there and many times a user is looking for a job run by say a headless user or other user. I guess you could filter only the jobs that user has acls to but that makes it more complicated. Do you have a concrete reason it should be protected? Note that this follow how other Hadoop UI's work. We worked with customers and they wish to filter and list apps based on the login users. But I don't have strong opinion it. I could change to what you suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user ajbozarth commented on the issue: https://github.com/apache/spark/pull/17582 Been following this but haven't had time to do a proper review, but @tgravescs since you brought up the UI vs API thing, as of 2.0 the UI gets it's list from the API so that's where the security has to be handled. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/17582 Sorry again the wording above and all the different configs are a bit confusing to me as to what the real issues are here. >Here actually has two list of acls, one is controlled by spark.acls.enabled, if user "A" is not added to this acl list, then user "A" cannot see the app list (//api/v1/applications). But if this app is run by user "A", then user "A" could still see the details of app, like (//api/v1/applications//jobs), this acl is controlled by "spark.history.ui.acls.enabled", and user "A" is automatically in the acl list (because of run by him). You are mixing things here. You say that if user "A" is not added to acl list he cannot see the app list. This is broken then and I assume only applies to rest api not UI? But I'm not sure what that has to do with your second sentence, if user "A" ran the app then of course he can see the details of the app, that is intended. I'm not sure what that has to do with the first issue? If you don't have spark.history.ui.acls.enabled then it is up to what the user set. Generally in any secure environment you should set spark.history.ui.acls.enabled=true and it should enforce acls no matter what user set. It might help for you to describe these in terms of configs. Which exact configs are set on the history server and which exact configs are set on the application side and which exact apis are being used (Rest vs Web UI). so all the urls you list are the REST API, is this only an issue with rest api or the actual web UI as well? It sounds like things are definitely broke there but I'm not sure it requires changing the configs just fixing the things that are broken. Its supposed to be that if spark.history.ui.acls.enable is enabled it doesn't matter what the setting of spark.acls.enable is, acls should always be enforced on the history server. see the description: https://spark.apache.org/docs/latest/monitoring.html Certain UI's don't have information that should be sensitive. I thought the list of applications was one of those things, all users should be able to see the entire list of applications. Nothing sensitive there, but once you look at the application details that should be acl'd. If someone added something sensitive then it should be protected or it should be moved from that page. My opinions on your response to @vanzin 1. No, there shouldn't be sensitive information there and many times a user is looking for a job run by say a headless user or other user. I guess you could filter only the jobs that user has acls to but that makes it more complicated. Do you have a concrete reason it should be protected? Note that this follow how other Hadoop UI's work. 2. That is just broken, event log should be protected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 @tgravescs @vanzin do you have any comment on this JIRA? A compromise is that any user could see all the app list but detailed information is still controlled by per app ACLs. But we should also fix event log download issue, currently anyone could download the event log if "spark.acls.enable" is disabled, even it is not permitted by HDFS. This is definitely a security hole should be fixed. What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17582 @tgravescs sorry for the confuse. >if base URL's ACL (spark.acls.enable) is enabled but user A has no view permission. User "A" cannot see the app list but could still access details of it's own app. Here actually has two list of acls, one is controlled by `spark.acls.enabled`, if user "A" is not added to this acl list, then user "A" cannot see the app list (`//api/v1/applications`). But if this app is run by user "A", then user "A" could still see the details of app, like (`//api/v1/applications//jobs`), this acl is controlled by "spark.history.ui.acls.enabled", and user "A" is automatically in the acl list (because of run by him). > if ACLs of base URL (spark.acls.enable) is disabled. Then user "A" could see the summary of all the apps, even some apps didn't run by user "A", but can only access its own app's details. If "spark.acls.enabled" is disabled, then `SecurityFilter` is not worked, so user "A" could access `//api/v1/applications`, which means user "A" could see all the applications even not run by him. This `//api/v1/applications` doesn't touch `spark.history.ui.acls.enabled`. > if ACLs of base URL (spark.acls.enable) is disabled, then user "A" could download any application's event log, even it is not run by user "A". This is the same issue as above. `//api/v1/applications//logs` is only controlled by "spark.acls.enable", not "spark.history.ui.acls.enable". So anyone could download any even logs if "spark.acls.enable" is disabled. So basically what I fixed is that: 1. disable the work of `spark.acls.enable`, which means `SecurityFilter` is not checked. 2. Using `spark.history.ui.acls.enable` to filter applications, application summary and application log based on users who run the app. So the result of my PR is: 1. history admin user could see/download/access any apps. 2. normal user could see/download/access apps run by him. @vanzin your suggestion is to only disable ACLs on the listing, that definitely simplifies the fix, but IMO that "all or nothing" solution is not so ideal: 1. any user could list all the apps, though cannot access the details if it is not run by him. For the sensitivity, is it better to even not show the apps not run by him? 2. currently if ACLs on listing is disabled, anyone could download event log, which on the other hand expose the security hole to other users. So IMO filtering based on users is better than "all or nothing" solution. Also it doesn't increase the code complex much. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17582 > user configured with "spark.admin.acls" (or group) or "spark.ui.view.acls" (or group), or the user who started SHS could list all the applications, otherwise none of them can be listed So to me this is the only bug; which means that maybe ACLs on the listing itself shouldn't ever be applied, and this PR should be a lot simpler, right? Most of it seem to be dealing with filtering the list of apps so that only applications the user can see are shown. I wonder if that's necessary, since the only thing that's showing is the existence of the application, not any data about it that could be considered sensitive. There's also a minor thing that the listing being different for different users might cause confusion; but if there's a good reason for filtering, then that concern can be overridden. I'm just not sure there is a good reason for it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/17582 Sorry but I'm confused by the explanation in the description. I didn't completely follow what problems you are seeing that aren't intended and I don't understand how you are proposing to fix. Can you please describe the design you are proposing in more detail? On the description can you please clarify each of your bullets? For instance: 1. if base URL's ACL (spark.acls.enable) is enabled but user A has no view permission. User "A" cannot see the app list but could still access details of it's own app. Are you saying user A is not in the list of acls or is? if they have no view permission then they shouldn't be able to see the app. I don't understnad what you mean by "could still access details of it's own app"? Is this user A's application (meaning they started it) and hence he would automatically be in the acl list? Clarifying the other bullets would be helpful as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75643/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75643/testReport)** for PR 17582 at commit [`e56c388`](https://github.com/apache/spark/commit/e56c3889524a835e7b07915996cd945a9d6a8f67). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75643/testReport)** for PR 17582 at commit [`e56c388`](https://github.com/apache/spark/commit/e56c3889524a835e7b07915996cd945a9d6a8f67). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75641/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75641/testReport)** for PR 17582 at commit [`bc1e53a`](https://github.com/apache/spark/commit/bc1e53a2820ef999b910c81017c0b91dd4fbabbd). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17582 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17582: [SPARK-20239][Core] Improve HistoryServer's ACL mechanis...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17582 **[Test build #75641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75641/testReport)** for PR 17582 at commit [`bc1e53a`](https://github.com/apache/spark/commit/bc1e53a2820ef999b910c81017c0b91dd4fbabbd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
