[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68156011 Jenkins, this is ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68156052 Super-minor process nit, but do you mind moving your comment into the PR description itself? The PR description automatically becomes the commit message, so keeping it up-to-date means less work for committers when they merge your PRs since they don't have to fix up the message by hand. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68156186 [Test build #24839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24839/consoleFull) for PR 3805 at commit [`41ede0e`](https://github.com/apache/spark/commit/41ede0ee67f77e09f2abe96c981167ed671e0504). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68156238 This class of issue could be a more general problem for our test-suites, since I think there are a number of places where we call things like `new SparkConf()` that might implicitly read defaults from the configuration file. I wonder if there's a more general fix, such as using `Utils.isTesting` to bypass the defaults loading in unit tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68156344 Also, the PR / JIRA title is confusing; I can't really guess what this patch does based on the title, since fix an implicit bug could mean many different things. A better title would be something like Do not read spark.executor.memory from spark-defaults.conf in SparkSubmitSuite. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68159278 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24839/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68159276 [Test build #24839 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24839/consoleFull) for PR 3805 at commit [`41ede0e`](https://github.com/apache/spark/commit/41ede0ee67f77e09f2abe96c981167ed671e0504). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68117570 The test 'includes jars passed in through --jarsâ in SparkSubmitSuite fails when spark.executor.memory is set at over 512MiB in conf/spark-default.conf. An exception is thrown as follows: Exception in thread main org.apache.spark.SparkException: Asked to launch cluster with 512 MB RAM / worker but requested 1024 MB/worker at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:1889) at org.apache.spark.SparkContext.init(SparkContext.scala:322) at org.apache.spark.deploy.JarCreationTest$.main(SparkSubmitSuite.scala:458) at org.apache.spark.deploy.JarCreationTest.main(SparkSubmitSuite.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4970] Fix an implicit bug in SparkSubmi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3805#issuecomment-68117596 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
