[GitHub] spark issue #16913: [SPARK-15531] [DEPLOY] Complement launcher JVM memory se...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16913 You should not set a global init heap size like this and this is pretty much exactly why. There is no need for the launcher to immediately requests that much memory otherwise. Please close this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16913: [SPARK-15531] [DEPLOY] Complement launcher JVM memory se...
Github user Pashugan commented on the issue: https://github.com/apache/spark/pull/16913 There must be some misunderstanding. May I have a chance you have a look at my micro-patch because it has nothing to do with the driver and its options. In fact, it fixes the call of the "launcher library", which is used to fill up a bash array, which is in turn used to run the actual driver. Then, my above explanations should hopefully become as clear as day. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16913: [SPARK-15531] [DEPLOY] Complement launcher JVM memory se...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16913 You should not be using JAVA_TOOL_OPTIONS. Set Spark's JVM properties directly with spark.driver.extraJavaOptions, for example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16913: [SPARK-15531] [DEPLOY] Complement launcher JVM memory se...
Github user Pashugan commented on the issue: https://github.com/apache/spark/pull/16913 JAVA_TOOL_OPTIONS doesn't take precedence over command line options (in oppose to _JAVA_OPTIONS). Thus, JVM is trying to start with an initial heap size set in JAVA_TOOL_OPTIONS and a max heap size overridden by the spark-class script, which is currently 128m. If I set a system-wide initial heap size bigger than 128m I'll get a conflict with the max heap size option hardcoded in the Spark code, so it looks to me like a Spark problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16913: [SPARK-15531] [DEPLOY] Complement launcher JVM memory se...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16913 Well, you need to set a larger max heap too. I can't see how this is a Spark problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16913: [SPARK-15531] [DEPLOY] Complement launcher JVM memory se...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16913 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
