[jira] [Commented] (SPARK-11154) make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options
[ https://issues.apache.org/jira/browse/SPARK-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000419#comment-15000419 ] Dustin Cote commented on SPARK-11154: - [~Kitard] I think the naming convention and strategy makes sense. Someone with more familiarity with the code base should probably comment on the files that need to change though. > make specificaition spark.yarn.executor.memoryOverhead consistent with > typical JVM options > -- > > Key: SPARK-11154 > URL: https://issues.apache.org/jira/browse/SPARK-11154 > Project: Spark > Issue Type: Improvement > Components: Documentation, Spark Submit >Reporter: Dustin Cote >Priority: Minor > > spark.yarn.executor.memoryOverhead is currently specified in megabytes by > default, but it would be nice to allow users to specify the size as though it > were a typical -Xmx option to a JVM where you can have 'm' and 'g' appended > to the end to explicitly specify megabytes or gigabytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-11154) make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options
Dustin Cote created SPARK-11154: --- Summary: make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options Key: SPARK-11154 URL: https://issues.apache.org/jira/browse/SPARK-11154 Project: Spark Issue Type: Improvement Components: Documentation, Spark Submit Reporter: Dustin Cote Priority: Minor spark.yarn.executor.memoryOverhead is currently specified in megabytes by default, but it would be nice to allow users to specify the size as though it were a typical -Xmx option to a JVM where you can have 'm' and 'g' appended to the end to explicitly specify megabytes or gigabytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11154) make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options
[ https://issues.apache.org/jira/browse/SPARK-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961195#comment-14961195 ] Dustin Cote commented on SPARK-11154: - That's absolutely a good point. It might be more confusing to have a default 'untagged' number be specified in megabytes and allow option appended 'g' for gigabytes. I think it would be safe to allow people to specify the 'm' at the end (this currently doesn't work), but I agree it would be a bad thing to move from a default of megabytes to bytes as it wouldn't be backward compatible. This might just be a usability quirk, but I figured I'd call it out since I see customers trying to use this like Xmx all the time. > make specificaition spark.yarn.executor.memoryOverhead consistent with > typical JVM options > -- > > Key: SPARK-11154 > URL: https://issues.apache.org/jira/browse/SPARK-11154 > Project: Spark > Issue Type: Improvement > Components: Documentation, Spark Submit >Reporter: Dustin Cote >Priority: Minor > > spark.yarn.executor.memoryOverhead is currently specified in megabytes by > default, but it would be nice to allow users to specify the size as though it > were a typical -Xmx option to a JVM where you can have 'm' and 'g' appended > to the end to explicitly specify megabytes or gigabytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-11085) Add support for HTTP proxy
Dustin Cote created SPARK-11085: --- Summary: Add support for HTTP proxy Key: SPARK-11085 URL: https://issues.apache.org/jira/browse/SPARK-11085 Project: Spark Issue Type: Improvement Components: Spark Shell, Spark Submit Reporter: Dustin Cote Add a way to update ivysettings.xml for the spark-shell and spark-submit to support proxy settings for clusters that need to access a remote repository through an http proxy. Typically this would be done like: JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=proxy.host -Dhttp.proxyPort=8080 -Dhttps.proxyHost=proxy.host.secure -Dhttps.proxyPort=8080" Directly in the ivysettings.xml would look like: Even better would be a way to customize the ivysettings.xml with command options. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11085) Add support for HTTP proxy
[ https://issues.apache.org/jira/browse/SPARK-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955001#comment-14955001 ] Dustin Cote commented on SPARK-11085: - [~sowen] The problem here is that the dependencies to be downloaded with --packages can't be reached because those settings do not get forwarded into the respective Spark client. I'll note this was being tried with Spark on YARN and the JAVA_OPTS was being set through spark.driver.extraJavaOptions. The ivysettings change was being done through ~/.m2/ivysettings.xml. It's more of a forwarding the settings to the Spark client issue. At least on CDH, the relevant ivysettings.xml is bundled in the assembly jar and apparently not modified by the two methods: :: loading settings :: url = jar:file:/opt/cloudera/parcels/CDH-5.4.2-1.cdh5.4.2.p819.487/jars/spark-assembly-1.3.0-cdh5.4.2-hadoop2.6.0-cdh5.4.2.jar!/org/apache/ivy/core/settings/ivysettings.xml This JIRA would be to come up with a way to modify or override this ivysettings.xml so that it can be used with proxy settings. > Add support for HTTP proxy > --- > > Key: SPARK-11085 > URL: https://issues.apache.org/jira/browse/SPARK-11085 > Project: Spark > Issue Type: Improvement > Components: Spark Shell, Spark Submit >Reporter: Dustin Cote >Priority: Minor > > Add a way to update ivysettings.xml for the spark-shell and spark-submit to > support proxy settings for clusters that need to access a remote repository > through an http proxy. Typically this would be done like: > JAVA_OPTS="$JAVA_OPTS -Dhttp.proxyHost=proxy.host -Dhttp.proxyPort=8080 > -Dhttps.proxyHost=proxy.host.secure -Dhttps.proxyPort=8080" > Directly in the ivysettings.xml would look like: > > proxyport="8080" > nonproxyhosts="nonproxy.host"/> > > Even better would be a way to customize the ivysettings.xml with command > options. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
