[jira] [Updated] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow
[
https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin updated SPARK-18750:
---
Fix Version/s: 2.0.3
> spark should be able to control the number of executor and should not throw
> stack overslow
> --
>
> Key: SPARK-18750
> URL: https://issues.apache.org/jira/browse/SPARK-18750
> Project: Spark
> Issue Type: Bug
>Reporter: Neerja Khattar
>Assignee: Marcelo Vanzin
> Fix For: 2.0.3, 2.1.1, 2.2.0
>
>
> When running Sql queries on large datasets. Job fails with stack overflow
> warning and it shows it is requesting lots of executors.
> Looks like there is no limit to number of executors or not even having an
> upperbound based on yarn available resources.
> {noformat}
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n5.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n8.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n2.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of
> 32770 executor(s).
> 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor
> containers, each with 1 cores and 6758 MB memory including 614 MB overhead
> 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of
> 52902 executor(s).
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n5.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n8.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n2.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of
> 32770 executor(s).
> 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor
> containers, each with 1 cores and 6758 MB memory including 614 MB overhead
> 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of
> 52902 executor(s).
> 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1
> time(s) in a row.
> java.lang.StackOverflowError
> at scala.collection.immutable.HashMap.$plus(HashMap.scala:57)
> at scala.collection.immutable.HashMap.$plus(HashMap.scala:36)
> at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
> at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
> at
> scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
> at
> scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
> at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at
> scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:24)
> at
> scala.collection.TraversableLike$class.$plus$plus(TraversableLike.scala:156)
> at
> scala.collection.AbstractTraversable.$plus$plus(Traversable.scala:105)
> at scala.collection.immutable.HashMap.$plus(HashMap.scala:60)
> at scala.collection.immutable.Map$Map4.updated(Map.scala:172)
> at scala.collection.immutable.Map$Map4.$plus(Map.scala:173)
> at scala.collection.immutable.Map$Map4.$plus(Map.scala:158)
> at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
> at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
> at
> scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
>
[jira] [Updated] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow
[
https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin updated SPARK-18750:
---
Description:
When running Sql queries on large datasets. Job fails with stack overflow
warning and it shows it is requesting lots of executors.
Looks like there is no limit to number of executors or not even having an
upperbound based on yarn available resources.
{noformat}
16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
bdtcstr61n5.svr.us.jpmchase.net:8041
16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
bdtcstr61n8.svr.us.jpmchase.net:8041
16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
bdtcstr61n2.svr.us.jpmchase.net:8041
16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of
32770 executor(s).
16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor
containers, each with 1 cores and 6758 MB memory including 614 MB overhead
16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of
52902 executor(s).
16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
bdtcstr61n5.svr.us.jpmchase.net:8041
16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
bdtcstr61n8.svr.us.jpmchase.net:8041
16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
bdtcstr61n2.svr.us.jpmchase.net:8041
16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of
32770 executor(s).
16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor
containers, each with 1 cores and 6758 MB memory including 614 MB overhead
16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of
52902 executor(s).
16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 time(s)
in a row.
java.lang.StackOverflowError
at scala.collection.immutable.HashMap.$plus(HashMap.scala:57)
at scala.collection.immutable.HashMap.$plus(HashMap.scala:36)
at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
at
scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
at
scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
at
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
at
scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at
scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:24)
at
scala.collection.TraversableLike$class.$plus$plus(TraversableLike.scala:156)
at
scala.collection.AbstractTraversable.$plus$plus(Traversable.scala:105)
at scala.collection.immutable.HashMap.$plus(HashMap.scala:60)
at scala.collection.immutable.Map$Map4.updated(Map.scala:172)
at scala.collection.immutable.Map$Map4.$plus(Map.scala:173)
at scala.collection.immutable.Map$Map4.$plus(Map.scala:158)
at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
at
scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
at
[jira] [Updated] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neerja Khattar updated SPARK-18750: --- Description: When running Sql queries on large datasets. Job fails with stack overflow warning and it shows it is requesting lots of executors. Looks like there is no limit to number of executors or not even having an upperbound based on yarn available resources. 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n5.svr.us.jpmchase.net:8041 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n8.svr.us.jpmchase.net:8041 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n2.svr.us.jpmchase.net:8041 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of 32770 executor(s). 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor containers, each with 1 cores and 6758 MB memory including 614 MB overhead 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of 52902 executor(s). 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. java.lang.StackOverflowError at scala.collection.immutable.HashMap.$plus(HashMap.scala:57) at scala.collection.immutable.HashMap.$plus(HashMap.scala:36) at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28) If you notice in the error above, YARN is trying to request 24576 executor containers, whereas the available cores are 1719. The Driver is requesting for 52902 executor(s), which too high. This exception should be fixed was: 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n5.svr.us.jpmchase.net:8041 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n8.svr.us.jpmchase.net:8041 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : bdtcstr61n2.svr.us.jpmchase.net:8041 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of 32770 executor(s). 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor containers, each with 1 cores and 6758 MB memory including 614 MB overhead 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of 52902 executor(s). 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. java.lang.StackOverflowError at scala.collection.immutable.HashMap.$plus(HashMap.scala:57) at scala.collection.immutable.HashMap.$plus(HashMap.scala:36) at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28) If you notice in the error above, YARN is trying to request 24576 executor containers, whereas the available cores are 1719. The Driver is requesting for 52902 executor(s), which too high. This exception should be fixed > spark should be able to control the number of executor and should not throw > stack overslow > -- > > Key: SPARK-18750 > URL: https://issues.apache.org/jira/browse/SPARK-18750 > Project: Spark > Issue Type: Bug >Reporter: Neerja Khattar > > When running Sql queries on large datasets. Job fails with stack overflow > warning and it shows it is requesting lots of executors. > Looks like there is no limit to number of executors or not even having an > upperbound based on yarn available resources. > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n5.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n8.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n2.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of > 32770 executor(s). > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor > containers, each with 1 cores and 6758 MB memory including 614 MB overhead > 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of > 52902 executor(s). > 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 > time(s) in a row. > java.lang.StackOverflowError > at scala.collection.immutable.HashMap.$plus(HashMap.scala:57) > at scala.collection.immutable.HashMap.$plus(HashMap.scala:36) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28) > If you notice in the error above, YARN is trying to request 24576 executor > containers, whereas the available cores are 1719. The Driver is requesting > for 52902 executor(s), which too high. > This exception should be fixed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
