date:20160502

Re: [build system] short downtime monday morning (5-2-16), 7-9am PDT

2016-05-02 Thread Reynold Xin

Thanks, Shane!

On Monday, May 2, 2016, shane knapp  wrote:

> workers -01 and -04 are back up, is is -06 (as i hit the wrong power
> button by accident).  :)
>
> -01 and -04 got hung on shutdown, so i'll investigate them and see
> what exactly happened.  regardless, we should be building happily!
>
> On Mon, May 2, 2016 at 8:44 AM, shane knapp  > wrote:
> > hey everyone!
> >
> > looks like two of the workers didn't survive a reboot, so i will need
> > to head to the colo and console in to see what's going on.
> >
> > sadly, one of the workers that didn't come back is -01, which runs the
> > doc builds.
> >
> > anyways, i will post another update within the hour with the status of
> > these two machines.  i'm also unpausing builds.
> >
> > On Mon, May 2, 2016 at 8:26 AM, shane knapp  > wrote:
> >> this is happening now.
> >>
> >> On Fri, Apr 29, 2016 at 12:52 PM, shane knapp  > wrote:
> >>> (copy-pasta of previous message)
> >>>
> >>> another project hosted on our jenkins (e-mission) needs anaconda scipy
> >>> upgraded from 0.15.1 to 0.17.0.  this will also upgrade a few other
> >>> libs, which i've included at the end of this email.
> >>>
> >>> i've spoken w/josh @ databricks and we don't believe that this will
> >>> impact the spark builds at all.  if this causes serious breakage, i
> >>> will roll everything back to pre-update.
> >>>
> >>> i have created a JIRA issue to look in to creating conda environments
> >>> for spark builds, something that we should have done long ago:
> >>>
> >>> https://issues.apache.org/jira/browse/SPARK-14905
> >>>
> >>> builds will be paused:  ~7am PDT
> >>> anaconda package updates:  ~8am
> >>> jenkins quiet time ends:  ~9am at the latest
> >>>
> >>> i do not expect the downtime to last very long, and will update this
> >>> thread w/updates as they come.
> >>>
> >>> here's what will be updated under anaconda:
> >>>
> >>> The following NEW packages will be INSTALLED:
> >>>
> >>> libgfortran: 3.0-0
> >>> mkl: 11.3.1-0
> >>> wheel:   0.29.0-py27_0
> >>>
> >>> The following packages will be UPDATED:
> >>>
> >>> conda:   3.10.1-py27_0 --> 4.0.5-py27_0
> >>> conda-env:   2.1.4-py27_0  --> 2.4.5-py27_0
> >>> numpy:   1.9.2-py27_0  --> 1.11.0-py27_0
> >>> openssl: 1.0.1k-1  --> 1.0.2g-0
> >>> pip: 6.1.1-py27_0  --> 8.1.1-py27_1
> >>> python:  2.7.9-2   --> 2.7.11-0
> >>> pyyaml:  3.11-py27_0   --> 3.11-py27_1
> >>> requests:2.6.0-py27_0  --> 2.9.1-py27_0
> >>> scipy:   0.15.1-np19py27_0 --> 0.17.0-np111py27_2
> >>> setuptools:  15.0-py27_0   --> 20.7.0-py27_0
> >>> sqlite:  3.8.4.1-1 --> 3.9.2-0
> >>> yaml:0.1.4-0   --> 0.1.6-0
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org 
> For additional commands, e-mail: dev-h...@spark.apache.org 
>
>

Re: [build system] short downtime monday morning (5-2-16), 7-9am PDT

2016-05-02 Thread shane knapp

workers -01 and -04 are back up, is is -06 (as i hit the wrong power
button by accident).  :)

-01 and -04 got hung on shutdown, so i'll investigate them and see
what exactly happened.  regardless, we should be building happily!

On Mon, May 2, 2016 at 8:44 AM, shane knapp  wrote:
> hey everyone!
>
> looks like two of the workers didn't survive a reboot, so i will need
> to head to the colo and console in to see what's going on.
>
> sadly, one of the workers that didn't come back is -01, which runs the
> doc builds.
>
> anyways, i will post another update within the hour with the status of
> these two machines.  i'm also unpausing builds.
>
> On Mon, May 2, 2016 at 8:26 AM, shane knapp  wrote:
>> this is happening now.
>>
>> On Fri, Apr 29, 2016 at 12:52 PM, shane knapp  wrote:
>>> (copy-pasta of previous message)
>>>
>>> another project hosted on our jenkins (e-mission) needs anaconda scipy
>>> upgraded from 0.15.1 to 0.17.0.  this will also upgrade a few other
>>> libs, which i've included at the end of this email.
>>>
>>> i've spoken w/josh @ databricks and we don't believe that this will
>>> impact the spark builds at all.  if this causes serious breakage, i
>>> will roll everything back to pre-update.
>>>
>>> i have created a JIRA issue to look in to creating conda environments
>>> for spark builds, something that we should have done long ago:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-14905
>>>
>>> builds will be paused:  ~7am PDT
>>> anaconda package updates:  ~8am
>>> jenkins quiet time ends:  ~9am at the latest
>>>
>>> i do not expect the downtime to last very long, and will update this
>>> thread w/updates as they come.
>>>
>>> here's what will be updated under anaconda:
>>>
>>> The following NEW packages will be INSTALLED:
>>>
>>> libgfortran: 3.0-0
>>> mkl: 11.3.1-0
>>> wheel:   0.29.0-py27_0
>>>
>>> The following packages will be UPDATED:
>>>
>>> conda:   3.10.1-py27_0 --> 4.0.5-py27_0
>>> conda-env:   2.1.4-py27_0  --> 2.4.5-py27_0
>>> numpy:   1.9.2-py27_0  --> 1.11.0-py27_0
>>> openssl: 1.0.1k-1  --> 1.0.2g-0
>>> pip: 6.1.1-py27_0  --> 8.1.1-py27_1
>>> python:  2.7.9-2   --> 2.7.11-0
>>> pyyaml:  3.11-py27_0   --> 3.11-py27_1
>>> requests:2.6.0-py27_0  --> 2.9.1-py27_0
>>> scipy:   0.15.1-np19py27_0 --> 0.17.0-np111py27_2
>>> setuptools:  15.0-py27_0   --> 20.7.0-py27_0
>>> sqlite:  3.8.4.1-1 --> 3.9.2-0
>>> yaml:0.1.4-0   --> 0.1.6-0

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: spark 2 segfault

2016-05-02 Thread Ted Yu

I plan to.

I am not that familiar with all the parts involved though :-)

On Mon, May 2, 2016 at 9:42 AM, Reynold Xin  wrote:

> Definitely looks like a bug.
>
> Ted - are you looking at this?
>
>
> On Mon, May 2, 2016 at 7:15 AM, Koert Kuipers  wrote:
>
>> Created issue:
>> https://issues.apache.org/jira/browse/SPARK-15062
>>
>> On Mon, May 2, 2016 at 6:48 AM, Ted Yu  wrote:
>>
>>> I tried the same statement using Spark 1.6.1
>>> There was no error with default memory setting.
>>>
>>> Suggest logging a bug.
>>>
>>> On May 1, 2016, at 9:22 PM, Koert Kuipers  wrote:
>>>
>>> Yeah I got that too, then I increased heap for tests to 8G to get error
>>> I showed earlier.
>>> On May 2, 2016 12:09 AM, "Ted Yu"  wrote:
>>>
 Using commit hash 90787de864b58a1079c23e6581381ca8ffe7685f and
 Java 1.7.0_67 , I got:

 scala> val dfComplicated = sc.parallelize(List((Map("1" -> "a"),
 List("b", "c")), (Map("2" -> "b"), List("d", "e".toDF
 ...
 dfComplicated: org.apache.spark.sql.DataFrame = [_1:
 map, _2: array]

 scala> dfComplicated.show
 java.lang.OutOfMemoryError: Java heap space
   at
 org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:229)
   at
 org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:821)
   at
 org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown
 Source)
   at
 org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:241)
   at
 org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121)
   at
 org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121)
   at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
   at
 scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
   at
 scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
   at
 scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
   at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
   at
 org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2121)
   at
 org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:54)
   at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2408)
   at org.apache.spark.sql.Dataset.org
 $apache$spark$sql$Dataset$$execute$1(Dataset.scala:2120)
   at org.apache.spark.sql.Dataset.org
 $apache$spark$sql$Dataset$$collect(Dataset.scala:2127)
   at
 org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1861)
   at
 org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1860)
   at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2438)
   at org.apache.spark.sql.Dataset.head(Dataset.scala:1860)
   at org.apache.spark.sql.Dataset.take(Dataset.scala:2077)
   at org.apache.spark.sql.Dataset.showString(Dataset.scala:238)
   at org.apache.spark.sql.Dataset.show(Dataset.scala:529)
   at org.apache.spark.sql.Dataset.show(Dataset.scala:489)
   at org.apache.spark.sql.Dataset.show(Dataset.scala:498)
   ... 6 elided

 scala>

 On Sun, May 1, 2016 at 8:34 PM, Koert Kuipers 
 wrote:

> by removing dependencies it turns into a different error, see below.
> the test is simply writing a DataFrame out to file and reading it back
> in. i see the error for all data sources (json, parquet, etc.).
>
> this is the data that i write out and read back in:
> val dfComplicated = sc.parallelize(List((Map("1" -> "a"), List("b",
> "c")), (Map("2" -> "b"), List("d", "e".toDF
>
>
> [info]   java.lang.RuntimeException: Error while decoding:
> java.lang.NegativeArraySizeException
> [info] createexternalrow(if (isnull(input[0, map]))
> null else staticinvoke(class
> org.apache.spark.sql.catalyst.util.ArrayBasedMapData$, 
> ObjectType(interface
> scala.collection.Map), toScalaMap, staticinvoke(class
> scala.collection.mutable.WrappedArray$, ObjectType(interface
> scala.collection.Seq), make,
> mapobjects(lambdavariable(MapObjects_loopValue16, MapObjects_loopIsNull17,
> StringType), lambdavariable(MapObjects_loopValue16,
> MapObjects_loopIsNull17, StringType).toString, input[0,
> map].keyArray).array, true), staticinvoke(class
> scala.collection.mutable.WrappedArray$, ObjectType(interface
> scala.collection.Seq), make,
>

Re: spark 2 segfault

2016-05-02 Thread Reynold Xin

Definitely looks like a bug.

Ted - are you looking at this?


On Mon, May 2, 2016 at 7:15 AM, Koert Kuipers  wrote:

> Created issue:
> https://issues.apache.org/jira/browse/SPARK-15062
>
> On Mon, May 2, 2016 at 6:48 AM, Ted Yu  wrote:
>
>> I tried the same statement using Spark 1.6.1
>> There was no error with default memory setting.
>>
>> Suggest logging a bug.
>>
>> On May 1, 2016, at 9:22 PM, Koert Kuipers  wrote:
>>
>> Yeah I got that too, then I increased heap for tests to 8G to get error I
>> showed earlier.
>> On May 2, 2016 12:09 AM, "Ted Yu"  wrote:
>>
>>> Using commit hash 90787de864b58a1079c23e6581381ca8ffe7685f and
>>> Java 1.7.0_67 , I got:
>>>
>>> scala> val dfComplicated = sc.parallelize(List((Map("1" -> "a"),
>>> List("b", "c")), (Map("2" -> "b"), List("d", "e".toDF
>>> ...
>>> dfComplicated: org.apache.spark.sql.DataFrame = [_1: map,
>>> _2: array]
>>>
>>> scala> dfComplicated.show
>>> java.lang.OutOfMemoryError: Java heap space
>>>   at
>>> org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:229)
>>>   at
>>> org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:821)
>>>   at
>>> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown
>>> Source)
>>>   at
>>> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:241)
>>>   at
>>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121)
>>>   at
>>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121)
>>>   at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>>>   at
>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>>>   at
>>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>>>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>>>   at
>>> scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>>>   at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
>>>   at
>>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2121)
>>>   at
>>> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:54)
>>>   at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2408)
>>>   at org.apache.spark.sql.Dataset.org
>>> $apache$spark$sql$Dataset$$execute$1(Dataset.scala:2120)
>>>   at org.apache.spark.sql.Dataset.org
>>> $apache$spark$sql$Dataset$$collect(Dataset.scala:2127)
>>>   at
>>> org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1861)
>>>   at
>>> org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1860)
>>>   at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2438)
>>>   at org.apache.spark.sql.Dataset.head(Dataset.scala:1860)
>>>   at org.apache.spark.sql.Dataset.take(Dataset.scala:2077)
>>>   at org.apache.spark.sql.Dataset.showString(Dataset.scala:238)
>>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:529)
>>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:489)
>>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:498)
>>>   ... 6 elided
>>>
>>> scala>
>>>
>>> On Sun, May 1, 2016 at 8:34 PM, Koert Kuipers  wrote:
>>>
 by removing dependencies it turns into a different error, see below.
 the test is simply writing a DataFrame out to file and reading it back
 in. i see the error for all data sources (json, parquet, etc.).

 this is the data that i write out and read back in:
 val dfComplicated = sc.parallelize(List((Map("1" -> "a"), List("b",
 "c")), (Map("2" -> "b"), List("d", "e".toDF


 [info]   java.lang.RuntimeException: Error while decoding:
 java.lang.NegativeArraySizeException
 [info] createexternalrow(if (isnull(input[0, map])) null
 else staticinvoke(class
 org.apache.spark.sql.catalyst.util.ArrayBasedMapData$, ObjectType(interface
 scala.collection.Map), toScalaMap, staticinvoke(class
 scala.collection.mutable.WrappedArray$, ObjectType(interface
 scala.collection.Seq), make,
 mapobjects(lambdavariable(MapObjects_loopValue16, MapObjects_loopIsNull17,
 StringType), lambdavariable(MapObjects_loopValue16,
 MapObjects_loopIsNull17, StringType).toString, input[0,
 map].keyArray).array, true), staticinvoke(class
 scala.collection.mutable.WrappedArray$, ObjectType(interface
 scala.collection.Seq), make,
 mapobjects(lambdavariable(MapObjects_loopValue18, MapObjects_loopIsNull19,
 StringType), lambdavariable(MapObjects_loopValue18,
 MapObjects_loopIsNull19, StringType).toString, input[0,
 map].valueArray).array, true), true), if (isnull(input[1,

Re: Cross Validator to work with K-Fold value of 1?

2016-05-02 Thread Nick Pentreath

There is a JIRA and PR around for supporting polynomial expansion with
degree 1. Offhand I can't recall if it's been merged
On Mon, 2 May 2016 at 17:45, Julio Antonio Soto de Vicente 
wrote:

> Hi,
>
> Same goes for the PolynomialExpansion in org.apache.spark.ml.feature. It
> would be dice to cross-validate with degree 1 polynomial expansion (this
> is, with no expansion at all) vs other degree polynomial expansions.
> Unfortunately, degree is forced to be >= 2.
>
> --
> Julio
>
> > El 2 may 2016, a las 9:05, Rahul Tanwani 
> escribió:
> >
> > Hi,
> >
> > In certain cases (mostly due to time constraints), we need some model to
> run
> > without cross validation. In such a case, since k-fold value for cross
> > validator cannot be one, we have to maintain two different code paths to
> > achieve both the scenarios (with and without cross validation).
> >
> > Would it be an okay idea to generalize the cross validator so it can work
> > with k-fold value of 1? The only purpose for this is to avoid maintaining
> > two different code paths and in functionality it should be similar to as
> if
> > the cross validation is not present.
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Cross-Validator-to-work-with-K-Fold-value-of-1-tp17404.html
> > Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> > For additional commands, e-mail: dev-h...@spark.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: Cross Validator to work with K-Fold value of 1?

2016-05-02 Thread Julio Antonio Soto de Vicente

Hi,

Same goes for the PolynomialExpansion in org.apache.spark.ml.feature. It would 
be dice to cross-validate with degree 1 polynomial expansion (this is, with no 
expansion at all) vs other degree polynomial expansions. Unfortunately, degree 
is forced to be >= 2.

--
Julio

> El 2 may 2016, a las 9:05, Rahul Tanwani  escribió:
> 
> Hi,
> 
> In certain cases (mostly due to time constraints), we need some model to run
> without cross validation. In such a case, since k-fold value for cross
> validator cannot be one, we have to maintain two different code paths to
> achieve both the scenarios (with and without cross validation).
> 
> Would it be an okay idea to generalize the cross validator so it can work
> with k-fold value of 1? The only purpose for this is to avoid maintaining
> two different code paths and in functionality it should be similar to as if
> the cross validation is not present.
> 
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-spark-developers-list.1001551.n3.nabble.com/Cross-Validator-to-work-with-K-Fold-value-of-1-tp17404.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Ever increasing physical memory for a Spark Application in YARN

2016-05-02 Thread Daniel Darabos

Hi Nitin,
Sorry for waking up this ancient thread. That's a fantastic set of JVM
flags! We just hit the same problem, but we haven't even discovered all
those flags for limiting memory growth. I wanted to ask if you ever
discovered anything further?

I see you also set -XX:NewRatio=3. This is a very important flag since
Spark 1.6.0. With unified memory management with the default
spark.memory.fraction=0.75 the cache will fill up 75% of the heap. The
default NewRatio is 2, so the cache will not fit in the old generation
pool, constantly triggering full GCs. With NewRatio=3 the old generation
pool is 75% of the heap, so it (just) fits the cache. We find this makes a
very significant performance difference in practice.

Perhaps this should be documented somewhere. Or the default
spark.memory.fraction should be 0.66, so that it works out with the default
JVM flags.

On Mon, Jul 27, 2015 at 6:08 PM, Nitin Goyal  wrote:

> I am running a spark application in YARN having 2 executors with Xms/Xmx as
> 32 Gigs and spark.yarn.excutor.memoryOverhead as 6 gigs.
>
> I am seeing that the app's physical memory is ever increasing and finally
> gets killed by node manager
>
> 2015-07-25 15:07:05,354 WARN
>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> Container [pid=10508,containerID=container_1437828324746_0002_01_03] is
> running beyond physical memory limits. Current usage: 38.0 GB of 38 GB
> physical memory used; 39.5 GB of 152 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1437828324746_0002_01_03 :
> |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> |- 10508 9563 10508 10508 (bash) 0 0 9433088 314 /bin/bash -c
> /usr/java/default/bin/java -server -XX:OnOutOfMemoryError='kill %p'
> -Xms32768m -Xmx32768m  -Dlog4j.configuration=log4j-executor.properties
> -XX:MetaspaceSize=512m -XX:+UseG1GC -XX:+PrintGCTimeStamps
> -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:gc.log
> -XX:AdaptiveSizePolicyOutputInterval=1  -XX:+UseGCLogFileRotation
> -XX:GCLogFileSize=500M -XX:NumberOfGCLogFiles=1
> -XX:MaxDirectMemorySize=3500M -XX:NewRatio=3 -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=36082
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false -XX:NativeMemoryTracking=detail
> -XX:ReservedCodeCacheSize=100M -XX:MaxMetaspaceSize=512m
> -XX:CompressedClassSpaceSize=256m
>
> -Djava.io.tmpdir=/data/yarn/datanode/nm-local-dir/usercache/admin/appcache/application_1437828324746_0002/container_1437828324746_0002_01_03/tmp
> '-Dspark.driver.port=43354'
>
> -Dspark.yarn.app.container.log.dir=/opt/hadoop/logs/userlogs/application_1437828324746_0002/container_1437828324746_0002_01_03
> org.apache.spark.executor.CoarseGrainedExecutorBackend
> akka.tcp://sparkDriver@nn1:43354/user/CoarseGrainedScheduler 1 dn3 6
> application_1437828324746_0002 1>
>
> /opt/hadoop/logs/userlogs/application_1437828324746_0002/container_1437828324746_0002_01_03/stdout
> 2>
>
> /opt/hadoop/logs/userlogs/application_1437828324746_0002/container_1437828324746_0002_01_03/stderr
>
>
> I diabled YARN's parameter "yarn.nodemanager.pmem-check-enabled" and
> noticed
> that physical memory usage went till 40 gigs
>
> I checked the total RSS in /proc/pid/smaps and it was same value as
> physical
> memory reported by Yarn and seen in top command.
>
> I checked that its not a problem with the heap but something is increasing
> in off heap/ native memory. I used tools like Visual VM but didn't find
> anything that's increasing there. MaxDirectMmeory also didn't exceed 600MB.
> Peak number of active threads was 70-80 and thread stack size didn't exceed
> 100MB. MetaspaceSize was around 60-70MB.
>
> FYI, I am on Spark 1.2 and Hadoop 2.4.0 and my spark application is based
> on
> Spark SQL and it's an HDFS read/write intensive application and caches data
> in Spark SQL's in-memory caching
>
> Any help would be highly appreciated. Or any hint that where should I look
> to debug memory leak or if any tool already there. Let me know if any other
> information is needed.
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Ever-increasing-physical-memory-for-a-Spark-Application-in-YARN-tp13446.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

Re: [build system] short downtime monday morning (5-2-16), 7-9am PDT

2016-05-02 Thread shane knapp

hey everyone!

looks like two of the workers didn't survive a reboot, so i will need
to head to the colo and console in to see what's going on.

sadly, one of the workers that didn't come back is -01, which runs the
doc builds.

anyways, i will post another update within the hour with the status of
these two machines.  i'm also unpausing builds.

On Mon, May 2, 2016 at 8:26 AM, shane knapp  wrote:
> this is happening now.
>
> On Fri, Apr 29, 2016 at 12:52 PM, shane knapp  wrote:
>> (copy-pasta of previous message)
>>
>> another project hosted on our jenkins (e-mission) needs anaconda scipy
>> upgraded from 0.15.1 to 0.17.0.  this will also upgrade a few other
>> libs, which i've included at the end of this email.
>>
>> i've spoken w/josh @ databricks and we don't believe that this will
>> impact the spark builds at all.  if this causes serious breakage, i
>> will roll everything back to pre-update.
>>
>> i have created a JIRA issue to look in to creating conda environments
>> for spark builds, something that we should have done long ago:
>>
>> https://issues.apache.org/jira/browse/SPARK-14905
>>
>> builds will be paused:  ~7am PDT
>> anaconda package updates:  ~8am
>> jenkins quiet time ends:  ~9am at the latest
>>
>> i do not expect the downtime to last very long, and will update this
>> thread w/updates as they come.
>>
>> here's what will be updated under anaconda:
>>
>> The following NEW packages will be INSTALLED:
>>
>> libgfortran: 3.0-0
>> mkl: 11.3.1-0
>> wheel:   0.29.0-py27_0
>>
>> The following packages will be UPDATED:
>>
>> conda:   3.10.1-py27_0 --> 4.0.5-py27_0
>> conda-env:   2.1.4-py27_0  --> 2.4.5-py27_0
>> numpy:   1.9.2-py27_0  --> 1.11.0-py27_0
>> openssl: 1.0.1k-1  --> 1.0.2g-0
>> pip: 6.1.1-py27_0  --> 8.1.1-py27_1
>> python:  2.7.9-2   --> 2.7.11-0
>> pyyaml:  3.11-py27_0   --> 3.11-py27_1
>> requests:2.6.0-py27_0  --> 2.9.1-py27_0
>> scipy:   0.15.1-np19py27_0 --> 0.17.0-np111py27_2
>> setuptools:  15.0-py27_0   --> 20.7.0-py27_0
>> sqlite:  3.8.4.1-1 --> 3.9.2-0
>> yaml:0.1.4-0   --> 0.1.6-0

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [build system] short downtime monday morning (5-2-16), 7-9am PDT

2016-05-02 Thread shane knapp

this is happening now.

On Fri, Apr 29, 2016 at 12:52 PM, shane knapp  wrote:
> (copy-pasta of previous message)
>
> another project hosted on our jenkins (e-mission) needs anaconda scipy
> upgraded from 0.15.1 to 0.17.0.  this will also upgrade a few other
> libs, which i've included at the end of this email.
>
> i've spoken w/josh @ databricks and we don't believe that this will
> impact the spark builds at all.  if this causes serious breakage, i
> will roll everything back to pre-update.
>
> i have created a JIRA issue to look in to creating conda environments
> for spark builds, something that we should have done long ago:
>
> https://issues.apache.org/jira/browse/SPARK-14905
>
> builds will be paused:  ~7am PDT
> anaconda package updates:  ~8am
> jenkins quiet time ends:  ~9am at the latest
>
> i do not expect the downtime to last very long, and will update this
> thread w/updates as they come.
>
> here's what will be updated under anaconda:
>
> The following NEW packages will be INSTALLED:
>
> libgfortran: 3.0-0
> mkl: 11.3.1-0
> wheel:   0.29.0-py27_0
>
> The following packages will be UPDATED:
>
> conda:   3.10.1-py27_0 --> 4.0.5-py27_0
> conda-env:   2.1.4-py27_0  --> 2.4.5-py27_0
> numpy:   1.9.2-py27_0  --> 1.11.0-py27_0
> openssl: 1.0.1k-1  --> 1.0.2g-0
> pip: 6.1.1-py27_0  --> 8.1.1-py27_1
> python:  2.7.9-2   --> 2.7.11-0
> pyyaml:  3.11-py27_0   --> 3.11-py27_1
> requests:2.6.0-py27_0  --> 2.9.1-py27_0
> scipy:   0.15.1-np19py27_0 --> 0.17.0-np111py27_2
> setuptools:  15.0-py27_0   --> 20.7.0-py27_0
> sqlite:  3.8.4.1-1 --> 3.9.2-0
> yaml:0.1.4-0   --> 0.1.6-0

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Requesting feedback for PR for SPARK-11962

2016-05-02 Thread Arun Allamsetty

Hi,

Since the 2.0.0 branch has been created and is now nearing feature freeze,
can SPARK-11962 get some love please. If we can decide if this should go
into 2.0.0 or 2.1.0, that would be great. Personally, I feel it can totally
go into 2.0.0 as the code is pretty much ready (except for the one bug that
I need your help with).

Thanks,
Arun

On Fri, Apr 29, 2016 at 1:00 PM, Arun Allamsetty 
wrote:

> Hi,
>
> I have submitted a PR for SPARK-11962 (
> https://github.com/apache/spark/pull/12708). I have most of it ready
> except I am not able to pin point the cause for a particular bug.
>
> In the PR I've added two major methods to Row, `attempt` and `getOption`.
> The former returns a `Try` while the latter returns an `Option`.
>
>- `attempt` was added after a comment was made in PR #10247
>, where it was suggested I
>not return `None` when certain exceptions are thrown. But in my
>opinion, throwing exceptions from a function which returns an `Option`
>is not a good use case.
>- I am not in love with the method name, `attempt`. Would welcome
>suggestions. I wanted to use `try` but it's a keyword.
>- So about the failing tests (which shouldn't fail in my opinion), in
>`RowTest`, all new tests testing for `ClassCastException` fail. I have
>modified tests as comments in the code blocks (with TODOs) which work, but
>should behave the same way as the actual test from what I can see. I am
>not sure what I am doing wrong in the code here.
>
> Would appreciate some feedback.
>
> Thanks,
>
> Arun
>
> P. S. I have sent similar emails to the dev mailing list before but I
> don't see them in the dev archives. I am guessing they were not received as
> I was not subscribed to the dev list. Hopefully this would work as I have
> subscribed now.
>

Re: spark 2 segfault

2016-05-02 Thread Koert Kuipers

Created issue:
https://issues.apache.org/jira/browse/SPARK-15062

On Mon, May 2, 2016 at 6:48 AM, Ted Yu  wrote:

> I tried the same statement using Spark 1.6.1
> There was no error with default memory setting.
>
> Suggest logging a bug.
>
> On May 1, 2016, at 9:22 PM, Koert Kuipers  wrote:
>
> Yeah I got that too, then I increased heap for tests to 8G to get error I
> showed earlier.
> On May 2, 2016 12:09 AM, "Ted Yu"  wrote:
>
>> Using commit hash 90787de864b58a1079c23e6581381ca8ffe7685f and
>> Java 1.7.0_67 , I got:
>>
>> scala> val dfComplicated = sc.parallelize(List((Map("1" -> "a"),
>> List("b", "c")), (Map("2" -> "b"), List("d", "e".toDF
>> ...
>> dfComplicated: org.apache.spark.sql.DataFrame = [_1: map,
>> _2: array]
>>
>> scala> dfComplicated.show
>> java.lang.OutOfMemoryError: Java heap space
>>   at
>> org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:229)
>>   at
>> org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:821)
>>   at
>> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown
>> Source)
>>   at
>> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:241)
>>   at
>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121)
>>   at
>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121)
>>   at
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>>   at
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>>   at
>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>>   at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
>>   at
>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2121)
>>   at
>> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:54)
>>   at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2408)
>>   at org.apache.spark.sql.Dataset.org
>> $apache$spark$sql$Dataset$$execute$1(Dataset.scala:2120)
>>   at org.apache.spark.sql.Dataset.org
>> $apache$spark$sql$Dataset$$collect(Dataset.scala:2127)
>>   at
>> org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1861)
>>   at
>> org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1860)
>>   at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2438)
>>   at org.apache.spark.sql.Dataset.head(Dataset.scala:1860)
>>   at org.apache.spark.sql.Dataset.take(Dataset.scala:2077)
>>   at org.apache.spark.sql.Dataset.showString(Dataset.scala:238)
>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:529)
>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:489)
>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:498)
>>   ... 6 elided
>>
>> scala>
>>
>> On Sun, May 1, 2016 at 8:34 PM, Koert Kuipers  wrote:
>>
>>> by removing dependencies it turns into a different error, see below.
>>> the test is simply writing a DataFrame out to file and reading it back
>>> in. i see the error for all data sources (json, parquet, etc.).
>>>
>>> this is the data that i write out and read back in:
>>> val dfComplicated = sc.parallelize(List((Map("1" -> "a"), List("b",
>>> "c")), (Map("2" -> "b"), List("d", "e".toDF
>>>
>>>
>>> [info]   java.lang.RuntimeException: Error while decoding:
>>> java.lang.NegativeArraySizeException
>>> [info] createexternalrow(if (isnull(input[0, map])) null
>>> else staticinvoke(class
>>> org.apache.spark.sql.catalyst.util.ArrayBasedMapData$, ObjectType(interface
>>> scala.collection.Map), toScalaMap, staticinvoke(class
>>> scala.collection.mutable.WrappedArray$, ObjectType(interface
>>> scala.collection.Seq), make,
>>> mapobjects(lambdavariable(MapObjects_loopValue16, MapObjects_loopIsNull17,
>>> StringType), lambdavariable(MapObjects_loopValue16,
>>> MapObjects_loopIsNull17, StringType).toString, input[0,
>>> map].keyArray).array, true), staticinvoke(class
>>> scala.collection.mutable.WrappedArray$, ObjectType(interface
>>> scala.collection.Seq), make,
>>> mapobjects(lambdavariable(MapObjects_loopValue18, MapObjects_loopIsNull19,
>>> StringType), lambdavariable(MapObjects_loopValue18,
>>> MapObjects_loopIsNull19, StringType).toString, input[0,
>>> map].valueArray).array, true), true), if (isnull(input[1,
>>> array])) null else staticinvoke(class
>>> scala.collection.mutable.WrappedArray$, ObjectType(interface
>>> scala.collection.Seq), make,
>>> mapobjects(lambdavariable(MapObjects_loopValue20, MapObjects_loopIsNull21,
>>> StringType),

Re: [ANNOUNCE] Spark branch-2.0

2016-05-02 Thread Pete Robbins

https://issues.apache.org/jira/browse/SPARK-13745

is really a defect and a blocker unless it is the decision to drop support
for Big Endian platforms. The PR has been reviewed and tested and I
strongly believe this needs to be targeted for 2.0.

On Mon, May 2, 2016 at 12:00 AM Reynold Xin  wrote:

> Hi devs,
>
> Three weeks ago I mentioned on the dev list creating branch-2.0
> (effectively "feature freeze") in 2 - 3 weeks. I've just created Spark's
> branch-2.0 to form the basis of the 2.0 release. We have closed ~ 1700
> issues. That's huge progress, and we should celebrate that.
>
> Compared with past releases when we cut the release branch, we have way
> fewer open issues. In the past we usually have 200 - 400 open issues when
> we cut the release branch. As of today we have less than 100 open issues
> for 2.0.0, and among these 14 critical and 2 blocker (Jersey dependency
> upgrade and some remaining issues in separating out local linear algebra
> library).
>
> What does this mean for committers?
>
> 0. For patches that should go into Spark 2.0.0, make sure you also merge
> them into not just master, but also branch-2.0.
>
> 1. In the next couple of days, sheppard some of the more important,
> straggler pull requests in.
>
> 2. Switch the focus from new feature development to bug fixes, stability
> improvements, finalizing API tweaks, and documentation.
>
> 3. Experimental features (e.g. R, structured streaming) can continue to be
> developed, provided that the changes don't impact the non-experimental
> features.
>
> 4. We should become increasingly conservative as time goes on, even for
> experimental features.
>
> 5. Please un-target or re-target issues if they don't make sense for 2.0.
> We should burn # issues down to ~ 0 by the time we have a release candidate.
>
> 7. If possible, reach out to users and start testing branch-2.0 to find
> bugs. The more testing we can do on real workloads before the release, the
> less bugs we will find in the actual Spark 2.0 release.
>
>
>
>
>

Re: spark 2 segfault

2016-05-02 Thread Ted Yu

I tried the same statement using Spark 1.6.1
There was no error with default memory setting. 

Suggest logging a bug. 

> On May 1, 2016, at 9:22 PM, Koert Kuipers  wrote:
> 
> Yeah I got that too, then I increased heap for tests to 8G to get error I 
> showed earlier.
> 
>> On May 2, 2016 12:09 AM, "Ted Yu"  wrote:
>> Using commit hash 90787de864b58a1079c23e6581381ca8ffe7685f and Java 1.7.0_67 
>> , I got:
>> 
>> scala> val dfComplicated = sc.parallelize(List((Map("1" -> "a"), List("b", 
>> "c")), (Map("2" -> "b"), List("d", "e".toDF
>> ...
>> dfComplicated: org.apache.spark.sql.DataFrame = [_1: map, _2: 
>> array]
>> 
>> scala> dfComplicated.show
>> java.lang.OutOfMemoryError: Java heap space
>>   at org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:229)
>>   at org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:821)
>>   at 
>> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown
>>  Source)
>>   at 
>> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:241)
>>   at 
>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121)
>>   at 
>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1$$anonfun$apply$13.apply(Dataset.scala:2121)
>>   at 
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>>   at 
>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>>   at 
>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>>   at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
>>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>>   at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
>>   at 
>> org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2121)
>>   at 
>> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:54)
>>   at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2408)
>>   at 
>> org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2120)
>>   at 
>> org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2127)
>>   at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1861)
>>   at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:1860)
>>   at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2438)
>>   at org.apache.spark.sql.Dataset.head(Dataset.scala:1860)
>>   at org.apache.spark.sql.Dataset.take(Dataset.scala:2077)
>>   at org.apache.spark.sql.Dataset.showString(Dataset.scala:238)
>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:529)
>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:489)
>>   at org.apache.spark.sql.Dataset.show(Dataset.scala:498)
>>   ... 6 elided
>> 
>> scala>
>> 
>>> On Sun, May 1, 2016 at 8:34 PM, Koert Kuipers  wrote:
>>> by removing dependencies it turns into a different error, see below.
>>> the test is simply writing a DataFrame out to file and reading it back in. 
>>> i see the error for all data sources (json, parquet, etc.).
>>> 
>>> this is the data that i write out and read back in:
>>> val dfComplicated = sc.parallelize(List((Map("1" -> "a"), List("b", "c")), 
>>> (Map("2" -> "b"), List("d", "e".toDF 
>>> 
>>> 
>>> [info]   java.lang.RuntimeException: Error while decoding: 
>>> java.lang.NegativeArraySizeException
>>> [info] createexternalrow(if (isnull(input[0, map])) null 
>>> else staticinvoke(class 
>>> org.apache.spark.sql.catalyst.util.ArrayBasedMapData$, ObjectType(interface 
>>> scala.collection.Map), toScalaMap, staticinvoke(class 
>>> scala.collection.mutable.WrappedArray$, ObjectType(interface 
>>> scala.collection.Seq), make, 
>>> mapobjects(lambdavariable(MapObjects_loopValue16, MapObjects_loopIsNull17, 
>>> StringType), lambdavariable(MapObjects_loopValue16, 
>>> MapObjects_loopIsNull17, StringType).toString, input[0, 
>>> map].keyArray).array, true), staticinvoke(class 
>>> scala.collection.mutable.WrappedArray$, ObjectType(interface 
>>> scala.collection.Seq), make, 
>>> mapobjects(lambdavariable(MapObjects_loopValue18, MapObjects_loopIsNull19, 
>>> StringType), lambdavariable(MapObjects_loopValue18, 
>>> MapObjects_loopIsNull19, StringType).toString, input[0, 
>>> map].valueArray).array, true), true), if (isnull(input[1, 
>>> array])) null else staticinvoke(class 
>>> scala.collection.mutable.WrappedArray$, ObjectType(interface 
>>> scala.collection.Seq), make, 
>>> mapobjects(lambdavariable(MapObjects_loopValue20, MapObjects_loopIsNull21, 
>>> StringType), lambdavariable(MapObjects_loopValue20, 
>>> MapObjects_loopIsNull21, StringType).toString, input[1, 
>>>

Cross Validator to work with K-Fold value of 1?

2016-05-02 Thread Rahul Tanwani

Hi,

In certain cases (mostly due to time constraints), we need some model to run
without cross validation. In such a case, since k-fold value for cross
validator cannot be one, we have to maintain two different code paths to
achieve both the scenarios (with and without cross validation).

Would it be an okay idea to generalize the cross validator so it can work
with k-fold value of 1? The only purpose for this is to avoid maintaining
two different code paths and in functionality it should be similar to as if
the cross validation is not present.





--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/Cross-Validator-to-work-with-K-Fold-value-of-1-tp17404.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [build system] short downtime monday morning (5-2-16), 7-9am PDT

Re: [build system] short downtime monday morning (5-2-16), 7-9am PDT

Re: spark 2 segfault

Re: spark 2 segfault

Re: Cross Validator to work with K-Fold value of 1?

Re: Cross Validator to work with K-Fold value of 1?

Re: Ever increasing physical memory for a Spark Application in YARN

Re: [build system] short downtime monday morning (5-2-16), 7-9am PDT

Re: [build system] short downtime monday morning (5-2-16), 7-9am PDT

Re: Requesting feedback for PR for SPARK-11962

Re: spark 2 segfault

Re: [ANNOUNCE] Spark branch-2.0

Re: spark 2 segfault

Cross Validator to work with K-Fold value of 1?

14 matches

Site Navigation

Mail list logo

Footer information