[jira] [Created] (SPARK-10361) model.predictAll() fails at user_product.first()
Velu nambi created SPARK-10361: -- Summary: model.predictAll() fails at user_product.first() Key: SPARK-10361 URL: https://issues.apache.org/jira/browse/SPARK-10361 Project: Spark Issue Type: Bug Components: MLlib, PySpark Affects Versions: 1.4.1, 1.3.1, 1.5.0 Environment: Windows 10, Python 2.7 and with all the three versions of Spark Reporter: Velu nambi This code, adapted from the documentation, fails when calling PredictAll() after an ALS.train() 15/08/31 00:11:45 ERROR PythonRDD: Python worker exited unexpectedly (crashed) java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at java.io.BufferedOutputStream.write(Unknown Source) at java.io.DataOutputStream.write(Unknown Source) at java.io.FilterOutputStream.write(Unknown Source) at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) at org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772) at org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208) 15/08/31 00:11:45 ERROR PythonRDD: This may have been caused by a prior exception: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at java.io.BufferedOutputStream.write(Unknown Source) at java.io.DataOutputStream.write(Unknown Source) at java.io.FilterOutputStream.write(Unknown Source) at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) at org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772) at org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208) 15/08/31 00:11:45 ERROR Executor: Exception in task 0.0 in stage 187.0 (TID 85) java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown Source) at java.io.BufferedOutputStream.write(Unknown Source) at java.io.DataOutputStream.write(Unknown Source) at java.io.FilterOutputStream.write(Unknown Source) at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) at org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772) at org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208) 15/08/31 00:11:45 WARN TaskSetManager: Lost task 0.0 in stage 187.0 (TID 85, localhost): java.net.SocketException: Connection reset by peer: socket write error at
[jira] [Commented] (SPARK-10361) model.predictAll() fails at user_product.first()
[ https://issues.apache.org/jira/browse/SPARK-10361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723621#comment-14723621 ] Velu nambi commented on SPARK-10361: Thanks [~srowen]. Is this a known issue, any suggestions ? > model.predictAll() fails at user_product.first() > > > Key: SPARK-10361 > URL: https://issues.apache.org/jira/browse/SPARK-10361 > Project: Spark > Issue Type: Bug > Components: MLlib, PySpark >Affects Versions: 1.3.1, 1.4.1, 1.5.0 > Environment: Windows 10, Python 2.7 and with all the three versions > of Spark >Reporter: Velu nambi > > This code, adapted from the documentation, fails when calling PredictAll() > after an ALS.train() > 15/08/31 00:11:45 ERROR PythonRDD: Python worker exited unexpectedly (crashed) > java.net.SocketException: Connection reset by peer: socket write error > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(Unknown Source) > at java.net.SocketOutputStream.write(Unknown Source) > at java.io.BufferedOutputStream.write(Unknown Source) > at java.io.DataOutputStream.write(Unknown Source) > at java.io.FilterOutputStream.write(Unknown Source) > at > org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at > org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) > at > org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772) > at > org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208) > 15/08/31 00:11:45 ERROR PythonRDD: This may have been caused by a prior > exception: > java.net.SocketException: Connection reset by peer: socket write error > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(Unknown Source) > at java.net.SocketOutputStream.write(Unknown Source) > at java.io.BufferedOutputStream.write(Unknown Source) > at java.io.DataOutputStream.write(Unknown Source) > at java.io.FilterOutputStream.write(Unknown Source) > at > org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at > org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) > at > org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772) > at > org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208) > 15/08/31 00:11:45 ERROR Executor: Exception in task 0.0 in stage 187.0 (TID > 85) > java.net.SocketException: Connection reset by peer: socket write error > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(Unknown Source) > at java.net.SocketOutputStream.write(Unknown Source) > at java.io.BufferedOutputStream.write(Unknown Source) > at java.io.DataOutputStream.write(Unknown Source) > at java.io.FilterOutputStream.write(Unknown Source) > at > org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at > org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) > at > org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248) > at
[jira] [Commented] (SPARK-10361) model.predictAll() fails at user_product.first()
[ https://issues.apache.org/jira/browse/SPARK-10361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723162#comment-14723162 ] Velu nambi commented on SPARK-10361: [~srowen] I'm running a standalone version of spark on windows. I didn't see any process crash or anything suspicious in the firewall logs -- let me know if I'm missing something ? > model.predictAll() fails at user_product.first() > > > Key: SPARK-10361 > URL: https://issues.apache.org/jira/browse/SPARK-10361 > Project: Spark > Issue Type: Bug > Components: MLlib, PySpark >Affects Versions: 1.3.1, 1.4.1, 1.5.0 > Environment: Windows 10, Python 2.7 and with all the three versions > of Spark >Reporter: Velu nambi > > This code, adapted from the documentation, fails when calling PredictAll() > after an ALS.train() > 15/08/31 00:11:45 ERROR PythonRDD: Python worker exited unexpectedly (crashed) > java.net.SocketException: Connection reset by peer: socket write error > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(Unknown Source) > at java.net.SocketOutputStream.write(Unknown Source) > at java.io.BufferedOutputStream.write(Unknown Source) > at java.io.DataOutputStream.write(Unknown Source) > at java.io.FilterOutputStream.write(Unknown Source) > at > org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at > org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) > at > org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772) > at > org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208) > 15/08/31 00:11:45 ERROR PythonRDD: This may have been caused by a prior > exception: > java.net.SocketException: Connection reset by peer: socket write error > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(Unknown Source) > at java.net.SocketOutputStream.write(Unknown Source) > at java.io.BufferedOutputStream.write(Unknown Source) > at java.io.DataOutputStream.write(Unknown Source) > at java.io.FilterOutputStream.write(Unknown Source) > at > org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at > org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) > at > org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:248) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1772) > at > org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:208) > 15/08/31 00:11:45 ERROR Executor: Exception in task 0.0 in stage 187.0 (TID > 85) > java.net.SocketException: Connection reset by peer: socket write error > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(Unknown Source) > at java.net.SocketOutputStream.write(Unknown Source) > at java.io.BufferedOutputStream.write(Unknown Source) > at java.io.DataOutputStream.write(Unknown Source) > at java.io.FilterOutputStream.write(Unknown Source) > at > org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:413) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at > org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:425) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at > org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28) > at > org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:425) > at >
[jira] [Created] (SPARK-10319) ALS training using PySpark throws a StackOverflowError
Velu nambi created SPARK-10319:
--
Summary: ALS training using PySpark throws a StackOverflowError
Key: SPARK-10319
URL: https://issues.apache.org/jira/browse/SPARK-10319
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.4.1
Environment: Windows 10, spark - 1.4.1,
Reporter: Velu nambi
When attempting to train a machine learning model using ALS in Spark's MLLib
(1.4) on windows, Pyspark always terminates with a StackoverflowError. I tried
adding the checkpoint as described in http://stackoverflow.com/a/31484461/36130
-- doesn't seem to help.
Here's the training code and stack trace:
{code:none}
ranks = [8, 12]
lambdas = [0.1, 10.0]
numIters = [10, 20]
bestModel = None
bestValidationRmse = float(inf)
bestRank = 0
bestLambda = -1.0
bestNumIter = -1
for rank, lmbda, numIter in itertools.product(ranks, lambdas, numIters):
ALS.checkpointInterval = 2
model = ALS.train(training, rank, numIter, lmbda)
validationRmse = computeRmse(model, validation, numValidation)
if (validationRmse bestValidationRmse):
bestModel = model
bestValidationRmse = validationRmse
bestRank = rank
bestLambda = lmbda
bestNumIter = numIter
testRmse = computeRmse(bestModel, test, numTest)
{code}
Stacktrace:
15/08/27 02:02:58 ERROR Executor: Exception in task 3.0 in stage 56.0 (TID 127)
java.lang.StackOverflowError
at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source)
at java.io.ObjectInputStream.readHandle(Unknown Source)
at java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10319) ALS training using PySpark throws a StackOverflowError
[
https://issues.apache.org/jira/browse/SPARK-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717270#comment-14717270
]
Velu nambi commented on SPARK-10319:
bq. do you see evidence of checkpointing in the logs?
Yes, I see a few files created in the Checkpoint directory.
ALS training using PySpark throws a StackOverflowError
--
Key: SPARK-10319
URL: https://issues.apache.org/jira/browse/SPARK-10319
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 1.4.1
Environment: Windows 10, spark - 1.4.1,
Reporter: Velu nambi
When attempting to train a machine learning model using ALS in Spark's MLLib
(1.4) on windows, Pyspark always terminates with a StackoverflowError. I
tried adding the checkpoint as described in
http://stackoverflow.com/a/31484461/36130 -- doesn't seem to help.
Here's the training code and stack trace:
{code:none}
ranks = [8, 12]
lambdas = [0.1, 10.0]
numIters = [10, 20]
bestModel = None
bestValidationRmse = float(inf)
bestRank = 0
bestLambda = -1.0
bestNumIter = -1
for rank, lmbda, numIter in itertools.product(ranks, lambdas, numIters):
ALS.checkpointInterval = 2
model = ALS.train(training, rank, numIter, lmbda)
validationRmse = computeRmse(model, validation, numValidation)
if (validationRmse bestValidationRmse):
bestModel = model
bestValidationRmse = validationRmse
bestRank = rank
bestLambda = lmbda
bestNumIter = numIter
testRmse = computeRmse(bestModel, test, numTest)
{code}
Stacktrace:
15/08/27 02:02:58 ERROR Executor: Exception in task 3.0 in stage 56.0 (TID
127)
java.lang.StackOverflowError
at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source)
at java.io.ObjectInputStream.readHandle(Unknown Source)
at java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-10319) ALS training using PySpark throws a StackOverflowError
[ https://issues.apache.org/jira/browse/SPARK-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717263#comment-14717263 ] Velu nambi edited comment on SPARK-10319 at 8/27/15 6:42 PM: - Yes it does seem similar to SPARK-5955, it works when I reduce the iterations to [5,10] (currently set to [10,20]). Here is the small stack trace from top of the stack, let me know 5/08/27 10:35:07 INFO DAGScheduler: Job 12 failed: count at ALS.scala:243, took 3.083999 s Traceback (most recent call last): File C:\Program Files (x86)\JetBrains\PyCharm Community Edition 4.5.3\helpers\pydev\pydevd.py, line 2358, in module globals = debugger.run(setup['file'], None, None, is_module) File C:\Program Files (x86)\JetBrains\PyCharm Community Edition 4.5.3\helpers\pydev\pydevd.py, line 1778, in run pydev_imports.execfile(file, globals, locals) # execute the script File C:/Users/PycharmProjects/MovieLensALS/MovieLensALS.py, line 129, in module model = ALS.train(training, rank, numIter, lmbda) File C:\spark-1.4.1\python\pyspark\mllib\recommendation.py, line 194, in train lambda_, blocks, nonnegative, seed) File C:\spark-1.4.1\python\pyspark\mllib\common.py, line 128, in callMLlibFunc return callJavaFunc(sc, api, *args) File C:\spark-1.4.1\python\pyspark\mllib\common.py, line 121, in callJavaFunc return _java2py(sc, func(*args)) File C:\Users\PyCharmVirtualEnv\MovieLensALSVirtEnv\lib\site-packages\py4j\java_gateway.py, line 813, in __call__ answer, self.gateway_client, self.target_id, self.name) File C:\Users\PyCharmVirtualEnv\MovieLensALSVirtEnv\lib\site-packages\py4j\protocol.py, line 308, in get_return_value format(target_id, ., name), value) py4j.protocol.Py4JJavaError: An error occurred while calling o145.trainALSModel. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 56.0 failed 1 times, most recent failure: Lost task 0.0 in stage 56.0 (TID 124, localhost): java.lang.StackOverflowError at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source) at java.io.ObjectInputStream.readHandle(Unknown Source) at java.io.ObjectInputStream.readClassDesc(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.$colon$colon.readObject(List.scala:362) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.$colon$colon.readObject(List.scala:366) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at
[jira] [Commented] (SPARK-10319) ALS training using PySpark throws a StackOverflowError
[ https://issues.apache.org/jira/browse/SPARK-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717263#comment-14717263 ] Velu nambi commented on SPARK-10319: Yes it does seem similar to SPARK-5955, it works when I set the reduce the iterations to [5,10] (currently set to [10,20]). Here is the small stack trace from top of the stack, let me know 5/08/27 10:35:07 INFO DAGScheduler: Job 12 failed: count at ALS.scala:243, took 3.083999 s Traceback (most recent call last): File C:\Program Files (x86)\JetBrains\PyCharm Community Edition 4.5.3\helpers\pydev\pydevd.py, line 2358, in module globals = debugger.run(setup['file'], None, None, is_module) File C:\Program Files (x86)\JetBrains\PyCharm Community Edition 4.5.3\helpers\pydev\pydevd.py, line 1778, in run pydev_imports.execfile(file, globals, locals) # execute the script File C:/Users/PycharmProjects/MovieLensALS/MovieLensALS.py, line 129, in module model = ALS.train(training, rank, numIter, lmbda) File C:\spark-1.4.1\python\pyspark\mllib\recommendation.py, line 194, in train lambda_, blocks, nonnegative, seed) File C:\spark-1.4.1\python\pyspark\mllib\common.py, line 128, in callMLlibFunc return callJavaFunc(sc, api, *args) File C:\spark-1.4.1\python\pyspark\mllib\common.py, line 121, in callJavaFunc return _java2py(sc, func(*args)) File C:\Users\PyCharmVirtualEnv\MovieLensALSVirtEnv\lib\site-packages\py4j\java_gateway.py, line 813, in __call__ answer, self.gateway_client, self.target_id, self.name) File C:\Users\PyCharmVirtualEnv\MovieLensALSVirtEnv\lib\site-packages\py4j\protocol.py, line 308, in get_return_value format(target_id, ., name), value) py4j.protocol.Py4JJavaError: An error occurred while calling o145.trainALSModel. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 56.0 failed 1 times, most recent failure: Lost task 0.0 in stage 56.0 (TID 124, localhost): java.lang.StackOverflowError at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source) at java.io.ObjectInputStream.readHandle(Unknown Source) at java.io.ObjectInputStream.readClassDesc(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.$colon$colon.readObject(List.scala:362) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at scala.collection.immutable.$colon$colon.readObject(List.scala:366) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at java.io.ObjectStreamClass.invokeReadObject(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.io.ObjectInputStream.readSerialData(Unknown Source) at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at
