[
https://issues.apache.org/jira/browse/SPARK-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717263#comment-14717263
]
Velu nambi commented on SPARK-10319:
------------------------------------
Yes it does seem similar to SPARK-5955, it works when I set the reduce the
iterations to [5,10] (currently set to [10,20]).
Here is the small stack trace from top of the stack, let me know
5/08/27 10:35:07 INFO DAGScheduler: Job 12 failed: count at ALS.scala:243, took
3.083999 s
Traceback (most recent call last):
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition
4.5.3\helpers\pydev\pydevd.py", line 2358, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition
4.5.3\helpers\pydev\pydevd.py", line 1778, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:/Users/PycharmProjects/MovieLensALS/MovieLensALS.py", line 129, in
<module>
model = ALS.train(training, rank, numIter, lmbda)
File "C:\spark-1.4.1\python\pyspark\mllib\recommendation.py", line 194, in
train
lambda_, blocks, nonnegative, seed)
File "C:\spark-1.4.1\python\pyspark\mllib\common.py", line 128, in
callMLlibFunc
return callJavaFunc(sc, api, *args)
File "C:\spark-1.4.1\python\pyspark\mllib\common.py", line 121, in
callJavaFunc
return _java2py(sc, func(*args))
File
"C:\Users\PyCharmVirtualEnv\MovieLensALSVirtEnv\lib\site-packages\py4j\java_gateway.py",
line 813, in __call__
answer, self.gateway_client, self.target_id, self.name)
File
"C:\Users\PyCharmVirtualEnv\MovieLensALSVirtEnv\lib\site-packages\py4j\protocol.py",
line 308, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o145.trainALSModel.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in
stage 56.0 failed 1 times, most recent failure: Lost task 0.0 in stage 56.0
(TID 124, localhost): java.lang.StackOverflowError
at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown
Source)
at java.io.ObjectInputStream.readHandle(Unknown Source)
at java.io.ObjectInputStream.readClassDesc(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
at java.io.ObjectInputStream.readObject(Unknown Source)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
at java.io.ObjectInputStream.readSerialData(Unknown Source)
at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
at java.io.ObjectInputStream.readObject0(Unknown Source)
> ALS training using PySpark throws a StackOverflowError
> ------------------------------------------------------
>
> Key: SPARK-10319
> URL: https://issues.apache.org/jira/browse/SPARK-10319
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.4.1
> Environment: Windows 10, spark - 1.4.1,
> Reporter: Velu nambi
>
> When attempting to train a machine learning model using ALS in Spark's MLLib
> (1.4) on windows, Pyspark always terminates with a StackoverflowError. I
> tried adding the checkpoint as described in
> http://stackoverflow.com/a/31484461/36130 -- doesn't seem to help.
> Here's the training code and stack trace:
> {code:none}
> ranks = [8, 12]
> lambdas = [0.1, 10.0]
> numIters = [10, 20]
> bestModel = None
> bestValidationRmse = float("inf")
> bestRank = 0
> bestLambda = -1.0
> bestNumIter = -1
> for rank, lmbda, numIter in itertools.product(ranks, lambdas, numIters):
> ALS.checkpointInterval = 2
> model = ALS.train(training, rank, numIter, lmbda)
> validationRmse = computeRmse(model, validation, numValidation)
> if (validationRmse < bestValidationRmse):
> bestModel = model
> bestValidationRmse = validationRmse
> bestRank = rank
> bestLambda = lmbda
> bestNumIter = numIter
> testRmse = computeRmse(bestModel, test, numTest)
> {code}
> Stacktrace:
> 15/08/27 02:02:58 ERROR Executor: Exception in task 3.0 in stage 56.0 (TID
> 127)
> java.lang.StackOverflowError
> at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source)
> at java.io.ObjectInputStream.readHandle(Unknown Source)
> at java.io.ObjectInputStream.readClassDesc(Unknown Source)
> at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
> at java.io.ObjectInputStream.readObject0(Unknown Source)
> at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
> at java.io.ObjectInputStream.readSerialData(Unknown Source)
> at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
> at java.io.ObjectInputStream.readObject0(Unknown Source)
> at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
> at java.io.ObjectInputStream.readSerialData(Unknown Source)
> at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
> at java.io.ObjectInputStream.readObject0(Unknown Source)
> at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
> at java.io.ObjectInputStream.readSerialData(Unknown Source)
> at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
> at java.io.ObjectInputStream.readObject0(Unknown Source)
> at java.io.ObjectInputStream.readObject(Unknown Source)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at java.io.ObjectStreamClass.invokeReadObject(Unknown Source)
> at java.io.ObjectInputStream.readSerialData(Unknown Source)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
