Hello Sooraj, I see you are using ipython notebook. Can you tell me are you on Windows OS or Linux based OS? I am using Windows 7 and I am new to Spark. I am trying to connect ipython with my local cluster based on CDH5.4. I followed these tutorials here but they are written on linux environment and hence not much help to me. I am able to launch ipython on localhost but cannot get it to work on the cluster
Sincerely, Ashish Dutt On Wed, Jul 8, 2015 at 5:49 PM, sooraj <[email protected]> wrote: > That turned out to be a silly data type mistake. At one point in the > iterative call, I was passing an integer value for the parameter 'alpha' of > the ALS train API, which was expecting a Double. So, py4j in fact > complained that it cannot take a method that takes an integer value for > that parameter. > > On 8 July 2015 at 12:35, sooraj <[email protected]> wrote: > >> Hi, >> >> I am using MLlib collaborative filtering API on an implicit preference >> data set. From a pySpark notebook, I am iteratively creating the matrix >> factorization model with the aim of measuring the RMSE for each combination >> of parameters for this API like the rank, lambda and alpha. After the code >> successfully completed six iterations, on the seventh call of the >> ALS.trainImplicit API, I get a confusing exception that says py4j cannot >> find the method trainImplicitALSmodel. The full trace is included at the >> end of the email. >> >> I am running Spark over YARN (yarn-client mode) with five executors. This >> error seems to be happening completely on the driver as I don't see any >> error on the Spark web interface. I have tried changing the >> spark.yarn.am.memory configuration value, but it doesn't help. Any >> suggestion on how to debug this will be very helpful. >> >> Thank you, >> Sooraj >> >> Here is the full error trace: >> >> ---------------------------------------------------------------------------Py4JError >> Traceback (most recent call >> last)<ipython-input-8-ad6ca35e7521> in <module>() 3 4 for index, >> (r, l, a, i) in enumerate(itertools.product(ranks, lambdas, alphas, >> iters)):----> 5 model = ALS.trainImplicit(scoreTableTrain, rank = r, >> iterations = i, lambda_ = l, alpha = a) 6 7 predictionsTrain >> = model.predictAll(userProductTrainRDD) >> /usr/local/spark-1.4/spark-1.4.0-bin-hadoop2.6/python/pyspark/mllib/recommendation.pyc >> in trainImplicit(cls, ratings, rank, iterations, lambda_, blocks, alpha, >> nonnegative, seed) 198 nonnegative=False, >> seed=None): 199 model = callMLlibFunc("trainImplicitALSModel", >> cls._prepare(ratings), rank,--> 200 >> iterations, lambda_, blocks, alpha, nonnegative, seed) 201 return >> MatrixFactorizationModel(model) 202 >> /usr/local/spark-1.4/spark-1.4.0-bin-hadoop2.6/python/pyspark/mllib/common.pyc >> in callMLlibFunc(name, *args) 126 sc = >> SparkContext._active_spark_context 127 api = >> getattr(sc._jvm.PythonMLLibAPI(), name)--> 128 return callJavaFunc(sc, >> api, *args) 129 130 >> /usr/local/spark-1.4/spark-1.4.0-bin-hadoop2.6/python/pyspark/mllib/common.pyc >> in callJavaFunc(sc, func, *args) 119 """ Call Java Function """ >> 120 args = [_py2java(sc, a) for a in args]--> 121 return >> _java2py(sc, func(*args)) 122 123 >> /usr/local/lib/python2.7/site-packages/py4j/java_gateway.pyc in >> __call__(self, *args) 536 answer = >> self.gateway_client.send_command(command) 537 return_value = >> get_return_value(answer, self.gateway_client,--> 538 >> self.target_id, self.name) 539 540 for temp_arg in temp_args: >> /usr/local/lib/python2.7/site-packages/py4j/protocol.pyc in >> get_return_value(answer, gateway_client, target_id, name) 302 >> raise Py4JError( 303 'An error occurred while >> calling {0}{1}{2}. Trace:\n{3}\n'.--> 304 >> format(target_id, '.', name, value)) 305 else: 306 >> raise Py4JError( >> Py4JError: An error occurred while calling o667.trainImplicitALSModel. Trace: >> py4j.Py4JException: Method trainImplicitALSModel([class >> org.apache.spark.api.java.JavaRDD, class java.lang.Integer, class >> java.lang.Integer, class java.lang.Integer, class java.lang.Integer, class >> java.lang.Double, class java.lang.Boolean, null]) does not exist >> at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:333) >> at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342) >> at py4j.Gateway.invoke(Gateway.java:252) >> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) >> at py4j.commands.CallCommand.execute(CallCommand.java:79) >> at py4j.GatewayConnection.run(GatewayConnection.java:207) >> at java.lang.Thread.run(Thread.java:724) >> >> >> >
