The compiled jar is not consistent with Python source, maybe you are using a older version pyspark, but with assembly jar of Spark Core 1.4?
On Sun, Jun 21, 2015 at 7:24 AM, Shaanan Cohney <shaan...@gmail.com> wrote: > > Hi all, > > > I'm having an issue running some code that works on a build of spark I made > (and still have) but now rebuilding it again, I get the below traceback. I > built it using the 1.4.0 release, profile hadoop-2.4 but version 2.7 and I'm > using python3. It's not vital to my work (as I can use my other build) but > I'd still like to figure out what's going on. > > Best, > shaananc > > Traceback (most recent call last): > File "factor.py", line 73, in <module> > main() > File "factor.py", line 53, in main > poly_filename = polysel.run(sc, parameters) > File "/home/ubuntu/spark_apps/polysel.py", line 90, in run > polysel1_bestpolys = run_polysel1(sc, parameters) > File "/home/ubuntu/spark_apps/polysel.py", line 72, in run_polysel1 > polysel1_bestpolys = [v for _, v in polysel1_polys.takeOrdered(nrkeep, > key=lambda s: s[0])] > File "/home/ubuntu/spark/python/pyspark/rdd.py", line 1198, in takeOrdered > return self.mapPartitions(lambda it: [heapq.nsmallest(num, it, > key)]).reduce(merge) > File "/home/ubuntu/spark/python/pyspark/rdd.py", line 762, in reduce > vals = self.mapPartitions(func).collect() > File "/home/ubuntu/spark/python/pyspark/rdd.py", line 736, in collect > port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) > File "/home/ubuntu/spark/python/pyspark/rdd.py", line 2343, in _jrdd > bvars, self.ctx._javaAccumulator) > File "/usr/local/lib/python3.4/dist-packages/py4j/java_gateway.py", line > 701, in __call__ > self._fqn) > File "/usr/local/lib/python3.4/dist-packages/py4j/protocol.py", line 304, > in get_return_value > format(target_id, '.', name, value)) > py4j.protocol.Py4JError: An error occurred while calling > None.org.apache.spark.api.python.PythonRDD. Trace: > py4j.Py4JException: Constructor org.apache.spark.api.python.PythonRDD([class > org.apache.spark.rdd.ParallelCollectionRDD, class [B, class > java.util.HashMap, class java.util.ArrayList, class java.lang.Boolean, class > java.lang.String, class java.util.ArrayList, class > org.apache.spark.Accumulator]) does not exist > at > py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:184) > at > py4j.reflection.ReflectionEngine.getConstructor(ReflectionEngine.java:202) > at py4j.Gateway.invoke(Gateway.java:213) > at > py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) > at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) > at py4j.GatewayConnection.run(GatewayConnection.java:207) > at java.lang.Thread.run(Thread.java:745) > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
