Hi all,
While I am testing some codes in PySpark, I met a weird issue.
This works fine at Spark 1.6.0 but it looks it does not for Spark 2.0.0.
When I simply run *logData = sc.textFile(path).coalesce(1) *with some big
files in stand-alone local mode without HDFS, this simply throws the
exception,
*_fill_function() takes exactly 4 arguments (5 given)*
I firstly wanted to open a Jira for this but feel like it is a too
primitive functionality and I felt like I might be doing this wrong.
The full error message is below:
16/03/07 11:12:44 INFO rdd.HadoopRDD: Input split:
file:/Users/hyukjinkwon/Desktop/workspace/local/spark-local-ade/spark/data/00_REF/20160119000000-20160215235900-TROI_STAT_ADE_0.DAT:2415919104+33554432
*16/03/07 11:12:44 INFO rdd.HadoopRDD: Input split:
file:/Users/hyukjinkwon/Desktop/workspace/local/spark-local-ade/spark/data/00_REF/20160119000000-20160215235900-TROI_STAT_ADE_0.DAT:805306368+33554432*
*16/03/07 11:12:44 INFO rdd.HadoopRDD: Input split:
file:/Users/hyukjinkwon/Desktop/workspace/local/spark-local-ade/spark/data/00_REF/20160119000000-20160215235900-TROI_STAT_ADE_0.DAT:0+33554432*
*16/03/07 11:12:44 INFO rdd.HadoopRDD: Input split:
file:/Users/hyukjinkwon/Desktop/workspace/local/spark-local-ade/spark/data/00_REF/20160119000000-20160215235900-TROI_STAT_ADE_0.DAT:1610612736+33554432*
*16/03/07 11:12:44 ERROR executor.Executor: Exception in task 2.0 in stage
0.0 (TID 2)*
*org.apache.spark.api.python.PythonException: Traceback (most recent call
last):*
* File "./python/pyspark/worker.py", line 98, in main*
* command = pickleSer._read_with_length(infile)*
* File "./python/pyspark/serializers.py", line 164, in _read_with_length*
* return self.loads(obj)*
* File "./python/pyspark/serializers.py", line 422, in loads*
* return pickle.loads(obj)*
*TypeError: ('_fill_function() takes exactly 4 arguments (5 given)',
<function _fill_function at 0x101e105f0>, (<function add_shuffle_key at
0x10612c488>, {'defaultdict': <type 'collections.defaultdict'>,
'get_used_memory': <function get_used_memory at 0x1027c8b18>, 'pack_long':
<function pack_long at 0x101e1ec08>}, None, {}, 'pyspark.rdd'))*
* at
org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:168)*
* at
org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:209)*
* at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:127)*
* at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:62)*
* at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)*
* at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)*
* at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:349)*
* at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)*
* at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)*
* at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:77)*
* at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:45)*
* at org.apache.spark.scheduler.Task.run(Task.scala:82)*
* at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)*
* at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)*
* at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)*
* at java.lang.Thread.run(Thread.java:745)*
*16/03/07 11:12:44 ERROR executor.Executor: Exception in task 3.0 in stage
0.0 (TID 3)*
*org.apache.spark.api.python.PythonException: Traceback (most recent call
last):*
* File "./python/pyspark/worker.py", line 98, in main*
* command = pickleSer._read_with_length(infile)*
* File "./python/pyspark/serializers.py", line 164, in _read_with_length*
* return self.loads(obj)*
* File "./python/pyspark/serializers.py", line 422, in loads*
* return pickle.loads(obj)*
*TypeError: ('_fill_function() takes exactly 4 arguments (5 given)',
<function _fill_function at 0x101e105f0>, (<function add_shuffle_key at
0x10612c488>, {'defaultdict': <type 'collections.defaultdict'>,
'get_used_memory': <function get_used_memory at 0x1027c8b18>, 'pack_long':
<function pack_long at 0x101e1ec08>}, None, {}, 'pyspark.rdd'))*
Thanks!
