Hi,
I have the following issue. I am trying to load a 2.5G JSON file from a
10-node Hadoop Cluster. Actually, I am trying to create a DataFrame, using
sqlContext.read.json("hdfs://master:9000/path/file.json").
The JSON file contains a parsed table(relation) from the TPCH benchmark.
After finishing some tasks, the job fails by throwing several
java.io.IOExceptions. For smaller files (eg 700M it works fine). I am
posting a part of the log and the whole stack trace below:
15/08/26 16:31:44 INFO TaskSetManager: Starting task 10.1 in stage 1.0 (TID
47, 192.168.5.146, ANY, 1416 bytes)
15/08/26 16:31:44 INFO TaskSetManager: Starting task 11.1 in stage 1.0 (TID
48, 192.168.5.150, ANY, 1416 bytes)
15/08/26 16:31:44 INFO TaskSetManager: Starting task 4.1 in stage 1.0 (TID
49, 192.168.5.149, ANY, 1416 bytes)
15/08/26 16:31:44 INFO TaskSetManager: Starting task 8.1 in stage 1.0 (TID
50, 192.168.5.246, ANY, 1416 bytes)
15/08/26 16:31:53 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID
17) in 104681 ms on 192.168.5.243 (27/35)
15/08/26 16:31:53 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID
15) in 105541 ms on 192.168.5.193 (28/35)
15/08/26 16:31:55 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID
18) in 107122 ms on 192.168.5.167 (29/35)
15/08/26 16:31:57 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID
12) in 109583 ms on 192.168.5.245 (30/35)
15/08/26 16:32:08 INFO TaskSetManager: Finished task 4.1 in stage 1.0 (TID
49) in 24135 ms on 192.168.5.149 (31/35)
15/08/26 16:32:13 WARN TaskSetManager: Lost task 2.0 in stage 1.0 (TID 9,
192.168.5.246): java.io.IOException: Too many bytes before newline:
2147483648
at
org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:249)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at
org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:134)
at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:239)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:216)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/SQLContext-read-json-path-throws-java-io-IOException-tp13841.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
