[
https://issues.apache.org/jira/browse/HADOOP-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003286#comment-14003286
]
Hudson commented on HADOOP-10614:
---------------------------------
FAILURE: Integrated in Hadoop-Hdfs-trunk #1754 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1754/])
HADOOP-10614. CBZip2InputStream is not threadsafe (Xiangrui Meng via Sandy
Ryza) (sandy:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1595521)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/bzip2/CBZip2InputStream.java
> CBZip2InputStream is not threadsafe
> -----------------------------------
>
> Key: HADOOP-10614
> URL: https://issues.apache.org/jira/browse/HADOOP-10614
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.2.1, 2.2.0
> Reporter: Xiangrui Meng
> Assignee: Xiangrui Meng
> Fix For: 1.3.0, 2.5.0
>
> Attachments: bzip2-2.diff, bzip2.diff
>
>
> Hadoop uses CBZip2InputStream to decode bzip2 files. However, the
> implementation is not threadsafe. This is not a really problem for Hadoop
> MapReduce because Hadoop runs each task in a separate JVM. But for other
> libraries that utilize multithreading and use Hadoop's InputFormat, e.g.,
> Spark, it will cause exceptions like the following:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 6
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.recvDecodingTables(CBZip2InputStream.java:729)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.getAndMoveToFrontDecode(CBZip2InputStream.java:795)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(CBZip2InputStream.java:499)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.changeStateToProcessABlock(CBZip2InputStream.java:330)
>
> org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.read(CBZip2InputStream.java:394)
>
> org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionInputStream.read(BZip2Codec.java:428)
> java.io.InputStream.read(InputStream.java:101)
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:205)
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:169)
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:176)
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:43)
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:198)
> org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:181)
> org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:35)
> scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1000)
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
> org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
>
> org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1077)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
> org.apache.spark.scheduler.Task.run(Task.scala:51)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:724)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
