[
https://issues.apache.org/jira/browse/TEZ-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3104:
---------------------------------
Priority: Critical (was: Major)
> Tez fails on Bzip2 intermediate output format
> ---------------------------------------------
>
> Key: TEZ-3104
> URL: https://issues.apache.org/jira/browse/TEZ-3104
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Priority: Critical
> Attachments: TEZ-3104.1.patch
>
>
> HADOOP_CLASSPATH="$TEZ_CONF_DIR:$TEZ_HOME/*:$TEZ_HOME/lib/*" yarn jar
> /home/gs/tez/current/tez-tests-*.jar mrrsleep
> -Dmapreduce.reduce.log.level=TRACE -Dtez.task.log.level=TRACE
> -Dtez.runtime.compress=true
> -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
>
> -Dmapred.map.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
> -Dmapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
> -Dmapreduce.reduce.shuffle.parallelcopies=30 -m 100 -ir 10 -r 100
> {noformat}
> 2016-02-09 02:31:36,605 [ERROR] [ShuffleAndMergeRunner {map}]
> |orderedgrouped.Shuffle|: map: ShuffleRunner failed with error
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
> error in shuffle in fetcher {map} #16
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:360)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:337)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.hadoop.io.compress.bzip2.BZip2DummyDecompressor.decompress(BZip2DummyDecompressor.java:32)
> at
> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
> at
> org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.readToMemory(IFile.java:626)
> at
> org.apache.tez.runtime.library.common.shuffle.ShuffleUtils.shuffleToMemory(ShuffleUtils.java:113)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyMapOutput(FetcherOrderedGrouped.java:502)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:279)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.fetchNext(FetcherOrderedGrouped.java:169)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run(FetcherOrderedGrouped.java:184)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
