[
https://issues.apache.org/jira/browse/TEZ-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated TEZ-2378:
-----------------------------
Target Version/s: 0.8.0
> In case Fetcher (unordered) fails to do local fetch, log in debug mode to
> reduce log size
> -----------------------------------------------------------------------------------------
>
> Key: TEZ-2378
> URL: https://issues.apache.org/jira/browse/TEZ-2378
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
>
> Following can be logged as debug mode as opposed to WARN level. May be
> counters can be added later to track the number of times it failed to do
> local-fetch.
> {noformat}
> 2015-04-28 05:41:45,487 WARN [Fetcher [Map_5] #15] shuffle.Fetcher: Failed to
> shuffle output of InputAttemptIdentifier [inputIdentifier=InputIdentifier
> [inputIndex=81], attemptNumber=0,
> pathComponent=attempt_1429683757595_0485_1_03_000081_0_10003,
> fetchTypeInfo=FINAL_MERGE_ENABLED, spillEventId=-1] from
> cn047-10.l42scl.hortonworks.com(local fetch)
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> output/attempt_1429683757595_0485_1_03_000081_0_10003/file.out.index in any
> of the configured local directories
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449)
> at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:612)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:592)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:537)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.doSharedFetch(Fetcher.java:353)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:192)
> at
> org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
