Rajesh Balamohan created TEZ-2378:
-------------------------------------
Summary: In case Fetcher (unordered) fails to do local fetch, log
in debug mode to reduce log size
Key: TEZ-2378
URL: https://issues.apache.org/jira/browse/TEZ-2378
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
Following can be logged as debug mode as opposed to WARN level. May be counters
can be added later to track the number of times it failed to do local-fetch.
{noformat}
2015-04-28 05:41:45,487 WARN [Fetcher [Map_5] #15] shuffle.Fetcher: Failed to
shuffle output of InputAttemptIdentifier [inputIdentifier=InputIdentifier
[inputIndex=81], attemptNumber=0,
pathComponent=attempt_1429683757595_0485_1_03_000081_0_10003,
fetchTypeInfo=FINAL_MERGE_ENABLED, spillEventId=-1] from
cn047-10.l42scl.hortonworks.com(local fetch)
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
output/attempt_1429683757595_0485_1_03_000081_0_10003/file.out.index in any of
the configured local directories
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:612)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:592)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:537)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher.doSharedFetch(Fetcher.java:353)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:192)
at
org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
