Rajesh Balamohan created TEZ-2378: ------------------------------------- Summary: In case Fetcher (unordered) fails to do local fetch, log in debug mode to reduce log size Key: TEZ-2378 URL: https://issues.apache.org/jira/browse/TEZ-2378 Project: Apache Tez Issue Type: Bug Reporter: Rajesh Balamohan
Following can be logged as debug mode as opposed to WARN level. May be counters can be added later to track the number of times it failed to do local-fetch. {noformat} 2015-04-28 05:41:45,487 WARN [Fetcher [Map_5] #15] shuffle.Fetcher: Failed to shuffle output of InputAttemptIdentifier [inputIdentifier=InputIdentifier [inputIndex=81], attemptNumber=0, pathComponent=attempt_1429683757595_0485_1_03_000081_0_10003, fetchTypeInfo=FINAL_MERGE_ENABLED, spillEventId=-1] from cn047-10.l42scl.hortonworks.com(local fetch) org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/attempt_1429683757595_0485_1_03_000081_0_10003/file.out.index in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:449) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164) at org.apache.tez.runtime.library.common.shuffle.Fetcher.getShuffleInputFileName(Fetcher.java:612) at org.apache.tez.runtime.library.common.shuffle.Fetcher.getTezIndexRecord(Fetcher.java:592) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doLocalDiskFetch(Fetcher.java:537) at org.apache.tez.runtime.library.common.shuffle.Fetcher.doSharedFetch(Fetcher.java:353) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:192) at org.apache.tez.runtime.library.common.shuffle.Fetcher.callInternal(Fetcher.java:72) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)