Rajesh Balamohan created TEZ-2180:
-------------------------------------
Summary: Unordered ShuffleManager's logProgress() reports pending
inputs instead of completed inputs
Key: TEZ-2180
URL: https://issues.apache.org/jira/browse/TEZ-2180
Project: Apache Tez
Issue Type: Bug
Reporter: Rajesh Balamohan
{code}
int inputsDone = numInputs - numCompletedInputs.get();
..
LOG.info("copy(" + inputsDone + " of " + numInputs +
". Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) "
+ mbpsFormat.format(transferRate) + " MB/s)");
{code}
This ends up printing the pending inputs instead of printing completed inputs.
e.g
{noformat}
2015-03-05 20:05:58,895 INFO [Fetcher [Map_1] #2] impl.ShuffleManager: copy(11
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.34 MB/s)
2015-03-05 20:05:59,563 INFO [Fetcher [Map_1] #2] impl.ShuffleManager: copy(10
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.31 MB/s)
2015-03-05 20:06:00,344 INFO [Fetcher [Map_1] #1] impl.ShuffleManager: copy(10
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.28 MB/s)
2015-03-05 20:06:01,139 INFO [Fetcher [Map_1] #0] impl.ShuffleManager: copy(9
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.25 MB/s)
2015-03-05 20:06:02,481 INFO [Fetcher [Map_1] #2] impl.ShuffleManager: copy(8
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.22 MB/s)
2015-03-05 20:06:02,682 INFO [Fetcher [Map_1] #1] impl.ShuffleManager: copy(7
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.23 MB/s)
2015-03-05 20:06:03,990 INFO [Fetcher [Map_1] #0] impl.ShuffleManager: copy(6
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.20 MB/s)
2015-03-05 20:06:05,099 INFO [Fetcher [Map_1] #2] impl.ShuffleManager: copy(5
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.14 MB/s)
2015-03-05 20:06:05,750 INFO [Fetcher [Map_1] #1] impl.ShuffleManager: copy(4
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.14 MB/s)
2015-03-05 20:06:05,780 INFO [Fetcher [Map_1] #2] impl.ShuffleManager: copy(3
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.14 MB/s)
2015-03-05 20:06:07,111 INFO [Fetcher [Map_1] #0] impl.ShuffleManager: copy(2
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.08 MB/s)
2015-03-05 20:06:07,194 INFO [Fetcher [Map_1] #1] impl.ShuffleManager: copy(1
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.09 MB/s)
2015-03-05 20:06:10,330 INFO [Fetcher [Map_1] #2] impl.ShuffleManager: copy(0
of 250. Transfer rate (CumulativeDataFetched/TimeSinceInputStarted)) 4.00 MB/s)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
