-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18832/
-----------------------------------------------------------
Review request for pig, Mark Wagner and Rohini Palaniswamy.
Bugs: PIG-3603
https://issues.apache.org/jira/browse/PIG-3603
Repository: pig-git
Description
-------
This patch adds the following counters to TezStats:
1) # of input/output records
2) hdfs bytes read/written
3) file bytes read/written
The job stats looks like this:
JobId: job_pigexec_0
TotalLaunchedTasks: 3
FileBytesRead: 2434726
FileBytesWritten: 4869516
HdfsBytesRead: 2219954
HdfsBytesWritten: 2433980
Input(s): Successfully read 10000 records (1109977 bytes) from:
"/user/pig/tests/data/singlefile/studentcomplextab10k"
: Successfully read 10000 records (1109977 bytes) from:
"/user/pig/tests/data/singlefile/studentcomplextab10k"
Output(s): Successfully stored 10393 records (2433980 bytes) in:
"hdfs://localhost:57063/tmp/temp90703803/tmp-1606775243"
This patch also includes the following changes in PigStats/JobStats classes:
1) Move getHdfsBytesRead() and getHdfsBytesWritten() from MRPigStatsUtil to
PigStatsUtil since these are not MR specific.
2) Move [MAP|REDUCE]_[OUT|IN]PUT_RECORDS from MRPigStatsUtil to PigStatsUtil
since Tez MRInput and MROutput also use them.
3) Fix a typo in JobStats#getAvgREduceTime(): REduce -> Reduce.
4) Fix white spaces.
Note that none of these changes breaks backward compatibility.
Diffs
-----
src/org/apache/pig/PigServer.java 2004edb
src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java 5d12091
src/org/apache/pig/tools/pigstats/InputStats.java 38c8372
src/org/apache/pig/tools/pigstats/JobStats.java 4484348
src/org/apache/pig/tools/pigstats/OutputStats.java 6a3e3eb
src/org/apache/pig/tools/pigstats/PigStats.java 3032728
src/org/apache/pig/tools/pigstats/PigStatsUtil.java e690b8d
src/org/apache/pig/tools/pigstats/ScriptState.java d58310d
src/org/apache/pig/tools/pigstats/mapreduce/MRJobStats.java 115ae1d
src/org/apache/pig/tools/pigstats/mapreduce/MRPigStatsUtil.java ed791fd
src/org/apache/pig/tools/pigstats/tez/TezStats.java 64d70e7
src/org/apache/pig/tools/pigstats/tez/TezTaskStats.java c3f1c3e
test/org/apache/pig/test/TestCombiner.java ae2135e
test/org/apache/pig/test/TestPigServer.java 8613c3b
Diff: https://reviews.apache.org/r/18832/diff/
Testing
-------
ant test-tez passes except TestTezCompiler (known).
tez e2e tests pass.
Thanks,
Cheolsoo Park
