----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59530/#review177699 -----------------------------------------------------------
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java Line 82 (original), 83 (patched) <https://reviews.apache.org/r/59530/#comment251429> I guess in spark2, the comment in onStageCompleted(SparkListenerStageCompleted stageCompleted) method should be uncommented, the code can be used in spark2. but these code can not be used in spark1.Maybe you need create JobMetricsListener1, JobMetricsListener2 src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java Lines 60-63 (patched) <https://reviews.apache.org/r/59530/#comment251427> why inputMetricExists,outputMetricExist,shuffleReadMetricExist,shuffleWriteMetricExist are deleted in SparkJobStats2.java? - kelly zhang On June 12, 2017, 9:20 p.m., Nandor Kollar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59530/ > ----------------------------------------------------------- > > (Updated June 12, 2017, 9:20 p.m.) > > > Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita. > > > Repository: pig-git > > > Description > ------- > > Upgrade to Spark 2.1 API using shims. > > > Diffs > ----- > > build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e > ivy.xml 971724380f086d214ce62c7ab7879b08b6926802 > ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 > > src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java > f81341233447203abc4800cc7b22a4f419e10262 > > src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java > c6351e01a48f297ea2e432401ffd65c4f27f8078 > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java > 83311dfa5bb25209a5366c2db7e8d483c31d94cd > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java > 382258e7ff9105aa397c5a2888df0c11e9562ec9 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java > b58415e7e18ca4cf1331beef06e9214600a51424 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java > f571b808839c2de9415a3e8e4b229a7f4b2eebd7 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java > fe1b54c8f128661d7d19c276d3bb2de7874d3086 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java > adf78ecab0da10d3b1a7fdde8af2b42dd899810f > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java > d1c43b1e06adc4c9fe45a83b81103333402e3756 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java > e003bbd95763b2d189ff9ec540c89abe52592420 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java > 00d29b44848546ed16dde2baa8c61b36939971b2 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java > c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java > baabfa090323e3bef087e259ce19df2e4c34dd63 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java > 3166fdc31745c013380492e089c83f3e853a3e6e > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java > 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc > src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java > c8cc03194b223d2ee181d73c6b651a6872cac6b6 > src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION > src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION > src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java > 61ccbcc9fd723f6e2e578a8476230c42d5587dfe > test/org/apache/pig/test/TestPigRunner.java > 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c > > > Diff: https://reviews.apache.org/r/59530/diff/4/ > > > Testing > ------- > > > Thanks, > > Nandor Kollar > >
