----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59530/#review178126 -----------------------------------------------------------
Looks good. Just minor changes for class names and ant target. build.xml Lines 795 (patched) <https://reviews.apache.org/r/59530/#comment251955> Can you rename the current "jar" target to "jar-simple" and actually make this one part of the jar target? HBase and Hadoop3 can be added to it in the future. This way anyone who builds with "ant clean jar" as today gets a pig jar usable across all HBase/Spark/Hadoop3 versions. Developers like us can use the jar-simple target to save time. src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java Lines 38 (patched) <https://reviews.apache.org/r/59530/#comment251956> SparkShims src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java Lines 68 (patched) <https://reviews.apache.org/r/59530/#comment251959> getInstance(). get() makes you think it is a ThreadLocal variable. src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java Lines 59 (patched) <https://reviews.apache.org/r/59530/#comment251957> Spark1Shims src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java Lines 46 (patched) <https://reviews.apache.org/r/59530/#comment251958> Spark2Shims src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java Lines 32 (patched) <https://reviews.apache.org/r/59530/#comment251960> Spark1JobStats src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java Lines 31 (patched) <https://reviews.apache.org/r/59530/#comment251961> Spark2JobStats - Rohini Palaniswamy On June 14, 2017, 9:13 a.m., Nandor Kollar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59530/ > ----------------------------------------------------------- > > (Updated June 14, 2017, 9:13 a.m.) > > > Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita. > > > Repository: pig-git > > > Description > ------- > > Upgrade to Spark 2.1 API using shims. > > > Diffs > ----- > > build.xml bba2b52d9354ab909ad26f969480806f6d91911c > ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 > ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 > > src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java > f81341233447203abc4800cc7b22a4f419e10262 > > src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java > 237fd9431a16226234d91059088f91aab346b83c > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java > 83311dfa5bb25209a5366c2db7e8d483c31d94cd > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java > 382258e7ff9105aa397c5a2888df0c11e9562ec9 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java > b58415e7e18ca4cf1331beef06e9214600a51424 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java > 130c8b9a747b176ce2b649ca6d5260527595fb76 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java > fe1b54c8f128661d7d19c276d3bb2de7874d3086 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java > adf78ecab0da10d3b1a7fdde8af2b42dd899810f > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java > d1c43b1e06adc4c9fe45a83b81103333402e3756 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java > e003bbd95763b2d189ff9ec540c89abe52592420 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java > 00d29b44848546ed16dde2baa8c61b36939971b2 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java > c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java > baabfa090323e3bef087e259ce19df2e4c34dd63 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java > 3166fdc31745c013380492e089c83f3e853a3e6e > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java > 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc > src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java > c8cc03194b223d2ee181d73c6b651a6872cac6b6 > src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION > src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION > src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java > 61ccbcc9fd723f6e2e578a8476230c42d5587dfe > test/org/apache/pig/test/TestPigRunner.java > ec08417f2f71deec514cab5cfb9d2f99520ad641 > > > Diff: https://reviews.apache.org/r/59530/diff/5/ > > > Testing > ------- > > > Thanks, > > Nandor Kollar > >
