----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59530/#review178421 -----------------------------------------------------------
build.xml Line 795 (original), 795 (patched) <https://reviews.apache.org/r/59530/#comment252369> Instead of changing all references of jar to jar-simple, you can just compile Spark 1 at the end so that dependencies for test remain same as doing jar-simple. HBase2 can be later added to the Spark2 compilation and HBase1 to Spark1. <target name="jar-core" depends="compile,ivy-buildJar" description="Create only pig core jar"> <buildJar svnString="${svn.revision}" outputFile="${output.jarfile.core}" includedJars="core.dependencies.jar"/> </target> <target name="jar" description="Create pig jar with Spark 1 and 2"> <echo>Compiling against Spark 2</echo> <antcall target="clean" inheritRefs="true" inheritall="true"/> <propertyreset name="sparkversion" value="2"/> <propertyreset name="src.exclude.dir" value="**/Spark1*.java" /> <antcall target="jar-core" inheritRefs="true" inheritall="true"/> <move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/> <echo>Compiling against Spark 1</echo> <antcall target="clean" inheritRefs="true" inheritall="true"/> <propertyreset name="sparkversion" value="1"/> <propertyreset name="src.exclude.dir" value="**/Spark2*.java" /> <antcall target="jar-simple" inheritRefs="true" inheritall="true"/> <jar update="yes" jarfile="${output.jarfile.core}"> <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/> </jar> <jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}"> <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/> </jar> <jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}"> <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/> </jar> <delete file="${basedir}/_pig-shims.jar"/> </target> - Rohini Palaniswamy On June 19, 2017, 9:41 a.m., Nandor Kollar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59530/ > ----------------------------------------------------------- > > (Updated June 19, 2017, 9:41 a.m.) > > > Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita. > > > Repository: pig-git > > > Description > ------- > > Upgrade to Spark 2.1 API using shims. > > > Diffs > ----- > > build.xml bba2b52d9354ab909ad26f969480806f6d91911c > ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 > ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 > > src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java > f81341233447203abc4800cc7b22a4f419e10262 > > src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java > PRE-CREATION > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java > 237fd9431a16226234d91059088f91aab346b83c > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java > 83311dfa5bb25209a5366c2db7e8d483c31d94cd > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java > 382258e7ff9105aa397c5a2888df0c11e9562ec9 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java > b58415e7e18ca4cf1331beef06e9214600a51424 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java > 130c8b9a747b176ce2b649ca6d5260527595fb76 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java > fe1b54c8f128661d7d19c276d3bb2de7874d3086 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java > adf78ecab0da10d3b1a7fdde8af2b42dd899810f > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java > d1c43b1e06adc4c9fe45a83b81103333402e3756 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java > e003bbd95763b2d189ff9ec540c89abe52592420 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java > 00d29b44848546ed16dde2baa8c61b36939971b2 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java > c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java > baabfa090323e3bef087e259ce19df2e4c34dd63 > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java > 3166fdc31745c013380492e089c83f3e853a3e6e > > src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java > 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc > src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION > src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION > src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java > c8cc03194b223d2ee181d73c6b651a6872cac6b6 > src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java > 61ccbcc9fd723f6e2e578a8476230c42d5587dfe > test/org/apache/pig/test/TestPigRunner.java > ec08417f2f71deec514cab5cfb9d2f99520ad641 > > > Diff: https://reviews.apache.org/r/59530/diff/6/ > > > Testing > ------- > > > Thanks, > > Nandor Kollar > >
