subject:"Re\: Review Request 59530\: PIG\-5157 Upgrade to Spark 2.0"

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-07-18 Thread kelly zhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180895
---


Ship it!




Ship It!

- kelly zhang


On July 18, 2017, 11:38 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated July 18, 2017, 11:38 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobGraphBuilder.java 
> 5eac045dc6b7b7f3e06cf177e97d19e7d7f43010 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobStatisticCollector.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java 
> 1541264c4088fbe4770e706fd8fc53907fa83796 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/15/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-07-18 Thread Nandor Kollar



> On July 18, 2017, 7:16 a.m., kelly zhang wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java
> > Line 133 (original), 133 (patched)
> > 
> >
> > How about change 
> > org.apache.pig.backend.hadoop.executionengine.spark.JobMetricsListener to 
> > org.apache.pig.backend.hadoop.executionengine.spark.JobStatisticCollector?
> > and rename JobMetricsListener#allJobMetrics to 
> > JobStatisticCollector#allJobStatistics?
> > 
> > 
> > Just suggestion to make the code more better.

Ok, makes sense, renamed the classes and the private field according to you 
recommendation.


- Nandor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180799
---


On July 18, 2017, 11:38 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated July 18, 2017, 11:38 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobGraphBuilder.java 
> 5eac045dc6b7b7f3e06cf177e97d19e7d7f43010 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobStatisticCollector.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java 
> 1541264c4088fbe4770e706fd8fc53907fa83796 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-07-18 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated July 18, 2017, 11:38 a.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/JobGraphBuilder.java 
5eac045dc6b7b7f3e06cf177e97d19e7d7f43010 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobStatisticCollector.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java 
1541264c4088fbe4770e706fd8fc53907fa83796 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/15/

Changes: https://reviews.apache.org/r/59530/diff/14-15/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-07-18 Thread kelly zhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180799
---




src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java
Line 133 (original), 133 (patched)


How about change 
org.apache.pig.backend.hadoop.executionengine.spark.JobMetricsListener to 
org.apache.pig.backend.hadoop.executionengine.spark.JobStatisticCollector?
and rename JobMetricsListener#allJobMetrics to 
JobStatisticCollector#allJobStatistics?

Just suggestion to make the code more better.


- kelly zhang


On July 17, 2017, 3:01 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated July 17, 2017, 3:01 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/14/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-07-17 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated July 17, 2017, 3:01 p.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

rename JobMetricsListener classes


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/14/

Changes: https://reviews.apache.org/r/59530/diff/13-14/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-07-17 Thread kelly zhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180657
---



Fix it then Ship it


src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
Line 32 (original), 32 (patched)


There are two JobMetricsListener.


org.apache.pig.backend.hadoop.executionengine.spark.Spark1Shims.JobMetricsListener
org.apache.pig.backend.hadoop.executionengine.spark.JobMetricsListener

Can you rename to make it more understandable?


- kelly zhang


On June 28, 2017, 8:55 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 28, 2017, 8:55 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/13/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-28 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 28, 2017, 8:55 a.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

e2e tests were hanging on Spark2 due to accidentally deleted onJobEnd method 
from Spark2 shim


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/13/

Changes: https://reviews.apache.org/r/59530/diff/12-13/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-27 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 27, 2017, 3:33 p.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml d274af9be3805f5167f11afed9d85f19e72916d8 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/12/

Changes: https://reviews.apache.org/r/59530/diff/11-12/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-23 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 23, 2017, 12:18 p.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

Commit tests broken for Spark, function adapters have to extend Serializable 
(like FlatMapFunction and PairFlatMapFunction).


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/11/

Changes: https://reviews.apache.org/r/59530/diff/10-11/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-23 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 23, 2017, 9:49 a.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

Synchronize on the correct object in JobMetricsListener, no need to change 
compile conf


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/10/

Changes: https://reviews.apache.org/r/59530/diff/9-10/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-22 Thread Rohini Palaniswamy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178716
---


Ship it!




Ship It!

- Rohini Palaniswamy


On June 22, 2017, 2:38 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 22, 2017, 2:38 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/9/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-22 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 22, 2017, 2:38 p.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

fix serialization problem for SparkShims


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/9/

Changes: https://reviews.apache.org/r/59530/diff/8-9/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-22 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 22, 2017, 2:07 p.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

Fix for java.lang.ClassNotFoundException: 
org.apache.spark.scheduler.SparkListenerInterface


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/8/

Changes: https://reviews.apache.org/r/59530/diff/7-8/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-21 Thread Nandor Kollar



> On June 20, 2017, 8:43 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Line 795 (original), 795 (patched)
> > 
> >
> > Instead of changing all references of jar to jar-simple, you can just 
> > compile Spark 1 at the end so that dependencies for test remain same as 
> > doing jar-simple. HBase2 can be later added to the Spark2 compilation and 
> > HBase1 to Spark1. 
> > 
> > 
> >  > description="Create only pig core jar">
> >  > outputFile="${output.jarfile.core}" includedJars="core.dependencies.jar"/>
> > 
> > 
> > 
> > Compiling against Spark 2
> > 
> > 
> > 
> >  > inheritall="true"/>
> >  > tofile="${basedir}/_pig-shims.jar"/>
> > 
> > Compiling against Spark 1
> > 
> > 
> > 
> >  > inheritall="true"/>
> > 
> >  > includes="**/Spark2*.class"/>
> > 
> >  > jarfile="${output.jarfile.backcompat-core-h2}">
> >  > includes="**/Spark2*.class"/>
> > 
> > 
> >  > includes="**/Spark2*.class"/>
> >   
> > 
> > 

Thanks Rohini, make sense. Modified the targets accordingly, could you please 
have a look at the latest patch?


- Nandor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178421
---


On June 21, 2017, 9:06 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 21, 2017, 9:06 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-21 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 21, 2017, 9:06 p.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

Address Rohini's latest comment: no need to change references for jar to 
jar-simple, and no need to delete build artifact


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/7/

Changes: https://reviews.apache.org/r/59530/diff/6-7/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-20 Thread Rohini Palaniswamy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178421
---




build.xml
Line 795 (original), 795 (patched)


Instead of changing all references of jar to jar-simple, you can just 
compile Spark 1 at the end so that dependencies for test remain same as doing 
jar-simple. HBase2 can be later added to the Spark2 compilation and HBase1 to 
Spark1. 






Compiling against Spark 2






Compiling against Spark 1












  




- Rohini Palaniswamy


On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 19, 2017, 9:41 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-19 Thread Adam Szita


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178236
---


Ship it!




Ship It!

- Adam Szita


On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 19, 2017, 9:41 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-19 Thread Adam Szita



> On June 16, 2017, 9:33 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Lines 795 (patched)
> > 
> >
> > Can you rename the current "jar" target to "jar-simple" and actually 
> > make this one part of the jar target? HBase and Hadoop3 can be added to it 
> > in the future. This way anyone who builds with "ant clean jar" as today 
> > gets a pig jar usable across all HBase/Spark/Hadoop3 versions. 
> > Developers like us can use the jar-simple target to save time.
> 
> Nandor Kollar wrote:
> Renamed targets as requested, should the default remain jar, or it should 
> be jar-simple instead?

I believe the default should stay "jar" for the same reasons expressed above


- Adam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178126
---


On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 19, 2017, 9:41 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-19 Thread Nandor Kollar



> On June 16, 2017, 9:33 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Lines 795 (patched)
> > 
> >
> > Can you rename the current "jar" target to "jar-simple" and actually 
> > make this one part of the jar target? HBase and Hadoop3 can be added to it 
> > in the future. This way anyone who builds with "ant clean jar" as today 
> > gets a pig jar usable across all HBase/Spark/Hadoop3 versions. 
> > Developers like us can use the jar-simple target to save time.

Renamed targets as requested, should the default remain jar, or it should be 
jar-simple instead?


- Nandor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178126
---


On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 19, 2017, 9:41 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-19 Thread Nandor Kollar



> On June 15, 2017, 10:18 a.m., Adam Szita wrote:
> > build.xml
> > Lines 795 (patched)
> > 
> >
> > This will leave the jar inside build directory having Spark2 classes 
> > only. We should fix that, so there is no discrepancy between pig jars 
> > generated.

Deleting the jar from build directory as discussed offline.


- Nandor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177990
---


On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 19, 2017, 9:41 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-19 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 19, 2017, 9:41 a.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

Address comments from latest review


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/6/

Changes: https://reviews.apache.org/r/59530/diff/5-6/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-16 Thread Rohini Palaniswamy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178126
---



Looks good. Just minor changes for class names and ant target.


build.xml
Lines 795 (patched)


Can you rename the current "jar" target to "jar-simple" and actually make 
this one part of the jar target? HBase and Hadoop3 can be added to it in the 
future. This way anyone who builds with "ant clean jar" as today gets a pig jar 
usable across all HBase/Spark/Hadoop3 versions. 
Developers like us can use the jar-simple target to save time.



src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java
Lines 38 (patched)


SparkShims



src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java
Lines 68 (patched)


getInstance(). get() makes you think it is a ThreadLocal variable.



src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java
Lines 59 (patched)


Spark1Shims



src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java
Lines 46 (patched)


Spark2Shims



src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java
Lines 32 (patched)


Spark1JobStats



src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java
Lines 31 (patched)


Spark2JobStats


- Rohini Palaniswamy


On June 14, 2017, 9:13 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 14, 2017, 9:13 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-15 Thread Adam Szita


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177990
---




build.xml
Lines 245 (patched)


Should we call this src.exclude.pattern instead of dir?



build.xml
Lines 795 (patched)


This will leave the jar inside build directory having Spark2 classes only. 
We should fix that, so there is no discrepancy between pig jars generated.


- Adam Szita


On June 14, 2017, 9:13 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 14, 2017, 9:13 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
>   ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
>   ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> 237fd9431a16226234d91059088f91aab346b83c 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  130c8b9a747b176ce2b649ca6d5260527595fb76 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> ec08417f2f71deec514cab5cfb9d2f99520ad641 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-14 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 14, 2017, 9:13 a.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

updated JobMetricsListener for Spark 1.x and 2.x, patch created on top of the 
latest trunk codebase


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml bba2b52d9354ab909ad26f969480806f6d91911c 
  ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588 
  ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
237fd9431a16226234d91059088f91aab346b83c 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 130c8b9a747b176ce2b649ca6d5260527595fb76 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
ec08417f2f71deec514cab5cfb9d2f99520ad641 


Diff: https://reviews.apache.org/r/59530/diff/5/

Changes: https://reviews.apache.org/r/59530/diff/4-5/


Testing
---


Thanks,

Nandor Kollar

RE: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-13 Thread Zhang, Liyun

No need to add null checks. Thanks for explanation. Please update the patch 
with latest trunk code and this makes me easy to test the code.

Best Regards
Kelly Zhang/Zhang,Liyun



-Original Message-
From: Nandor Kollar [mailto:nore...@reviews.apache.org] On Behalf Of Nandor 
Kollar
Sent: Tuesday, June 13, 2017 3:52 PM
To: Rohini Palaniswamy <rohini.adi...@gmail.com>; Zhang, Liyun 
<liyun.zh...@intel.com>; Adam Szita <sz...@cloudera.com>
Cc: Zhang, Liyun <liyun.zh...@intel.com>; pig <dev@pig.apache.org>; Nandor 
Kollar <nkol...@cloudera.com>
Subject: Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0



> On June 13, 2017, 7:33 a.m., kelly zhang wrote:
> > src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java
> > Lines 60-63 (patched)
> > <https://reviews.apache.org/r/59530/diff/4/?file=1749088#file1749088line60>
> >
> > why 
> > inputMetricExists,outputMetricExist,shuffleReadMetricExist,shuffleWriteMetricExist
> >  are deleted in SparkJobStats2.java?

These metrics are no longer Optional, but have an initialized value. For 
example this is the shuffleWriteMetrics: val shuffleWriteMetrics: 
ShuffleWriteMetrics = new ShuffleWriteMetrics(); I don't think we need to check 
for the existence of these metrics, they are initialized all the time, but if 
you think it is better to check for null before using them, I can add null 
checks. for it.


- Nandor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177699
---


On June 12, 2017, 9:20 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 12, 2017, 9:20 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
>   ivy.xml 971724380f086d214ce62c7ab7879b08b6926802 
>   ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> c6351e01a48f297ea2e432401ffd65c4f27f8078 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-13 Thread Nandor Kollar



> On June 13, 2017, 7:33 a.m., kelly zhang wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
> > Line 82 (original), 83 (patched)
> > 
> >
> > I guess in spark2, the comment in 
> > onStageCompleted(SparkListenerStageCompleted stageCompleted) method should 
> > be uncommented, the code can be used in spark2.
> > but these code can not be used in spark1.Maybe you need create 
> > JobMetricsListener1, JobMetricsListener2

Indeed, there should be two separate JobMetricsListener: one for Spark 1.x and 
one for Spark 2.x. Thanks for catching this, will do this and update the review.


- Nandor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177699
---


On June 12, 2017, 9:20 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 12, 2017, 9:20 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
>   ivy.xml 971724380f086d214ce62c7ab7879b08b6926802 
>   ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> c6351e01a48f297ea2e432401ffd65c4f27f8078 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-13 Thread Nandor Kollar



> On June 13, 2017, 7:33 a.m., kelly zhang wrote:
> > src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java
> > Lines 60-63 (patched)
> > 
> >
> > why 
> > inputMetricExists,outputMetricExist,shuffleReadMetricExist,shuffleWriteMetricExist
> >  are deleted in SparkJobStats2.java?

These metrics are no longer Optional, but have an initialized value. For 
example this is the shuffleWriteMetrics: val shuffleWriteMetrics: 
ShuffleWriteMetrics = new ShuffleWriteMetrics(); I don't think we need to check 
for the existence of these metrics, they are initialized all the time, but if 
you think it is better to check for null before using them, I can add null 
checks. for it.


- Nandor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177699
---


On June 12, 2017, 9:20 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 12, 2017, 9:20 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
>   ivy.xml 971724380f086d214ce62c7ab7879b08b6926802 
>   ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> c6351e01a48f297ea2e432401ffd65c4f27f8078 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-13 Thread kelly zhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177699
---




src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
Line 82 (original), 83 (patched)


I guess in spark2, the comment in 
onStageCompleted(SparkListenerStageCompleted stageCompleted) method should be 
uncommented, the code can be used in spark2.
but these code can not be used in spark1.Maybe you need create 
JobMetricsListener1, JobMetricsListener2



src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java
Lines 60-63 (patched)


why 
inputMetricExists,outputMetricExist,shuffleReadMetricExist,shuffleWriteMetricExist
 are deleted in SparkJobStats2.java?


- kelly zhang


On June 12, 2017, 9:20 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated June 12, 2017, 9:20 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
>   ivy.xml 971724380f086d214ce62c7ab7879b08b6926802 
>   ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
>  PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
>  PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> c6351e01a48f297ea2e432401ffd65c4f27f8078 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java 
> PRE-CREATION 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> c8cc03194b223d2ee181d73c6b651a6872cac6b6 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION 
>   src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
> 61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
>   test/org/apache/pig/test/TestPigRunner.java 
> 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-12 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 12, 2017, 9:20 p.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

Assemble jar for both Spark 1.x and 2.x


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
  ivy.xml 971724380f086d214ce62c7ab7879b08b6926802 
  ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java
 PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java
 PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
c6351e01a48f297ea2e432401ffd65c4f27f8078 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java 
PRE-CREATION 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION 
  src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 
61ccbcc9fd723f6e2e578a8476230c42d5587dfe 
  test/org/apache/pig/test/TestPigRunner.java 
1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c 


Diff: https://reviews.apache.org/r/59530/diff/4/

Changes: https://reviews.apache.org/r/59530/diff/3-4/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-06-02 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated June 2, 2017, 3:20 p.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Changes
---

fix for ClassCastException in skew join revealed while executing e2e tests


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
  ivy/spark2.properties PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
 PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
 PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
 PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
 PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
 PRE-CREATION 
  shims/src/spark1/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
 PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
 PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
 PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
 PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
 PRE-CREATION 
  shims/src/spark2/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f81341233447203abc4800cc7b22a4f419e10262 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
c6351e01a48f297ea2e432401ffd65c4f27f8078 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311dfa5bb25209a5366c2db7e8d483c31d94cd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e7ff9105aa397c5a2888df0c11e9562ec9 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e7e18ca4cf1331beef06e9214600a51424 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c8f128661d7d19c276d3bb2de7874d3086 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1e06adc4c9fe45a83b8110402e3756 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd95763b2d189ff9ec540c89abe52592420 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b44848546ed16dde2baa8c61b36939971b2 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa090323e3bef087e259ce19df2e4c34dd63 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc31745c013380492e089c83f3e853a3e6e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
c8cc03194b223d2ee181d73c6b651a6872cac6b6 


Diff: https://reviews.apache.org/r/59530/diff/3/

Changes: https://reviews.apache.org/r/59530/diff/2-3/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-05-26 Thread Rohini Palaniswamy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review176232
---


Ship it!




Ship It!

- Rohini Palaniswamy


On May 25, 2017, 7:13 a.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated May 25, 2017, 7:13 a.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml 4040fce 
>   ivy/spark2.properties PRE-CREATION 
>   
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  PRE-CREATION 
>   
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
>  PRE-CREATION 
>   
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
>  PRE-CREATION 
>   shims/src/spark1/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> PRE-CREATION 
>   
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  PRE-CREATION 
>   
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
>  PRE-CREATION 
>   
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
>  PRE-CREATION 
>   shims/src/spark2/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f813412 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> c6351e0 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311df 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  f571b80 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ec 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b4 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba31 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa0 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d48 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc031 
> 
> 
> Diff: https://reviews.apache.org/r/59530/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nandor Kollar
> 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-05-25 Thread Nandor Kollar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
---

(Updated May 25, 2017, 7:13 a.m.)


Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.


Repository: pig-git


Description
---

Upgrade to Spark 2.1 API using shims.


Diffs (updated)
-

  build.xml 4040fce 
  ivy/spark2.properties PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
 PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
 PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
 PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
 PRE-CREATION 
  
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
 PRE-CREATION 
  shims/src/spark1/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
 PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
 PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
 PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
 PRE-CREATION 
  
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
 PRE-CREATION 
  shims/src/spark2/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
PRE-CREATION 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java 
f813412 
  src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
c6351e0 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
 83311df 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
 382258e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
 b58415e 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
 f571b80 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
 fe1b54c 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
 adf78ec 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
 d1c43b1 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
 e003bbd 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
 00d29b4 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
 c55ba31 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
 baabfa0 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
 3166fdc 
  
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
 3a50d48 
  src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc031 


Diff: https://reviews.apache.org/r/59530/diff/2/

Changes: https://reviews.apache.org/r/59530/diff/1-2/


Testing
---


Thanks,

Nandor Kollar

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-05-24 Thread Adam Szita



> On May 24, 2017, 9:21 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Lines 251 (patched)
> > 
> >
> > Can just load one property file for spark2, leaving the spark 1.6 
> > version in libraries.properties

AFAIK property values in ant are immutable, so once we set e.g. 
spark.version=1.6.1 by loading the default property file and then we load the 
other propery file for spark2 I think it will stay unchanged and we cannot 
override it for anything else e.g. spark.version=2.1.1
Also this way it looks clearer I think.


- Adam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review175998
---


On May 24, 2017, 4:21 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated May 24, 2017, 4:21 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
>   ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 
>   ivy/spark1.properties PRE-CREATION 
>   ivy/spark2.properties PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
>  PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
>  PRE-CREATION 
>   shims/src/spark16/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
>  PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
>  PRE-CREATION 
>   shims/src/spark21/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> c6351e01a48f297ea2e432401ffd65c4f27f8078 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

2017-05-24 Thread Rohini Palaniswamy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review175998
---




build.xml
Lines 243 (patched)


Please have sparkversion as 1 and 2 and the shims directory names as well.



build.xml
Lines 251 (patched)


Can just load one property file for spark2, leaving the spark 1.6 version 
in libraries.properties


- Rohini Palaniswamy


On May 24, 2017, 4:21 p.m., Nandor Kollar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> ---
> 
> (Updated May 24, 2017, 4:21 p.m.)
> 
> 
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
> 
> 
> Repository: pig-git
> 
> 
> Description
> ---
> 
> Upgrade to Spark 2.1 API using shims.
> 
> 
> Diffs
> -
> 
>   build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e 
>   ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3 
>   ivy/spark1.properties PRE-CREATION 
>   ivy/spark2.properties PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
>  PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
>  PRE-CREATION 
>   shims/src/spark16/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java
>  PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java
>  PRE-CREATION 
>   
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java
>  PRE-CREATION 
>   shims/src/spark21/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
> PRE-CREATION 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
>  f81341233447203abc4800cc7b22a4f419e10262 
>   src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 
> c6351e01a48f297ea2e432401ffd65c4f27f8078 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java
>  83311dfa5bb25209a5366c2db7e8d483c31d94cd 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java
>  382258e7ff9105aa397c5a2888df0c11e9562ec9 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java
>  b58415e7e18ca4cf1331beef06e9214600a51424 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java
>  f571b808839c2de9415a3e8e4b229a7f4b2eebd7 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java
>  fe1b54c8f128661d7d19c276d3bb2de7874d3086 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java
>  adf78ecab0da10d3b1a7fdde8af2b42dd899810f 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java
>  d1c43b1e06adc4c9fe45a83b8110402e3756 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java
>  e003bbd95763b2d189ff9ec540c89abe52592420 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java
>  00d29b44848546ed16dde2baa8c61b36939971b2 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java
>  c55ba3145495a53d69db2dd56434dcc9b3bf8ed5 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java
>  baabfa090323e3bef087e259ce19df2e4c34dd63 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java
>  3166fdc31745c013380492e089c83f3e853a3e6e 
>   
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java
>  3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc 
>   src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java 
>

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

RE: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0

34 matches

Site Navigation

Mail list logo

Footer information