This is an automatically generated e-mail. To reply, visit:

(Updated Feb. 19, 2018, 1:54 p.m.)

Review request for pig, Daniel Dai, Koji Noguchi, Rohini Palaniswamy, and Adam 

Repository: pig-git


This is an initial patch that adds Hadoop 3 support to Pig in addition to 
Hadoop 2.

Major modifications:
 * No breaking API change was introduced in Hadoop 3, the current code compiles 
with Hadoop 3
 * hadoopversion property tells which mode the tests should run, the default is 
hadoop 2
 * Hadoop 3 introduced a security fix, only whitelisted environment variables 
are passed to MiniCluster
 * In Hadoop 3 hadoop-site.xml is deprecated, and is replaced by core-site.xml, 
hdfs-site.xml and mapred-site.xml. I decided to write the config into all of 
these files in MiniCluster.java (into hadoop-site.xml too to stay compatible 
with Hadoop 2) for the shake of simplicity, we might want to have different 
files for Hadoop 2 and separate the properties for Hadoop 3.
 * TestErrorHandling.java: small format change in error message, modified the 
assert so it works on both on Hadoop 2 and Hadoop 3
 * HadoopShims: code is identical with Hadoop 2, not sure if we need shims any 
more. I think we should move it to the src instead.
 * Split properties into 3 files: common properties, Hadoop 2 and Hadoop 3 
specific properties
 * ivy.xml: new config for Hadoop 3
 * build.xml: new target to package both hadoop2 and hadoop3 - not sure that 
this is needed, if we move shims, the I think we don't need this target
 * HBase unit test fails on Hadoop 3 (as per https://hbase.apache.org/book.html 
HBase 1.x is not tested against Hadoop 2)

Diffs (updated)

  bin/pig 3fcf165106cccbe75fc1c61ea74732456ae50fc7 
  bin/pig.py b6c396579c54359f430c6e74d055ec7f27ae2197 
  build.xml 8bcbe5e4d60b793412dd5490518928b17308da3b 
  ivy.xml 3ac675190f15528674361eda924af61fc1d07613 
  ivy/libraries-h2.properties PRE-CREATION 
  ivy/libraries-h3.properties PRE-CREATION 
  ivy/libraries.properties 800b75edea300d6ff4d0a55481a1b3ed5e3be6ea 
  test/org/apache/pig/test/MapReduceMiniCluster.java PRE-CREATION 
  test/org/apache/pig/test/Util.java 788a72fe3ceca08ec61ae425a393b5b0936454f4 

Diff: https://reviews.apache.org/r/65239/diff/2/

Changes: https://reviews.apache.org/r/65239/diff/1-2/



Nandor Kollar

Reply via email to