Re: Review Request 23960: PIG-4074 mapreduce.client.submit.file.replication is not honored in cached files

Fri, 25 Jul 2014 18:24:07 -0700 

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23960/
-----------------------------------------------------------

(Updated July 26, 2014, 1:23 a.m.)


Review request for pig, Daniel Dai and Rohini Palaniswamy.


Changes
-------

Fix one typo. I've reviewed my patch 3 times, so it should be good now.


Bugs: PIG-4074
    https://issues.apache.org/jira/browse/PIG-4074


Repository: pig-git


Description
-------

This patch includes two changes-
1) Adds dfs.setReplication() after dfs.copyFromLocal() to honor 
mapreduce.client.submit.file.replication (or mapred.submit.replication).
2) Creates MRConfiguration and refactor all the hard-coded MR properties 
(mapred.xxx and mapreduce.xxx) into static variables.


Diffs (updated)
-----

  
contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/HadoopJobHistoryLoader.java
 19045fe 
  
contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/IndexedStorage.java
 b2096fb 
  
contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/MultiStorage.java
 e65cc37 
  
contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/hiverc/HiveRCOutputFormat.java
 cd378d4 
  
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/TestCSVExcelStorage.java
 f0bad12 
  
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/TestCSVStorage.java
 6824a72 
  
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/TestDBStorage.java
 01bb29c 
  
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/TestIndexedStorage.java
 51f368b 
  
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/TestXMLLoader.java
 a340919 
  
contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java
 7d40ba0 
  
shims/src/hadoop20/org/apache/pig/backend/hadoop/executionengine/shims/HadoopShims.java
 9230480 
  
shims/src/hadoop23/org/apache/pig/backend/hadoop/executionengine/shims/HadoopShims.java
 1beee62 
  shims/test/hadoop20/org/apache/pig/test/MiniCluster.java 2c4559d 
  shims/test/hadoop23/org/apache/pig/test/MiniCluster.java ba97577 
  shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 6b3615b 
  src/org/apache/pig/backend/hadoop/datastorage/ConfigurationUtil.java 73e018a 
  src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java 
41c2b7c 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java
 2e2e737 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MRConfiguration.java
 e69de29 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java
 65a8c3b 
  
src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigOutputFormat.java
 854e598 
  src/org/apache/pig/backend/hadoop/executionengine/tez/PigProcessor.java 
17f3d89 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 
8492532 
  src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java 
2503494 
  src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java 
208ff99 
  
src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java 
4737b91 
  src/org/apache/pig/backend/hadoop/streaming/HadoopExecutableManager.java 
c369b26 
  src/org/apache/pig/builtin/PigStorage.java 675f138 
  src/org/apache/pig/builtin/TrevniStorage.java b6ae63c 
  src/org/apache/pig/data/SchemaTupleFrontend.java 9544d6c 
  src/org/apache/pig/impl/PigContext.java 46b83de 
  src/org/apache/pig/impl/builtin/DefaultIndexableLoader.java 22370f9 
  src/org/apache/pig/impl/builtin/GFCross.java fbb9883 
  src/org/apache/pig/impl/builtin/ReadScalars.java ced4065 
  src/org/apache/pig/impl/builtin/StreamingUDF.java 6cc967d 
  src/org/apache/pig/impl/util/UDFContext.java 0c639e9 
  src/org/apache/pig/impl/util/Utils.java 926a945 
  src/org/apache/pig/parser/QueryParserUtils.java 68fce96 
  src/org/apache/pig/scripting/ScriptingOutputCapturer.java 9fe7375 
  src/org/apache/pig/tools/pigstats/mapreduce/MRJobStats.java 7b58358 
  test/org/apache/pig/data/TestSchemaTuple.java baf76e9 
  test/org/apache/pig/parser/TestQueryParserUtils.java 4bd74f0 
  test/org/apache/pig/test/TestAccumulator.java 198ec15 
  test/org/apache/pig/test/TestBZip.java 82eac24 
  test/org/apache/pig/test/TestCombiner.java 36ed484 
  test/org/apache/pig/test/TestGFCross.java 76448cd 
  test/org/apache/pig/test/TestHBaseStorage.java 2957f9c 
  test/org/apache/pig/test/TestJobControlCompiler.java 6efee61 
  test/org/apache/pig/test/TestMRExecutionEngine.java 2f5ccdc 
  test/org/apache/pig/test/TestMapReduce.java 0b0e833 
  test/org/apache/pig/test/TestMergeJoin.java 37a69b6 
  test/org/apache/pig/test/TestMergeJoinOuter.java 942f615 
  test/org/apache/pig/test/TestMultiQueryBasic.java d5cf074 
  test/org/apache/pig/test/TestMultiQueryLocal.java 20fd979 
  test/org/apache/pig/test/TestParser.java 679b6ac 
  test/org/apache/pig/test/TestPigContext.java fe28e13 
  test/org/apache/pig/test/TestPigProgressReporting.java 4922e99 
  test/org/apache/pig/test/TestPigRunner.java b09ecd3 
  test/org/apache/pig/test/TestPigStorage.java f839869 
  test/org/apache/pig/test/TestPoissonSampleLoader.java be32594 
  test/org/apache/pig/test/TestStore.java 47a8606 
  test/org/apache/pig/test/TestStoreInstances.java d689c49 
  test/org/apache/pig/test/Util.java 507f2e2 
  test/org/apache/pig/tez/TestTezAutoParallelism.java 963bddf 
  
test/perf/pigmix/src/java/org/apache/pig/test/pigmix/datagen/DataGenerator.java 
8741d68 

Diff: https://reviews.apache.org/r/23960/diff/


Testing
-------


Thanks,

Cheolsoo Park

Reply via email to