----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24627/#review52679 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java <https://reviews.apache.org/r/24627/#comment91637> All these were exisiting code taken from MergeMapper.java. Anyways, I rewrote the comment in the new patch. Also merged fixTmpPath and fixTmpPathConcatenate method to single method in the new patch. ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java <https://reviews.apache.org/r/24627/#comment91638> Updated in new patch. ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java <https://reviews.apache.org/r/24627/#comment91639> Updated comment in new patch. ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java <https://reviews.apache.org/r/24627/#comment91640> This is all gone in new patch. ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java <https://reviews.apache.org/r/24627/#comment91641> Thats the eclipse. I used intellij. Perhaps both seems to do the opposite. :) ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java <https://reviews.apache.org/r/24627/#comment91642> Fixed it. ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java <https://reviews.apache.org/r/24627/#comment91643> If I use Map interface here, then I need to cast it to LinkedHashMap when I set aliasToWork(). To avoid casting I am using ListkedHashMap on LHS. ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java <https://reviews.apache.org/r/24627/#comment91644> Fixed it. ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java <https://reviews.apache.org/r/24627/#comment91645> This is a dummy output format which is set in DagUtils/MergeFileTask. I am just using to make sure the operator pipeline for fast file merge is initialized properly. If the operator pipeline is wrongly initialized with say TS -> FS, then FS will get record writer from this output format throwing RuntimeException. If OFM or RFM operators are initialized then this will never be called. The RCFile and ORC file writers handles output file opening and closing themselves. It does not use the standard record writer interfaces for writing the output. Both RCFile and ORC use custom interfaces for block level and stripe level writing respectively. - Prasanth_J On Sept. 6, 2014, 2:03 a.m., Prasanth_J wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/24627/ > ----------------------------------------------------------- > > (Updated Sept. 6, 2014, 2:03 a.m.) > > > Review request for hive and Gunther Hagleitner. > > > Bugs: HIVE-7704 > https://issues.apache.org/jira/browse/HIVE-7704 > > > Repository: hive-git > > > Description > ------- > > Currently tez falls back to MR task for merge file task. It will beneficial > to convert the merge file tasks to tez task to make use of the performance > gains from tez. > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54e2b18 > itests/src/test/resources/testconfiguration.properties 99049ca > > ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/OperatorType.java > 6f23575 > ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java e076683 > ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 7477199 > ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java 8946221 > ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 3d74459 > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 2d9b9c3 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java 4ff568d1 > > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileRecordProcessor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MergeFileTezProcessor.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java 994721f > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 831e6a5 > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileInputFormat.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileMapper.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileOutputFormat.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileTask.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeInputFormat.java > 4651920 > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeMapper.java 6c691b1 > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeOutputFormat.java > a3ce699 > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeTask.java c30476b > ql/src/java/org/apache/hadoop/hive/ql/io/merge/MergeWork.java 9efee3c > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileMergeMapper.java > 13ec642 > > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFileStripeMergeInputFormat.java > a6c92fb > ql/src/java/org/apache/hadoop/hive/ql/io/orc/Writer.java c391b0e > ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java 195d60e > > ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileBlockMergeInputFormat.java > 6809c79 > > ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java > dee6b1c > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 7129ed8 > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java 11a9419 > ql/src/java/org/apache/hadoop/hive/ql/plan/FileMergeDesc.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/OrcFileMergeDesc.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/RCFileMergeDesc.java > PRE-CREATION > ql/src/test/queries/clientpositive/list_bucket_dml_8.q 9e81b8d > ql/src/test/queries/clientpositive/orc_merge1.q ee65b98 > ql/src/test/queries/clientpositive/orc_merge5.q PRE-CREATION > ql/src/test/queries/clientpositive/orc_merge6.q PRE-CREATION > ql/src/test/queries/clientpositive/orc_merge7.q PRE-CREATION > ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out ea37c36 > ql/src/test/results/clientpositive/list_bucket_dml_10.q.out e9367ac > ql/src/test/results/clientpositive/list_bucket_dml_4.q.out 99496d5 > ql/src/test/results/clientpositive/list_bucket_dml_6.q.out d5deadb > ql/src/test/results/clientpositive/list_bucket_dml_7.q.out 4aea4db > ql/src/test/results/clientpositive/list_bucket_dml_9.q.out f94a3cc > ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out 0899648 > ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out 0653469 > ql/src/test/results/clientpositive/orc_createas1.q.out b0c58dd > ql/src/test/results/clientpositive/orc_merge1.q.out fc3e206 > ql/src/test/results/clientpositive/orc_merge3.q.out 258f538 > ql/src/test/results/clientpositive/orc_merge5.q.out PRE-CREATION > ql/src/test/results/clientpositive/orc_merge6.q.out PRE-CREATION > ql/src/test/results/clientpositive/orc_merge7.q.out PRE-CREATION > ql/src/test/results/clientpositive/rcfile_createas1.q.out cdfa036 > ql/src/test/results/clientpositive/rcfile_merge1.q.out ac6a2bd > ql/src/test/results/clientpositive/rcfile_merge2.q.out d8a61f3 > ql/src/test/results/clientpositive/rcfile_merge3.q.out 5c717b5 > ql/src/test/results/clientpositive/tez/orc_merge1.q.out f79f878 > ql/src/test/results/clientpositive/tez/orc_merge5.q.out PRE-CREATION > ql/src/test/results/clientpositive/tez/orc_merge6.q.out PRE-CREATION > ql/src/test/results/clientpositive/tez/orc_merge7.q.out PRE-CREATION > ql/src/test/results/clientpositive/tez/orc_merge_incompat1.q.out > PRE-CREATION > ql/src/test/results/clientpositive/tez/orc_merge_incompat2.q.out > PRE-CREATION > ql/src/test/results/clientpositive/union_remove_10.q.out 74b9e68 > ql/src/test/results/clientpositive/union_remove_11.q.out e0bd498 > ql/src/test/results/clientpositive/union_remove_12.q.out 033475f > ql/src/test/results/clientpositive/union_remove_13.q.out 3c7f8fa > ql/src/test/results/clientpositive/union_remove_14.q.out 315c958 > ql/src/test/results/clientpositive/union_remove_16.q.out 721caa0 > ql/src/test/results/clientpositive/union_remove_9.q.out 5a9c5a2 > > Diff: https://reviews.apache.org/r/24627/diff/ > > > Testing > ------- > > > Thanks, > > Prasanth_J > >
