Re: Review Request 25495: HIVE-7776, enable sample10.q
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25495/ --- (Updated 十月 10, 2014, 9:05 a.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Changes --- update sample10.q.output for fixed partition file name. Bugs: HIVE-7776 https://issues.apache.org/jira/browse/HIVE-7776 Repository: hive-git Description --- Hive get task Id through 2 ways in Utilities::getTaskId: get parameter value of mapred.task.id from configuration. generate random value while #1 return null. set mapred.task.id on executor side as we can build it through TaskContext now. Diffs (updated) - itests/src/test/resources/testconfiguration.properties 468a441 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 6d1e57b ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 94a6141 ql/src/test/results/clientpositive/spark/sample10.q.out PRE-CREATION Diff: https://reviews.apache.org/r/25495/diff/ Testing --- Thanks, chengxiang li
Re: Review Request 25495: HIVE-7776, enable sample10.q
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25495/#review56189 --- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java https://reviews.apache.org/r/25495/#comment96513 Can we combine this line with the previous one? ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java https://reviews.apache.org/r/25495/#comment96517 Same as above. - Xuefu Zhang On Oct. 10, 2014, 9:05 a.m., chengxiang li wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25495/ --- (Updated Oct. 10, 2014, 9:05 a.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-7776 https://issues.apache.org/jira/browse/HIVE-7776 Repository: hive-git Description --- Hive get task Id through 2 ways in Utilities::getTaskId: get parameter value of mapred.task.id from configuration. generate random value while #1 return null. set mapred.task.id on executor side as we can build it through TaskContext now. Diffs - itests/src/test/resources/testconfiguration.properties 468a441 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 6d1e57b ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 94a6141 ql/src/test/results/clientpositive/spark/sample10.q.out PRE-CREATION Diff: https://reviews.apache.org/r/25495/diff/ Testing --- Thanks, chengxiang li
Re: Review Request 25495: HIVE-7776, enable sample10.q
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25495/ --- (Updated 九月 29, 2014, 9:10 a.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-7776 https://issues.apache.org/jira/browse/HIVE-7776 Repository: hive-git Description (updated) --- Hive get task Id through 2 ways in Utilities::getTaskId: get parameter value of mapred.task.id from configuration. generate random value while #1 return null. set mapred.task.id on executor side as we can build it through TaskContext now. Diffs - itests/src/test/resources/testconfiguration.properties 155abad ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 3ff0782 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 02f9d99 ql/src/test/results/clientpositive/spark/sample10.q.out PRE-CREATION Diff: https://reviews.apache.org/r/25495/diff/ Testing --- Thanks, chengxiang li
Re: Review Request 25495: HIVE-7776, enable sample10.q
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25495/ --- (Updated 九月 29, 2014, 9:11 a.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-7776 https://issues.apache.org/jira/browse/HIVE-7776 Repository: hive-git Description --- Hive get task Id through 2 ways in Utilities::getTaskId: get parameter value of mapred.task.id from configuration. generate random value while #1 return null. set mapred.task.id on executor side as we can build it through TaskContext now. Diffs (updated) - itests/src/test/resources/testconfiguration.properties 89243fc ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 1674d4b ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 0b8b7c9 ql/src/test/results/clientpositive/spark/sample10.q.out PRE-CREATION Diff: https://reviews.apache.org/r/25495/diff/ Testing --- Thanks, chengxiang li
Re: Review Request 25495: HIVE-7776, enable sample10.q
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25495/ --- (Updated 九月 29, 2014, 9:13 a.m.) Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-7776 https://issues.apache.org/jira/browse/HIVE-7776 Repository: hive-git Description --- Hive get task Id through 2 ways in Utilities::getTaskId: get parameter value of mapred.task.id from configuration. generate random value while #1 return null. set mapred.task.id on executor side as we can build it through TaskContext now. Diffs (updated) - itests/src/test/resources/testconfiguration.properties 89243fc ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java 1674d4b ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HivePairFlatMapFunction.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java 0b8b7c9 ql/src/test/results/clientpositive/spark/sample10.q.out PRE-CREATION Diff: https://reviews.apache.org/r/25495/diff/ Testing --- Thanks, chengxiang li
Review Request 25495: HIVE-7776, enable sample10.q
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25495/ --- Review request for hive, Brock Noland and Xuefu Zhang. Bugs: HIVE-7776 https://issues.apache.org/jira/browse/HIVE-7776 Repository: hive-git Description --- Hive get task Id through 2 ways in Utilities::getTaskId: get parameter value of mapred.task.id from configuration. generate random value while #1 return null. Currently, Hive on Spark can't get parameter value of mapred.task.id from configuration. FileSinkOperator use taskid to distinct different bucket file name, FileSinkOperator should take taskid as field variable and initiate it only once since one FileSinkOperator instance only refered in one task. but FileSinkOperator call Utilities::getTaskId to get new taskId each time, for this issue, it would cause more bucket files than bucket number, which lead to unexpected result of tablesample queries. Diffs - itests/src/test/resources/testconfiguration.properties 155abad ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 3ff0782 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 02f9d99 ql/src/test/results/clientpositive/spark/sample10.q.out PRE-CREATION Diff: https://reviews.apache.org/r/25495/diff/ Testing --- Thanks, chengxiang li
