[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025639#comment-13025639 ] Ning Zhang commented on HIVE-2125: -- The new patch looks good to me and addressed most of Carl's comments. Also about comments in the test .q files, I'm aware of the bug Yongqiang mentioned (test failed in some cases if comments are present, but passed otherwise). We should fix that bug in QTestsUtil and make a group effort to add comments to .q files as much as possible. Carl, what do you think? alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025083#comment-13025083 ] jirapos...@reviews.apache.org commented on HIVE-2125: - bq. On 2011-04-26 05:33:19, Carl Steinbach wrote: bq. trunk/ql/src/test/queries/clientpositive/alter_merge_index.q, line 1 bq. https://reviews.apache.org/r/665/diff/1/?file=17323#file17323line1 bq. bq. Please add some comments here too. Carl, what kind of comments do you want here? bq. On 2011-04-26 05:33:19, Carl Steinbach wrote: bq. trunk/ql/src/test/queries/clientnegative/alter_merge_index.q, line 1 bq. https://reviews.apache.org/r/665/diff/1/?file=17322#file17322line1 bq. bq. Please add some comments explaining what it is that you're testing, e.g. why is this a negative test? Why is expected to fail? bq. why is this a negative test? Why is expected to fail? should i put a positive test on a negative test dir? Should we explain all test cases in client negative dir? Come on, please put some positive suggestions. - Yongqiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/665/#review557 --- On 2011-04-25 22:47:17, Yongqiang He wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/665/ bq. --- bq. bq. (Updated 2011-04-25 22:47:17) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. alter table concatenate fails and deletes data bq. bq. It is because the number of reducers is set to -1. bq. bq. In this patch, it is set to zero. bq. bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job. bq. bq. bq. This addresses bug HIVE-2125. bq. https://issues.apache.org/jira/browse/HIVE-2125 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 bq.trunk/conf/hive-default.xml 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599 bq.trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/665/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Yongqiang bq. bq. alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025098#comment-13025098 ] jirapos...@reviews.apache.org commented on HIVE-2125: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/665/#review560 --- trunk/conf/hive-default.xml https://reviews.apache.org/r/665/#comment1187 Besides describing the effect of this configuration property, can you also please explain the motivation? Why would the user want to set this to true? Is there a valid reason for setting it to false? trunk/ql/src/test/queries/clientnegative/alter_merge_index.q https://reviews.apache.org/r/665/#comment1183 Should we explain all test cases in client negative dir? Emphatically yes! And since I'm a helpful person I'll even write the comment for you: Verify that an error is thrown if the user attempts to concatenate an indexed table when hive.exec.concatenate.check.index=true Also, please explicitly set hive.exec.concatenate.check.index=true at the beginning of this test. Also, please change the name of this file and the name of the table so that it matches the semantics of the ALTER TABLE command, i.e. alter_concatenate_indexed_table.q and src_rc_concatenate_test. trunk/ql/src/test/queries/clientpositive/alter_merge_index.q https://reviews.apache.org/r/665/#comment1186 How about this: * Verify that an indexed table can be concatenated when hive.exec.concatenate.check.index=false. * Verify that a partitioned table can be concatenated when hive.exec.concatenate.check.index=false. The point of the comments is help the reader distinguish between details that are relevant to the intended purpose of the test from those that are not. trunk/ql/src/test/queries/clientpositive/alter_merge_index.q https://reviews.apache.org/r/665/#comment1188 Please replace merge with concatenate. A better name for this test would be alter_concatenate_indexed_table.q trunk/ql/src/test/queries/clientpositive/alter_merge_index.q https://reviews.apache.org/r/665/#comment1185 These tables get cleaned up automatically. Please remove the drop statements. - Carl On 2011-04-25 22:47:17, Yongqiang He wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/665/ bq. --- bq. bq. (Updated 2011-04-25 22:47:17) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. alter table concatenate fails and deletes data bq. bq. It is because the number of reducers is set to -1. bq. bq. In this patch, it is set to zero. bq. bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job. bq. bq. bq. This addresses bug HIVE-2125. bq. https://issues.apache.org/jira/browse/HIVE-2125 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 bq.trunk/conf/hive-default.xml 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599 bq.trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/665/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Yongqiang bq. bq. alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate;
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025484#comment-13025484 ] jirapos...@reviews.apache.org commented on HIVE-2125: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/665/ --- (Updated 2011-04-26 22:27:30.586885) Review request for hive. Changes --- address Carl's comments about .q files Summary --- alter table concatenate fails and deletes data It is because the number of reducers is set to -1. In this patch, it is set to zero. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job. This addresses bug HIVE-2125. https://issues.apache.org/jira/browse/HIVE-2125 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096925 trunk/conf/hive-default.xml 1096925 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096925 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096925 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096925 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096925 trunk/ql/src/test/queries/clientnegative/alter_concatenate_indexed_table.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/alter_concatenate_indexed_table.q PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_concatenate_indexed_table.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/alter_concatenate_indexed_table.q.out PRE-CREATION Diff: https://reviews.apache.org/r/665/diff Testing --- Thanks, Yongqiang alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025485#comment-13025485 ] He Yongqiang commented on HIVE-2125: Carl, the comments for the config is the hive-site.xml. so i think it should not be added to .q files. And also putting comments before 'set' command can lead to some random errors. And all other comments are addressed in this new patch. alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch, HIVE-2125.2.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13024993#comment-13024993 ] He Yongqiang commented on HIVE-2125: It is because the number of reducers is set to -1. In this patch, it is set to zero. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job. alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13024994#comment-13024994 ] jirapos...@reviews.apache.org commented on HIVE-2125: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/665/ --- Review request for hive. Summary --- alter table concatenate fails and deletes data It is because the number of reducers is set to -1. In this patch, it is set to zero. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job. This addresses bug HIVE-2125. https://issues.apache.org/jira/browse/HIVE-2125 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 trunk/conf/hive-default.xml 1096599 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599 trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION Diff: https://reviews.apache.org/r/665/diff Testing --- Thanks, Yongqiang alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025005#comment-13025005 ] jirapos...@reviews.apache.org commented on HIVE-2125: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/665/#review554 --- trunk/conf/hive-default.xml https://reviews.apache.org/r/665/#comment1169 can you make the indentation consistent with the other property elements? trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/665/#comment1170 So after adding this, does the block-level merge after INSERT OVERWRITE be automatically supported? - Ning On 2011-04-25 22:47:17, Yongqiang He wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/665/ bq. --- bq. bq. (Updated 2011-04-25 22:47:17) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. alter table concatenate fails and deletes data bq. bq. It is because the number of reducers is set to -1. bq. bq. In this patch, it is set to zero. bq. bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job. bq. bq. bq. This addresses bug HIVE-2125. bq. https://issues.apache.org/jira/browse/HIVE-2125 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 bq.trunk/conf/hive-default.xml 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599 bq.trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/665/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Yongqiang bq. bq. alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025014#comment-13025014 ] jirapos...@reviews.apache.org commented on HIVE-2125: - bq. On 2011-04-25 23:20:59, Ning Zhang wrote: bq. trunk/conf/hive-default.xml, line 1042 bq. https://reviews.apache.org/r/665/diff/1/?file=17317#file17317line1042 bq. bq. can you make the indentation consistent with the other property elements? It shows the same indentation on my local. So it might just be an review board display issue. bq. On 2011-04-25 23:20:59, Ning Zhang wrote: bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java, line 1203 bq. https://reviews.apache.org/r/665/diff/1/?file=17321#file17321line1203 bq. bq. So after adding this, does the block-level merge after INSERT OVERWRITE be automatically supported? No. Not automatically supported. We still need to do some work there. but it is a separate issue. - Yongqiang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/665/#review554 --- On 2011-04-25 22:47:17, Yongqiang He wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/665/ bq. --- bq. bq. (Updated 2011-04-25 22:47:17) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. alter table concatenate fails and deletes data bq. bq. It is because the number of reducers is set to -1. bq. bq. In this patch, it is set to zero. bq. bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job. bq. bq. bq. This addresses bug HIVE-2125. bq. https://issues.apache.org/jira/browse/HIVE-2125 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 bq.trunk/conf/hive-default.xml 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599 bq.trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/665/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Yongqiang bq. bq. alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2125) alter table concatenate fails and deletes data
[ https://issues.apache.org/jira/browse/HIVE-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13025078#comment-13025078 ] jirapos...@reviews.apache.org commented on HIVE-2125: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/665/#review557 --- trunk/conf/hive-default.xml https://reviews.apache.org/r/665/#comment1177 The indentation looks the same in your editor because these are TABs. Please remove, and then configure your editor to disallow TABs. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/665/#comment1176 Spacing. trunk/ql/src/test/queries/clientnegative/alter_merge_index.q https://reviews.apache.org/r/665/#comment1174 Please add some comments explaining what it is that you're testing, e.g. why is this a negative test? Why is expected to fail? trunk/ql/src/test/queries/clientpositive/alter_merge_index.q https://reviews.apache.org/r/665/#comment1175 Please add some comments here too. - Carl On 2011-04-25 22:47:17, Yongqiang He wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/665/ bq. --- bq. bq. (Updated 2011-04-25 22:47:17) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. alter table concatenate fails and deletes data bq. bq. It is because the number of reducers is set to -1. bq. bq. In this patch, it is set to zero. bq. bq. Also added a move task as the child task of the merge task. added a conf to control whether to check index or not, and add the job name for the merge job. bq. bq. bq. This addresses bug HIVE-2125. bq. https://issues.apache.org/jira/browse/HIVE-2125 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1096599 bq.trunk/conf/hive-default.xml 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1096599 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1096599 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1096599 bq.trunk/ql/src/test/queries/clientnegative/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/alter_merge_index.q PRE-CREATION bq.trunk/ql/src/test/results/clientnegative/alter_merge_index.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientpositive/alter_merge_index.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/665/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Yongqiang bq. bq. alter table concatenate fails and deletes data -- Key: HIVE-2125 URL: https://issues.apache.org/jira/browse/HIVE-2125 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: He Yongqiang Priority: Critical Attachments: HIVE-2125.1.patch the number of reducers is not set by this command (unlike other hive queries). since mapred.reduce.tasks=-1 (to let hive infer this automatically) - jobtracker fails the job (number of reducers cannot be negative) hive alter table ad_imps_2 partition(ds='2009-06-16') concatenate; alter table ad_imps_2 partition(ds='2009-06-16') concatenate; Starting Job = job_201103101203_453180, Tracking URL = http://curium.data.facebook.com:50030/jobdetails.jsp?jobid=job_201103101203_453180 Kill Command = /mnt/vol/hive/sites/curium/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=curium.data.facebook.com:50029 -kill job_201103101203_453180 Hadoop job information for null: number of mappers: 0; number of reducers: 0 2011-04-22 10:21:24,046 null map = 100%, reduce = 100% Ended Job = job_201103101203_453180 with errors Moved to trash: /user/facebook/warehouse/ad_imps_2/_backup.ds=2009-06-16 after the job fails - the partition is deleted thankfully it's still in trash -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
