[jira] [Commented] (HIVE-2090) Add DROP DATABASE ... CASCADE/RESTRICT
[ https://issues.apache.org/jira/browse/HIVE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022625#comment-13022625 ] Carl Steinbach commented on HIVE-2090: -- +1. Will commit if tests pass. Add DROP DATABASE ... CASCADE/RESTRICT Key: HIVE-2090 URL: https://issues.apache.org/jira/browse/HIVE-2090 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-2090.1.patch, HIVE-2090.2.patch, HIVE-2090.3.patch, HIVE-2090.4.patch, HIVE-2090.5.patch, HIVE-2090.6.patch A DROP DATABASE ... FORCE will be useful, when we use a database for isolation when doing some tests. Being able to force cleaning up the database will make test cleaning up easier. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2090) Add DROP DATABASE ... CASCADE/RESTRICT
[ https://issues.apache.org/jira/browse/HIVE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2090: - Status: Open (was: Patch Available) @Siying: I'm getting errors in DDLSemanticAnalyzer when I try to apply the patch. Can you please rebase it to trunk? Thanks! Add DROP DATABASE ... CASCADE/RESTRICT Key: HIVE-2090 URL: https://issues.apache.org/jira/browse/HIVE-2090 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-2090.1.patch, HIVE-2090.2.patch, HIVE-2090.3.patch, HIVE-2090.4.patch, HIVE-2090.5.patch, HIVE-2090.6.patch A DROP DATABASE ... FORCE will be useful, when we use a database for isolation when doing some tests. Being able to force cleaning up the database will make test cleaning up easier. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2070) SHOW GRANT grantTime field should be a human-readable timestamp
[ https://issues.apache.org/jira/browse/HIVE-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022642#comment-13022642 ] Carl Steinbach commented on HIVE-2070: -- +1. Will commit if tests pass. SHOW GRANT grantTime field should be a human-readable timestamp --- Key: HIVE-2070 URL: https://issues.apache.org/jira/browse/HIVE-2070 Project: Hive Issue Type: Improvement Components: Query Processor, Security Reporter: Jonathan Natkins Assignee: Jonathan Natkins Attachments: HIVE-2070.1.patch Unix timestamps are not super useful when trying to interpret metadata hive show grant user foo on table bar; database default table bar principalName foo principalType USER privilege Select grantTime 1300828549 grantor natty -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission
[ https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022670#comment-13022670 ] jirapos...@reviews.apache.org commented on HIVE-2093: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/566/#review517 --- trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/566/#comment1053 Spelling trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/566/#comment1058 Why not make this a boolean function? I understand that the convention in this class is to have functions that return 0, or something more than 0, but it looks like all of the other functions can be boolean valued as well. Returning an integer in this situation is confusing. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/566/#comment1059 Same deal here. Can you change this to return a boolean value? trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/566/#comment1061 You can eliminate a couple lines of code by using foreach syntax instead of an explicit iterator. trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java https://reviews.apache.org/r/566/#comment1062 I draw the line at nested ternary statements. Can you please use if/else instead? trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java https://reviews.apache.org/r/566/#comment1063 Nice to have: move DATABASE to the beginning of the list. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java https://reviews.apache.org/r/566/#comment1064 Why do you need to initialize the lock manager here, but not above in analyzeLockDatabase()? trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g https://reviews.apache.org/r/566/#comment1065 should be unlock database statement trunk/ql/src/test/queries/clientnegative/lockneg6.q https://reviews.apache.org/r/566/#comment1066 Please add some comments explaining what it is you're trying to test so that I can quickly tell how this differs from lockneg5, lockneg7, etc, etc. Please also consider changing the name of the file to something more descriptive so that I don't even have to read the comments. trunk/ql/src/test/results/clientnegative/authorization_fail_create_db.q.out https://reviews.apache.org/r/566/#comment1067 Please make this less ambiguous and easier to parse by capitalizing SHOW GRANT. trunk/ql/src/test/results/clientnegative/exim_15_part_nonpart.q.out https://reviews.apache.org/r/566/#comment1068 Any chance I can get a look at closed-source-hive? ;) - Carl On 2011-04-19 19:25:22, Siying Dong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/566/ bq. --- bq. bq. (Updated 2011-04-19 19:25:22) bq. bq. bq. Review request for hive, Yongqiang He and namit jain. bq. bq. bq. Summary bq. --- bq. bq. Still need to change some old tests' outputs. bq. bq. bq. This addresses bug HIVE-2093. bq. https://issues.apache.org/jira/browse/HIVE-2093 bq. bq. bq. Diffs bq. - bq. bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 1095164 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1095164 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1095164 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LockDatabaseDesc.java PRE-CREATION bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowLocksDesc.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/UnlockDatabaseDesc.java PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/authorization_fail_create_db.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/authorization_fail_drop_db.q
[jira] [Created] (HIVE-2122) Remove usage of deprecated methods from org.apache.hadoop.io package
Remove usage of deprecated methods from org.apache.hadoop.io package Key: HIVE-2122 URL: https://issues.apache.org/jira/browse/HIVE-2122 Project: Hive Issue Type: Bug Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Priority: Minor Serde code uses some deprecated methods from org.apache.hadoop.io. package. We should remove them. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2090) Add DROP DATABASE ... CASCADE/RESTRICT
[ https://issues.apache.org/jira/browse/HIVE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2090: -- Attachment: HIVE-2090.7.patch rebase Add DROP DATABASE ... CASCADE/RESTRICT Key: HIVE-2090 URL: https://issues.apache.org/jira/browse/HIVE-2090 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-2090.1.patch, HIVE-2090.2.patch, HIVE-2090.3.patch, HIVE-2090.4.patch, HIVE-2090.5.patch, HIVE-2090.6.patch, HIVE-2090.7.patch A DROP DATABASE ... FORCE will be useful, when we use a database for isolation when doing some tests. Being able to force cleaning up the database will make test cleaning up easier. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2090) Add DROP DATABASE ... CASCADE/RESTRICT
[ https://issues.apache.org/jira/browse/HIVE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2090: -- Status: Patch Available (was: Open) Add DROP DATABASE ... CASCADE/RESTRICT Key: HIVE-2090 URL: https://issues.apache.org/jira/browse/HIVE-2090 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-2090.1.patch, HIVE-2090.2.patch, HIVE-2090.3.patch, HIVE-2090.4.patch, HIVE-2090.5.patch, HIVE-2090.6.patch, HIVE-2090.7.patch A DROP DATABASE ... FORCE will be useful, when we use a database for isolation when doing some tests. Being able to force cleaning up the database will make test cleaning up easier. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2090) Add DROP DATABASE ... CASCADE/RESTRICT
[ https://issues.apache.org/jira/browse/HIVE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022880#comment-13022880 ] Carl Steinbach commented on HIVE-2090: -- I'm running tests now. Add DROP DATABASE ... CASCADE/RESTRICT Key: HIVE-2090 URL: https://issues.apache.org/jira/browse/HIVE-2090 Project: Hive Issue Type: New Feature Components: Metastore Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-2090.1.patch, HIVE-2090.2.patch, HIVE-2090.3.patch, HIVE-2090.4.patch, HIVE-2090.5.patch, HIVE-2090.6.patch, HIVE-2090.7.patch A DROP DATABASE ... FORCE will be useful, when we use a database for isolation when doing some tests. Being able to force cleaning up the database will make test cleaning up easier. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Building Hive using Eclipse on PC
Hello, I used Eclipse to get Hive through SVN (on my PC) following instructions at: http://hadoop.apache.org/ . However, I cannot build Hive through Eclipse (not getting 'Build Project'). Is there Wiki anywhere about 'Build Hive' ? Can anyone help ? Thanks, S. Sarkar
[jira] [Updated] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission
[ https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2093: -- Attachment: HIVE.2093.4.patch address Carl's comments. create/drop database should populate inputs/outputs and check concurrency and user permission - Key: HIVE-2093 URL: https://issues.apache.org/jira/browse/HIVE-2093 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Namit Jain Assignee: Siying Dong Attachments: HIVE.2093.1.patch, HIVE.2093.2.patch, HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.4.patch concurrency and authorization are needed for create/drop table. Also to make concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS DATABASE -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission
[ https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2093: -- Attachment: (was: HIVE.2093.4.patch) create/drop database should populate inputs/outputs and check concurrency and user permission - Key: HIVE-2093 URL: https://issues.apache.org/jira/browse/HIVE-2093 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Namit Jain Assignee: Siying Dong Attachments: HIVE.2093.1.patch, HIVE.2093.2.patch, HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.5.patch concurrency and authorization are needed for create/drop table. Also to make concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS DATABASE -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission
[ https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2093: -- Attachment: HIVE.2093.5.patch create/drop database should populate inputs/outputs and check concurrency and user permission - Key: HIVE-2093 URL: https://issues.apache.org/jira/browse/HIVE-2093 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Namit Jain Assignee: Siying Dong Attachments: HIVE.2093.1.patch, HIVE.2093.2.patch, HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.5.patch concurrency and authorization are needed for create/drop table. Also to make concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS DATABASE -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: review board for HIVE-2093
On 2011-04-21 07:41:22, Carl Steinbach wrote: trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java, line 2080 https://reviews.apache.org/r/566/diff/3/?file=15972#file15972line2080 Same deal here. Can you change this to return a boolean value? I'm going to follow convention this time, instead of making radical changes in such a JIRA. If you feel we should cleaning them up, we should open a separate JIRA. On 2011-04-21 07:41:22, Carl Steinbach wrote: trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java, line 699 https://reviews.apache.org/r/566/diff/3/?file=15976#file15976line699 Why do you need to initialize the lock manager here, but not above in analyzeLockDatabase()? If inputs and outputs are not empty, lock manager will automatically start. I look at the codes again. We probably don't need to put Db in inputs and outputs in such query, so that I remove them and set this value. - Siying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/566/#review517 --- On 2011-04-19 19:25:22, Siying Dong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/566/ --- (Updated 2011-04-19 19:25:22) Review request for hive, Yongqiang He and namit jain. Summary --- Still need to change some old tests' outputs. This addresses bug HIVE-2093. https://issues.apache.org/jira/browse/HIVE-2093 Diffs - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LockDatabaseDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowLocksDesc.java 1095164 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/UnlockDatabaseDesc.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/authorization_fail_create_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/authorization_fail_drop_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg6.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg7.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg8.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg9.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/database.q 1095164 trunk/ql/src/test/results/clientnegative/authorization_fail_create_db.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/authorization_fail_drop_db.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/database_create_already_exists.q.out 1095164 trunk/ql/src/test/results/clientnegative/database_create_invalid_name.q.out 1095164 trunk/ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out 1095164 trunk/ql/src/test/results/clientnegative/database_drop_not_empty.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_01_nonpart_over_loaded.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_02_all_part_over_overlap.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_03_nonpart_noncompat_colschema.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_04_nonpart_noncompat_colnumber.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_05_nonpart_noncompat_coltype.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_06_nonpart_noncompat_storage.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_07_nonpart_noncompat_ifof.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_08_nonpart_noncompat_serde.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_09_nonpart_noncompat_serdeparam.q.out 1095164 trunk/ql/src/test/results/clientnegative/exim_10_nonpart_noncompat_bucketing.q.out 1095164
Re: Review Request: review board for HIVE-2093
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/566/ --- (Updated 2011-04-21 22:21:03.518662) Review request for hive, Yongqiang He and namit jain. Changes --- addressing most of Carl's comments except incline. Summary --- Still need to change some old tests' outputs. This addresses bug HIVE-2093. https://issues.apache.org/jira/browse/HIVE-2093 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LockDatabaseDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowLocksDesc.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/UnlockDatabaseDesc.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/authorization_fail_create_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/authorization_fail_drop_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg_query_tbl_in_locked_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg_try_db_lock_conflict.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg_try_drop_locked_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg_try_lock_db_in_use.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/database.q 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_2.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_3.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_4.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_5.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_6.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_7.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_create_db.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/authorization_fail_drop_db.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/authorization_part.q.out 1095795 trunk/ql/src/test/results/clientnegative/database_create_already_exists.q.out 1095795 trunk/ql/src/test/results/clientnegative/database_create_invalid_name.q.out 1095795 trunk/ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out 1095795 trunk/ql/src/test/results/clientnegative/database_drop_not_empty.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_01_nonpart_over_loaded.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_02_all_part_over_overlap.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_03_nonpart_noncompat_colschema.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_04_nonpart_noncompat_colnumber.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_05_nonpart_noncompat_coltype.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_06_nonpart_noncompat_storage.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_07_nonpart_noncompat_ifof.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_08_nonpart_noncompat_serde.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_09_nonpart_noncompat_serdeparam.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_10_nonpart_noncompat_bucketing.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_11_nonpart_noncompat_sorting.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_13_nonnative_import.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_14_nonpart_part.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_15_part_nonpart.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_16_part_noncompat_schema.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_17_part_spec_underspec.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_18_part_spec_missing.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_19_external_over_existing.q.out 1095795
[jira] [Updated] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission
[ https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2093: -- Status: Patch Available (was: Open) create/drop database should populate inputs/outputs and check concurrency and user permission - Key: HIVE-2093 URL: https://issues.apache.org/jira/browse/HIVE-2093 Project: Hive Issue Type: Bug Components: Metastore, Security Reporter: Namit Jain Assignee: Siying Dong Attachments: HIVE.2093.1.patch, HIVE.2093.2.patch, HIVE.2093.3.patch, HIVE.2093.4.patch, HIVE.2093.5.patch concurrency and authorization are needed for create/drop table. Also to make concurrency work, it's better to have LOCK/UNLOCK DATABASE and SHOW LOCKS DATABASE -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission
[ https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022958#comment-13022958 ] jirapos...@reviews.apache.org commented on HIVE-2093: - bq. On 2011-04-21 07:41:22, Carl Steinbach wrote: bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java, line 2080 bq. https://reviews.apache.org/r/566/diff/3/?file=15972#file15972line2080 bq. bq. Same deal here. Can you change this to return a boolean value? I'm going to follow convention this time, instead of making radical changes in such a JIRA. If you feel we should cleaning them up, we should open a separate JIRA. bq. On 2011-04-21 07:41:22, Carl Steinbach wrote: bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java, line 699 bq. https://reviews.apache.org/r/566/diff/3/?file=15976#file15976line699 bq. bq. Why do you need to initialize the lock manager here, but not above in analyzeLockDatabase()? If inputs and outputs are not empty, lock manager will automatically start. I look at the codes again. We probably don't need to put Db in inputs and outputs in such query, so that I remove them and set this value. - Siying --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/566/#review517 --- On 2011-04-19 19:25:22, Siying Dong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/566/ bq. --- bq. bq. (Updated 2011-04-19 19:25:22) bq. bq. bq. Review request for hive, Yongqiang He and namit jain. bq. bq. bq. Summary bq. --- bq. bq. Still need to change some old tests' outputs. bq. bq. bq. This addresses bug HIVE-2093. bq. https://issues.apache.org/jira/browse/HIVE-2093 bq. bq. bq. Diffs bq. - bq. bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 1095164 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1095164 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1095164 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LockDatabaseDesc.java PRE-CREATION bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowLocksDesc.java 1095164 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/UnlockDatabaseDesc.java PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/authorization_fail_create_db.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/authorization_fail_drop_db.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/lockneg6.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/lockneg7.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/lockneg8.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/lockneg9.q PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/database.q 1095164 bq. trunk/ql/src/test/results/clientnegative/authorization_fail_create_db.q.out PRE-CREATION bq. trunk/ql/src/test/results/clientnegative/authorization_fail_drop_db.q.out PRE-CREATION bq. trunk/ql/src/test/results/clientnegative/database_create_already_exists.q.out 1095164 bq. trunk/ql/src/test/results/clientnegative/database_create_invalid_name.q.out 1095164 bq. trunk/ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out 1095164 bq.trunk/ql/src/test/results/clientnegative/database_drop_not_empty.q.out 1095164 bq. trunk/ql/src/test/results/clientnegative/exim_01_nonpart_over_loaded.q.out 1095164 bq. trunk/ql/src/test/results/clientnegative/exim_02_all_part_over_overlap.q.out 1095164 bq. trunk/ql/src/test/results/clientnegative/exim_03_nonpart_noncompat_colschema.q.out 1095164 bq. trunk/ql/src/test/results/clientnegative/exim_04_nonpart_noncompat_colnumber.q.out 1095164 bq. trunk/ql/src/test/results/clientnegative/exim_05_nonpart_noncompat_coltype.q.out 1095164 bq.
[jira] [Commented] (HIVE-2093) create/drop database should populate inputs/outputs and check concurrency and user permission
[ https://issues.apache.org/jira/browse/HIVE-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022959#comment-13022959 ] jirapos...@reviews.apache.org commented on HIVE-2093: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/566/ --- (Updated 2011-04-21 22:21:03.518662) Review request for hive, Yongqiang He and namit jain. Changes --- addressing most of Carl's comments except incline. Summary --- Still need to change some old tests' outputs. This addresses bug HIVE-2093. https://issues.apache.org/jira/browse/HIVE-2093 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/ReadEntity.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/LockDatabaseDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ShowLocksDesc.java 1095795 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/UnlockDatabaseDesc.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/authorization_fail_create_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/authorization_fail_drop_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg_query_tbl_in_locked_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg_try_db_lock_conflict.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg_try_drop_locked_db.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/lockneg_try_lock_db_in_use.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/database.q 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_2.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_3.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_4.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_5.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_6.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_7.q.out 1095795 trunk/ql/src/test/results/clientnegative/authorization_fail_create_db.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/authorization_fail_drop_db.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/authorization_part.q.out 1095795 trunk/ql/src/test/results/clientnegative/database_create_already_exists.q.out 1095795 trunk/ql/src/test/results/clientnegative/database_create_invalid_name.q.out 1095795 trunk/ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out 1095795 trunk/ql/src/test/results/clientnegative/database_drop_not_empty.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_01_nonpart_over_loaded.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_02_all_part_over_overlap.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_03_nonpart_noncompat_colschema.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_04_nonpart_noncompat_colnumber.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_05_nonpart_noncompat_coltype.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_06_nonpart_noncompat_storage.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_07_nonpart_noncompat_ifof.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_08_nonpart_noncompat_serde.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_09_nonpart_noncompat_serdeparam.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_10_nonpart_noncompat_bucketing.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_11_nonpart_noncompat_sorting.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_13_nonnative_import.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_14_nonpart_part.q.out 1095795 trunk/ql/src/test/results/clientnegative/exim_15_part_nonpart.q.out 1095795
Re: Review Request: Addressing Carl's comments.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/#review522 --- trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1070 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1071 formatting trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1072 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1073 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1074 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java https://reviews.apache.org/r/618/#comment1075 This change should go in a separate ticket. What does it mean to finalize a partition? trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1076 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1087 Instead of passing in raw Table/Partition/Database objects please wrap these objects in containers, e.g. CreateTableEvent, DropTableEvent, etc. Q: Whats the advantage of wrapper container objects? The advantage of container objects is that it allows us to evolve the API without breaking compatibility with older clients. With the current interface if I want to add a parameter to one of these methods I will break compatibility, but if we use container objects I can add a new field to the container without affecting older clients. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1077 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1078 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1079 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1080 Please remove this and include it in a separate patch. We need to discuss what finalizing a partition actually means, and how it relates to locking. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/618/#comment1083 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/618/#comment1082 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/618/#comment1081 spacing trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1086 spacing - Carl On 2011-04-18 18:44:05, Ashutosh Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/ --- (Updated 2011-04-18 18:44:05) Review request for hive and Carl Steinbach. Summary --- Addresses Carl's comment for previous patch. Also added a new method finalizePartition in metastore through which metastore client can indicate to metastore that partition can be finalized. This addresses bug HIVE-2038. https://issues.apache.org/jira/browse/HIVE-2038 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1094688 trunk/conf/hive-default.xml 1094688 trunk/metastore/if/hive_metastore.thrift 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1094688 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1094688 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1094688 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1094688 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1094688 trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1094688
[jira] [Updated] (HIVE-2038) Metastore listener
[ https://issues.apache.org/jira/browse/HIVE-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2038: - Status: Open (was: Patch Available) @Ashutosh: please see my comments here: https://reviews.apache.org/r/618/ Metastore listener -- Key: HIVE-2038 URL: https://issues.apache.org/jira/browse/HIVE-2038 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive-2038.patch, metastore_listener.patch, metastore_listener.patch, metastore_listener.patch Provide to way to observe changes happening on Metastore -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2038) Metastore listener
[ https://issues.apache.org/jira/browse/HIVE-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022969#comment-13022969 ] jirapos...@reviews.apache.org commented on HIVE-2038: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/#review522 --- trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1070 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1071 formatting trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1072 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1073 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/618/#comment1074 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java https://reviews.apache.org/r/618/#comment1075 This change should go in a separate ticket. What does it mean to finalize a partition? trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1076 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1087 Instead of passing in raw Table/Partition/Database objects please wrap these objects in containers, e.g. CreateTableEvent, DropTableEvent, etc. Q: Whats the advantage of wrapper container objects? The advantage of container objects is that it allows us to evolve the API without breaking compatibility with older clients. With the current interface if I want to add a parameter to one of these methods I will break compatibility, but if we use container objects I can add a new field to the container without affecting older clients. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1077 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1078 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1079 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1080 Please remove this and include it in a separate patch. We need to discuss what finalizing a partition actually means, and how it relates to locking. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/618/#comment1083 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/618/#comment1082 spacing trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java https://reviews.apache.org/r/618/#comment1081 spacing trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java https://reviews.apache.org/r/618/#comment1086 spacing - Carl On 2011-04-18 18:44:05, Ashutosh Chauhan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/618/ bq. --- bq. bq. (Updated 2011-04-18 18:44:05) bq. bq. bq. Review request for hive and Carl Steinbach. bq. bq. bq. Summary bq. --- bq. bq. Addresses Carl's comment for previous patch. bq. Also added a new method finalizePartition in metastore through which metastore client can indicate to metastore that partition can be finalized. bq. bq. bq. This addresses bug HIVE-2038. bq. https://issues.apache.org/jira/browse/HIVE-2038 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1094688 bq.trunk/conf/hive-default.xml 1094688 bq.trunk/metastore/if/hive_metastore.thrift 1094688 bq.trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1094688 bq.trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1094688 bq. trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1094688 bq. trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1094688 bq.
[jira] [Created] (HIVE-2123) CommandNeedRetryException needs release locks
CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2123: -- Attachment: HIVE-2123.1.patch CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2123: -- Attachment: (was: HIVE-2123.1.patch) CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Addressing Carl's comments.
On 2011-04-21 22:51:56, Carl Steinbach wrote: trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java, line 41 https://reviews.apache.org/r/618/diff/1/?file=15927#file15927line41 Instead of passing in raw Table/Partition/Database objects please wrap these objects in containers, e.g. CreateTableEvent, DropTableEvent, etc. Q: Whats the advantage of wrapper container objects? The advantage of container objects is that it allows us to evolve the API without breaking compatibility with older clients. With the current interface if I want to add a parameter to one of these methods I will break compatibility, but if we use container objects I can add a new field to the container without affecting older clients. It seems like this same concern also dictates that MetaStoreListener should be an abstract class instead of an interface. Can you please make this change? - Carl --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/#review522 --- On 2011-04-18 18:44:05, Ashutosh Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/ --- (Updated 2011-04-18 18:44:05) Review request for hive and Carl Steinbach. Summary --- Addresses Carl's comment for previous patch. Also added a new method finalizePartition in metastore through which metastore client can indicate to metastore that partition can be finalized. This addresses bug HIVE-2038. https://issues.apache.org/jira/browse/HIVE-2038 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1094688 trunk/conf/hive-default.xml 1094688 trunk/metastore/if/hive_metastore.thrift 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1094688 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1094688 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1094688 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1094688 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1094688 trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/NoOpListener.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1094688 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java PRE-CREATION Diff: https://reviews.apache.org/r/618/diff Testing --- Thanks, Ashutosh
[jira] [Updated] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2123: -- Attachment: HIVE-2123.1.patch CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2123: -- Assignee: Siying Dong Status: Patch Available (was: Open) CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2038) Metastore listener
[ https://issues.apache.org/jira/browse/HIVE-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023036#comment-13023036 ] jirapos...@reviews.apache.org commented on HIVE-2038: - bq. On 2011-04-21 22:51:56, Carl Steinbach wrote: bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java, line 41 bq. https://reviews.apache.org/r/618/diff/1/?file=15927#file15927line41 bq. bq. Instead of passing in raw Table/Partition/Database objects please wrap these objects in containers, e.g. CreateTableEvent, DropTableEvent, etc. bq. bq. Q: Whats the advantage of wrapper container objects? bq. bq. The advantage of container objects is that it allows us to evolve the API without breaking compatibility with older clients. With the current interface if I want to add a parameter to one of these methods I will break compatibility, but if we use container objects I can add a new field to the container without affecting older clients. It seems like this same concern also dictates that MetaStoreListener should be an abstract class instead of an interface. Can you please make this change? - Carl --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/#review522 --- On 2011-04-18 18:44:05, Ashutosh Chauhan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/618/ bq. --- bq. bq. (Updated 2011-04-18 18:44:05) bq. bq. bq. Review request for hive and Carl Steinbach. bq. bq. bq. Summary bq. --- bq. bq. Addresses Carl's comment for previous patch. bq. Also added a new method finalizePartition in metastore through which metastore client can indicate to metastore that partition can be finalized. bq. bq. bq. This addresses bug HIVE-2038. bq. https://issues.apache.org/jira/browse/HIVE-2038 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1094688 bq.trunk/conf/hive-default.xml 1094688 bq.trunk/metastore/if/hive_metastore.thrift 1094688 bq.trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1094688 bq.trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1094688 bq. trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1094688 bq. trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1094688 bq. trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1094688 bq. trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1094688 bq. trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1094688 bq.trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java PRE-CREATION bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/NoOpListener.java PRE-CREATION bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1094688 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java PRE-CREATION bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/618/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Ashutosh bq. bq. Metastore listener -- Key: HIVE-2038 URL: https://issues.apache.org/jira/browse/HIVE-2038 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive-2038.patch, metastore_listener.patch, metastore_listener.patch, metastore_listener.patch Provide to way to observe changes happening on Metastore -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Addressing Carl's comments.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/#review526 --- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/618/#comment1090 I think it should be possible to specify multiple metastore listeners. Can you please make the following changes: * Change the name of the conf property from hive.metastore.listener to hive.metastore.event.listeners and update the description to indicate that this is a comma separated list of listener classnames. * Change the default value for this property to be an empty string. * Update the code that initializes these listeners. You can cut and paste the code that's used to initialize hive.exec.pre.hooks - Carl On 2011-04-18 18:44:05, Ashutosh Chauhan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/ --- (Updated 2011-04-18 18:44:05) Review request for hive and Carl Steinbach. Summary --- Addresses Carl's comment for previous patch. Also added a new method finalizePartition in metastore through which metastore client can indicate to metastore that partition can be finalized. This addresses bug HIVE-2038. https://issues.apache.org/jira/browse/HIVE-2038 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1094688 trunk/conf/hive-default.xml 1094688 trunk/metastore/if/hive_metastore.thrift 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1094688 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1094688 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1094688 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1094688 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1094688 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1094688 trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1094688 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/NoOpListener.java PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1094688 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java PRE-CREATION trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java PRE-CREATION Diff: https://reviews.apache.org/r/618/diff Testing --- Thanks, Ashutosh
[jira] [Commented] (HIVE-2038) Metastore listener
[ https://issues.apache.org/jira/browse/HIVE-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023037#comment-13023037 ] jirapos...@reviews.apache.org commented on HIVE-2038: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/618/#review526 --- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/618/#comment1090 I think it should be possible to specify multiple metastore listeners. Can you please make the following changes: * Change the name of the conf property from hive.metastore.listener to hive.metastore.event.listeners and update the description to indicate that this is a comma separated list of listener classnames. * Change the default value for this property to be an empty string. * Update the code that initializes these listeners. You can cut and paste the code that's used to initialize hive.exec.pre.hooks - Carl On 2011-04-18 18:44:05, Ashutosh Chauhan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/618/ bq. --- bq. bq. (Updated 2011-04-18 18:44:05) bq. bq. bq. Review request for hive and Carl Steinbach. bq. bq. bq. Summary bq. --- bq. bq. Addresses Carl's comment for previous patch. bq. Also added a new method finalizePartition in metastore through which metastore client can indicate to metastore that partition can be finalized. bq. bq. bq. This addresses bug HIVE-2038. bq. https://issues.apache.org/jira/browse/HIVE-2038 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1094688 bq.trunk/conf/hive-default.xml 1094688 bq.trunk/metastore/if/hive_metastore.thrift 1094688 bq.trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1094688 bq.trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1094688 bq. trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1094688 bq. trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1094688 bq. trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1094688 bq. trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1094688 bq. trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1094688 bq.trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java PRE-CREATION bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1094688 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/NoOpListener.java PRE-CREATION bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1094688 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java PRE-CREATION bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/618/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Ashutosh bq. bq. Metastore listener -- Key: HIVE-2038 URL: https://issues.apache.org/jira/browse/HIVE-2038 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive-2038.patch, metastore_listener.patch, metastore_listener.patch, metastore_listener.patch Provide to way to observe changes happening on Metastore -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2117) insert overwrite ignoring partition location
[
https://issues.apache.org/jira/browse/HIVE-2117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Hunt updated HIVE-2117:
---
Attachment: HIVE-2117_br07.patch
This patch is a work in progress, it resolves this jira in branch 0.7 while
maintaining compatibility with the requirements from HIVE-1707. All unit tests
are passing with this patch applied, also fixes the example I provided in the
description (managed and external table).
I'm still working on two aspects of this JIRA; 1) creating a patch for trunk,
and 2) adding unit tests to verify this behavior.
insert overwrite ignoring partition location
Key: HIVE-2117
URL: https://issues.apache.org/jira/browse/HIVE-2117
Project: Hive
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Patrick Hunt
Priority: Critical
Attachments: HIVE-2117_br07.patch, data.txt
The following code works differently in 0.5.0 vs 0.7.0.
In 0.5.0 the partition location is respected.
However in 0.7.0 while the initial partition is create with the specified
location path/parta, the insert overwrite ... results in the partition
written to path/dt=a (note that path is the same in both cases).
{code}
create table foo_stg (bar INT, car INT);
load data local inpath 'data.txt' into table foo_stg;
create table foo4 (bar INT, car INT) partitioned by (dt STRING) LOCATION
'/user/hive/warehouse/foo4';
alter table foo4 add partition (dt='a') location
'/user/hive/warehouse/foo4/parta';
from foo_stg fs insert overwrite table foo4 partition (dt='a') select *;
{code}
From what I can tell HIVE-1707 introduced this via a change to
org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Path, String,
MapString, String, boolean, boolean)
specifically:
{code}
+ Path partPath = new Path(tbl.getDataLocation().getPath(),
+ Warehouse.makePartPath(partSpec));
+
+ Path newPartPath = new Path(loadPath.toUri().getScheme(), loadPath
+ .toUri().getAuthority(), partPath.toUri().getPath());
{code}
Reading the description on HIVE-1707 it seems that this may have been done
purposefully, however given the partition location is explicitly specified
for the partition in question it seems like that should be honored (esp give
the table location has not changed).
This difference in behavior is causing a regression in existing production
Hive based code. I'd like to take a stab at addressing this, any suggestions?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2117) insert overwrite ignoring partition location
[
https://issues.apache.org/jira/browse/HIVE-2117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Hunt updated HIVE-2117:
---
Priority: Blocker (was: Critical)
Affects Version/s: 0.8.0
insert overwrite ignoring partition location
Key: HIVE-2117
URL: https://issues.apache.org/jira/browse/HIVE-2117
Project: Hive
Issue Type: Bug
Affects Versions: 0.7.0, 0.8.0
Reporter: Patrick Hunt
Priority: Blocker
Attachments: HIVE-2117_br07.patch, data.txt
The following code works differently in 0.5.0 vs 0.7.0.
In 0.5.0 the partition location is respected.
However in 0.7.0 while the initial partition is create with the specified
location path/parta, the insert overwrite ... results in the partition
written to path/dt=a (note that path is the same in both cases).
{code}
create table foo_stg (bar INT, car INT);
load data local inpath 'data.txt' into table foo_stg;
create table foo4 (bar INT, car INT) partitioned by (dt STRING) LOCATION
'/user/hive/warehouse/foo4';
alter table foo4 add partition (dt='a') location
'/user/hive/warehouse/foo4/parta';
from foo_stg fs insert overwrite table foo4 partition (dt='a') select *;
{code}
From what I can tell HIVE-1707 introduced this via a change to
org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Path, String,
MapString, String, boolean, boolean)
specifically:
{code}
+ Path partPath = new Path(tbl.getDataLocation().getPath(),
+ Warehouse.makePartPath(partSpec));
+
+ Path newPartPath = new Path(loadPath.toUri().getScheme(), loadPath
+ .toUri().getAuthority(), partPath.toUri().getPath());
{code}
Reading the description on HIVE-1707 it seems that this may have been done
purposefully, however given the partition location is explicitly specified
for the partition in question it seems like that should be honored (esp give
the table location has not changed).
This difference in behavior is causing a regression in existing production
Hive based code. I'd like to take a stab at addressing this, any suggestions?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023045#comment-13023045 ] He Yongqiang commented on HIVE-2123: can you create a review board for this one? CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: CommandNeedRetryException needs release locks and some related code cleaning up
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/646/ --- Review request for hive, Yongqiang He and namit jain. Summary --- now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. This addresses bug HIVE-2123. https://issues.apache.org/jira/browse/HIVE-2123 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1095838 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095838 Diff: https://reviews.apache.org/r/646/diff Testing --- Thanks, Siying
[jira] [Commented] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023049#comment-13023049 ] jirapos...@reviews.apache.org commented on HIVE-2123: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/646/ --- Review request for hive, Yongqiang He and namit jain. Summary --- now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. This addresses bug HIVE-2123. https://issues.apache.org/jira/browse/HIVE-2123 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1095838 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095838 Diff: https://reviews.apache.org/r/646/diff Testing --- Thanks, Siying CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023054#comment-13023054 ] He Yongqiang commented on HIVE-2123: +1, running tests CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023057#comment-13023057 ] Carl Steinbach commented on HIVE-2123: -- @Yongqiang: Please give Siying a chance to respond to my comments before committing. CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023056#comment-13023056 ] jirapos...@reviews.apache.org commented on HIVE-2123: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/646/#review527 --- trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java https://reviews.apache.org/r/646/#comment1091 There's a try/catch block here, so why not do this the Java way and throw an exception instead of returning a status code? trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/646/#comment1092 Can you replace this with a DriverException class? trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/646/#comment1093 spacing - Carl On 2011-04-22 00:11:43, Siying Dong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/646/ bq. --- bq. bq. (Updated 2011-04-22 00:11:43) bq. bq. bq. Review request for hive, Yongqiang He and namit jain. bq. bq. bq. Summary bq. --- bq. bq. now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. bq. bq. bq. This addresses bug HIVE-2123. bq. https://issues.apache.org/jira/browse/HIVE-2123 bq. bq. bq. Diffs bq. - bq. bq.trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1095838 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095838 bq. bq. Diff: https://reviews.apache.org/r/646/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Siying bq. bq. CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2038) Metastore listener
[ https://issues.apache.org/jira/browse/HIVE-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023084#comment-13023084 ] Ashutosh Chauhan commented on HIVE-2038: bq. We need to discuss what finalizing a partition actually means, and how it relates to locking. Meaning of finalizing is not defined by MetaStore but by listener hook. It is left to listener to take whatever action it thinks is appropriate. I didn't get your concern w.r.t locking. Can you explain what you are thinking here w.r.t locking? Metastore listener -- Key: HIVE-2038 URL: https://issues.apache.org/jira/browse/HIVE-2038 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive-2038.patch, metastore_listener.patch, metastore_listener.patch, metastore_listener.patch Provide to way to observe changes happening on Metastore -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marquis Wang updated HIVE-1803: --- Attachment: HIVE-1803.12.patch New patch that implements John's suggestions about adding the hive.exec.rowoffset configuration variable. This patch fixes the issues with column numbers in explains. John, I'm still seeing some test failures in tests such as combine2.q, bucketmapjoin1.q, bucketmapjoin4.q. It looks like one of the numRows outputs is saying zero rows instead of some non-zero number before in an explain in each of these tests. I'm not really sure what could be causing this and don't see anything in this patch that can affect these tests. Do you have any ideas? Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023095#comment-13023095 ] John Sichi commented on HIVE-1803: -- I noticed that in one of my attempts with your previous patches. I think it's the exact same problem you were hitting very early on in the clinic in the Hudson setup, having to do with the stats temp database not getting deleted? Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-2124) Specified functions in the partitioning predicates should not generate a M/R job.
Specified functions in the partitioning predicates should not generate a M/R
job.
-
Key: HIVE-2124
URL: https://issues.apache.org/jira/browse/HIVE-2124
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.7.0, 0.6.0, 0.5.0
Reporter: Esteban Gutierrez
Priority: Minor
For certain situations specifying which functions should be evaluated once
would help to make syntax simpler to avoid launching M/R jobs.
Example:
# myhql.time=`date +%s` - constant
# counting rows from the last 30 days generates a M/R job using all the
partitions
$ hive -hiveconf myhql.time=`date +%s` -e SELECT COUNT(*) FROM mybigtable
WHERE mypartition =
from_unixtime(\${hiveconf:myhql.time}-2592000,'-MM-dd');
Suggested feature:
# will scan only the right partitions
$ hive -hiveconf hive.partition.evaluateonce=unix_timestamp -e SELECT COUNT(*)
FROM mybigtable WHERE mypartition =
from_unixtime(unix_timestamp()-2592000,'-MM-dd');
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023097#comment-13023097 ] Marquis Wang commented on HIVE-1803: I don't see anything that needs to be deleted in my checkout. Where is the stats temp database? Also, if you think it might just be something on our side, can you just run the tests and see if it passes for you? When I ran them I didn't see any other issues besides those, I don't think. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
[ https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023096#comment-13023096 ] jirapos...@reviews.apache.org commented on HIVE-1644: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/ --- (Updated 2011-04-22 03:50:54.602032) Review request for hive. Changes --- HIVE-1644.15.patch Summary --- Review request for HIVE-1644.12.patch This addresses bug HIVE-1644. https://issues.apache.org/jira/browse/HIVE-1644 Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2cdaeb6 conf/hive-default.xml 79ea477 ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8 ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 69ee03b ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c41bb32 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION Diff: https://reviews.apache.org/r/558/diff Testing --- Thanks, Russell use filter pushdown for automatically accessing indexes --- Key: HIVE-1644 URL: https://issues.apache.org/jira/browse/HIVE-1644 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: John Sichi Assignee: Russell Melick Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch, HIVE-1644.11.patch, HIVE-1644.12.patch, HIVE-1644.13.patch, HIVE-1644.14.patch, HIVE-1644.2.patch, HIVE-1644.3.patch, HIVE-1644.4.patch, HIVE-1644.5.patch, HIVE-1644.6.patch, HIVE-1644.7.patch, HIVE-1644.8.patch, HIVE-1644.9.patch HIVE-1226 provides utilities for analyzing filters which have been pushed down to a table scan. The next step is to use these for selecting available indexes and generating access plans for those indexes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: HIVE-1644 Use filter pushdown for automatically accessing indexes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/#review528 --- ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java https://reviews.apache.org/r/558/#comment1094 When we run a query on a non-partitioned table, we get a single partition in queryPartitions of the whole table (with an empty partSpec). Then, when we add the partition columns to the list of indexed column, we end up adding all the columns in the src table, instead of just the partitioned ones. If we make sure the partSpec isn't empty, this doesn't happen. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/558/#comment1095 We need to cast the work in this task to MapredWork in order to get the input size out (line 176). I'm not sure if this is the best place to do that checking. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/558/#comment1096 See above comment about MapredWork - Russell On 2011-04-22 03:50:54, Russell Melick wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/ --- (Updated 2011-04-22 03:50:54) Review request for hive. Summary --- Review request for HIVE-1644.12.patch This addresses bug HIVE-1644. https://issues.apache.org/jira/browse/HIVE-1644 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2cdaeb6 conf/hive-default.xml 79ea477 ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8 ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 69ee03b ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446 ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c41bb32 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION Diff: https://reviews.apache.org/r/558/diff Testing --- Thanks, Russell
[jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
[ https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023098#comment-13023098 ] jirapos...@reviews.apache.org commented on HIVE-1644: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/558/#review528 --- ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java https://reviews.apache.org/r/558/#comment1094 When we run a query on a non-partitioned table, we get a single partition in queryPartitions of the whole table (with an empty partSpec). Then, when we add the partition columns to the list of indexed column, we end up adding all the columns in the src table, instead of just the partitioned ones. If we make sure the partSpec isn't empty, this doesn't happen. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/558/#comment1095 We need to cast the work in this task to MapredWork in order to get the input size out (line 176). I'm not sure if this is the best place to do that checking. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java https://reviews.apache.org/r/558/#comment1096 See above comment about MapredWork - Russell On 2011-04-22 03:50:54, Russell Melick wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/558/ bq. --- bq. bq. (Updated 2011-04-22 03:50:54) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Review request for HIVE-1644.12.patch bq. bq. bq. This addresses bug HIVE-1644. bq. https://issues.apache.org/jira/browse/HIVE-1644 bq. bq. bq. Diffs bq. - bq. bq.common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2cdaeb6 bq.conf/hive-default.xml 79ea477 bq.ql/src/java/org/apache/hadoop/hive/ql/Driver.java ca337a8 bq.ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 69ee03b bq.ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b bq.ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446 bq.ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c bq.ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2 bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 374e123 bq.ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java c41bb32 bq.ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 bq.ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/558/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Russell bq. bq. use filter pushdown for automatically accessing indexes --- Key: HIVE-1644 URL: https://issues.apache.org/jira/browse/HIVE-1644 Project: Hive Issue Type: Improvement Components: Indexing Affects
Re: Review Request: CommandNeedRetryException needs release locks and some related code cleaning up
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/646/#review529 --- trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java https://reviews.apache.org/r/646/#comment1097 I tried to make some minimum change, but it looks much more difficult than I thought. The whole structure is built based on return codes, instead of exceptions. What I can do is that I can make Driver.run() not to throw any exception. In that case, we don't get two approaches twisted. - Siying On 2011-04-22 00:11:43, Siying Dong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/646/ --- (Updated 2011-04-22 00:11:43) Review request for hive, Yongqiang He and namit jain. Summary --- now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. This addresses bug HIVE-2123. https://issues.apache.org/jira/browse/HIVE-2123 Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1095838 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095838 Diff: https://reviews.apache.org/r/646/diff Testing --- Thanks, Siying
[jira] [Commented] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023102#comment-13023102 ] jirapos...@reviews.apache.org commented on HIVE-2123: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/646/#review529 --- trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java https://reviews.apache.org/r/646/#comment1097 I tried to make some minimum change, but it looks much more difficult than I thought. The whole structure is built based on return codes, instead of exceptions. What I can do is that I can make Driver.run() not to throw any exception. In that case, we don't get two approaches twisted. - Siying On 2011-04-22 00:11:43, Siying Dong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/646/ bq. --- bq. bq. (Updated 2011-04-22 00:11:43) bq. bq. bq. Review request for hive, Yongqiang He and namit jain. bq. bq. bq. Summary bq. --- bq. bq. now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. bq. bq. bq. This addresses bug HIVE-2123. bq. https://issues.apache.org/jira/browse/HIVE-2123 bq. bq. bq. Diffs bq. - bq. bq.trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1095838 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095838 bq. bq. Diff: https://reviews.apache.org/r/646/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Siying bq. bq. CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2123: -- Attachment: HIVE-2123.2.patch I changed Driver.execute() not to throw any exception. CommandNeedRetryException is translated to another return status code in this level. CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch, HIVE-2123.2.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2123: -- Attachment: (was: HIVE-2123.2.patch) CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch, HIVE-2123.2.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2123: -- Attachment: HIVE-2123.2.patch CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch, HIVE-2123.2.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: CommandNeedRetryException needs release locks and some related code cleaning up
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/646/ --- (Updated 2011-04-22 04:22:16.136899) Review request for hive, Yongqiang He and namit jain. Changes --- execute() not to throw any exception. Summary --- now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. This addresses bug HIVE-2123. https://issues.apache.org/jira/browse/HIVE-2123 Diffs (updated) - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1095838 trunk/hwi/src/java/org/apache/hadoop/hive/hwi/HWISessionItem.java 1095838 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1095838 trunk/ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessor.java 1095838 Diff: https://reviews.apache.org/r/646/diff Testing --- Thanks, Siying
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023121#comment-13023121 ] John Sichi commented on HIVE-1803: -- Yeah, I can give it a try and see if it passes. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
