[jira] [Assigned] (HIVE-9543) MetaException(message:Metastore contains multiple versions)
[
https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fuqiang cao reassigned HIVE-9543:
-
Assignee: fuqiang cao
> MetaException(message:Metastore contains multiple versions)
> ---
>
> Key: HIVE-9543
> URL: https://issues.apache.org/jira/browse/HIVE-9543
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.13.1
>Reporter: Junyong Li
>Assignee: fuqiang cao
>
> When i run bin/hive command, i got the following exception:
> {noformat}
> Logging initialized using configuration in
> jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
> ... 12 more
> Caused by: MetaException(message:Metastore contains multiple versions)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:356)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:171)
> ... 17 more
> {noformat}
> And i have found two record in metastore table VERSION. after reading source
> code, i found following code maybe cause the problem:
> In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289:
> {noformat}
> String schemaVer = getMetaStoreSchemaVersion();
> if (schemaVer == null) {
> // metastore has no schema version information
> if
[jira] [Updated] (HIVE-9543) MetaException(message:Metastore contains multiple versions)
[
https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fuqiang cao updated HIVE-9543:
--
Assignee: (was: fuqiang cao)
> MetaException(message:Metastore contains multiple versions)
> ---
>
> Key: HIVE-9543
> URL: https://issues.apache.org/jira/browse/HIVE-9543
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.13.1
>Reporter: Junyong Li
>
> When i run bin/hive command, i got the following exception:
> {noformat}
> Logging initialized using configuration in
> jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
> ... 12 more
> Caused by: MetaException(message:Metastore contains multiple versions)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:356)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:171)
> ... 17 more
> {noformat}
> And i have found two record in metastore table VERSION. after reading source
> code, i found following code maybe cause the problem:
> In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289:
> {noformat}
> String schemaVer = getMetaStoreSchemaVersion();
> if (schemaVer == null) {
> // metastore has no schema version information
> if (strictValidation) {
> throw
[jira] [Commented] (HIVE-11600) Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
[
https://issues.apache.org/jira/browse/HIVE-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731829#comment-14731829
]
Lefty Leverenz commented on HIVE-11600:
---
Doc note: Should documentation for this wait until HIVE-11738 is done?
Version: I'm putting 2.0.0 in Fixed Version/s, so if that's wrong please
change it.
> Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
>
>
> Key: HIVE-11600
> URL: https://issues.apache.org/jira/browse/HIVE-11600
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-11600.01.patch, HIVE-11600.02.patch,
> HIVE-11600.03.patch, HIVE-11600.04.patch, HIVE-11600.05.patch
>
>
> Current hive only support single column in clause, e.g.,
> {code}select * from src where col0 in (v1,v2,v3);{code}
> We want it to support
> {code}select * from src where (col0,col1+3) in
> ((col0+v1,v2),(v3,v4-col1));{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11600) Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
[
https://issues.apache.org/jira/browse/HIVE-11600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-11600:
--
Fix Version/s: 2.0.0
> Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
>
>
> Key: HIVE-11600
> URL: https://issues.apache.org/jira/browse/HIVE-11600
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.0.0
>
> Attachments: HIVE-11600.01.patch, HIVE-11600.02.patch,
> HIVE-11600.03.patch, HIVE-11600.04.patch, HIVE-11600.05.patch
>
>
> Current hive only support single column in clause, e.g.,
> {code}select * from src where col0 in (v1,v2,v3);{code}
> We want it to support
> {code}select * from src where (col0,col1+3) in
> ((col0+v1,v2),(v3,v4-col1));{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11743) HBase Port conflict for MiniHBaseCluster
[ https://issues.apache.org/jira/browse/HIVE-11743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731872#comment-14731872 ] Thejas M Nair commented on HIVE-11743: -- +1 > HBase Port conflict for MiniHBaseCluster > > > Key: HIVE-11743 > URL: https://issues.apache.org/jira/browse/HIVE-11743 > Project: Hive > Issue Type: Sub-task > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: hbase-metastore-branch > > Attachments: HIVE-11743.1.patch > > > HMaster http port conflict. Presumably a bug in HBaseTestingUtility. It > suppose not to use default port for everything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9756) LLAP: use log4j 2 for llap
[ https://issues.apache.org/jira/browse/HIVE-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731842#comment-14731842 ] Prasanth Jayachandran commented on HIVE-9756: - I think this patch will swap out the MDC just before running. Does that solve your previous issue with MDC? > LLAP: use log4j 2 for llap > -- > > Key: HIVE-9756 > URL: https://issues.apache.org/jira/browse/HIVE-9756 > Project: Hive > Issue Type: Sub-task >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran > Attachments: HIVE-9756.1.patch, HIVE-9756.2.patch > > > For the INFO logging, we'll need to use the log4j-jcl 2.x upgrade-path to get > throughput friendly logging. > http://logging.apache.org/log4j/2.0/manual/async.html#Performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11663) Auto load/unload custom udf function for hive cli and hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-11663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731851#comment-14731851 ] Lefty Leverenz commented on HIVE-11663: --- [~liuzongquan@mls], sorry about the delay. Here's my review of the configuration parameters in HiveConf.java, patch 2. # Why did you remove four comment lines starting with "// Hadoop Configuration Properties"? # Formatting: For hive.server2.customized.udf.jars.path and hive.server2.customized.udf.properties please insert newlines (\n) in the descriptions as shown on lines 324 and 325, start the description of hive.server2.customized.udf.jars.path on a new line, and use standard indenting. # Capitalization: For hive.server2.customized.udf.jars.path please change "The path on hdfs" to "The path on HDFS" and "means the hdfs path" to "means the HDFS path". # Edit: For hive.server2.customized.udf.jars.path please change "When hdfs:// missing" to "When hdfs:// is missing". # Edits: For hive.server2.customized.udf.properties please change "customized udf" to "customized UDF", change "defination" to "definition", change "It's must be under $" to "It must be under the $", and change "The patern is as below." to "The pattern is". A Review Board entry for this would make it easier to review, because comments could be posted on particular lines. > Auto load/unload custom udf function for hive cli and hiveserver2 > - > > Key: HIVE-11663 > URL: https://issues.apache.org/jira/browse/HIVE-11663 > Project: Hive > Issue Type: Improvement > Components: CLI, Configuration >Affects Versions: 0.14.0, 1.0.0, 1.0.1, 1.1.1, 1.2.1 >Reporter: liuzongquan >Assignee: liuzongquan > Labels: features, patch > Attachments: HIVE-11663-2.patch > > Original Estimate: 96h > Time Spent: 96h > Remaining Estimate: 0h > > when adding custom functions used in hiveserver2, the most method is re-build > the hive source code, re-dist and restart hiveserver2. This way will produce > big cost for service user and cluster manager. The best way, in my opinion, > the custom udf should be like a plugin to the hiveserver2 and hive cli, and > users can add and remove at run-time, especially for hiveserver2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6892) Permission inheritance issues
[ https://issues.apache.org/jira/browse/HIVE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731859#comment-14731859 ] Lefty Leverenz commented on HIVE-6892: -- What needs to be changed in the wiki? (Perhaps it should be a separate JIRA issue.) * [Permission Inheritance in Hive | https://cwiki.apache.org/confluence/display/Hive/Permission+Inheritance+in+Hive] > Permission inheritance issues > - > > Key: HIVE-6892 > URL: https://issues.apache.org/jira/browse/HIVE-6892 > Project: Hive > Issue Type: Bug > Components: Security >Affects Versions: 0.13.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > *HDFS Background* > * When a file or directory is created, its owner is the user identity of the > client process, and its group is inherited from parent (the BSD rule). > Permissions are taken from default umask. Extended Acl's are taken from > parent unless they are set explicitly. > *Goals* > To reduce need to set fine-grain file security props after every operation, > users may want the following Hive warehouse file/dir to auto-inherit security > properties from their directory parents: > * Directories created by new database/table/partition/bucket > * Files added to tables via load/insert > * Table directories exported/imported (open question of whether exported > table inheriting perm from new parent needs another flag) > What may be inherited: > * Basic file permission > * Groups (already done by HDFS for new directories) > * Extended ACL's (already done by HDFS for new directories) > *Behavior* > * When "hive.warehouse.subdir.inherit.perms" flag is enabled in Hive, Hive > will try to do all above inheritances. In the future, we can add more flags > for more finer-grained control. > * Failure by Hive to inherit will not cause operation to fail. Rule of thumb > of when security-prop inheritance will happen is the following: > ** To run chmod, a user must be the owner of the file, or else a super-user. > ** To run chgrp, a user must be the owner of files, or else a super-user. > ** Hence, user that hive runs as (either 'hive' or the logged-in user in case > of impersonation), must be super-user or owner of the file whose security > properties are going to be changed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9543) MetaException(message:Metastore contains multiple versions)
[
https://issues.apache.org/jira/browse/HIVE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731888#comment-14731888
]
fuqiang cao commented on HIVE-9543:
---
hiļ¼have you worked out any approach to solve the problem? i have the same
problem. thanks. my QQ:363190371 email:rain841...@163.com
> MetaException(message:Metastore contains multiple versions)
> ---
>
> Key: HIVE-9543
> URL: https://issues.apache.org/jira/browse/HIVE-9543
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.13.1
>Reporter: Junyong Li
>
> When i run bin/hive command, i got the following exception:
> {noformat}
> Logging initialized using configuration in
> jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException:
> java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:62)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
> ... 7 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
> ... 12 more
> Caused by: MetaException(message:Metastore contains multiple versions)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
> at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:356)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:54)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:171)
> ... 17 more
> {noformat}
> And i have found two record in metastore table VERSION. after reading source
> code, i found following code maybe cause the problem:
> In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289:
> {noformat}
> String schemaVer = getMetaStoreSchemaVersion();
>
[jira] [Commented] (HIVE-11703) Make RegExp and RLike reserved keywords
[ https://issues.apache.org/jira/browse/HIVE-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14731831#comment-14731831 ] Lefty Leverenz commented on HIVE-11703: --- Version note: This is fixed in master (version 2.0.0) with the patch for HIVE-11600. I'm adding 2.0.0 to Fix Version/s. Doc note: REGEXP and RLIKE are currently listed as nonreserved keywords in the wiki, so they need to be moved to the reserved list with version notes. * [DDL -- Non-reserved Keywords | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-Non-reservedKeywords] > Make RegExp and RLike reserved keywords > --- > > Key: HIVE-11703 > URL: https://issues.apache.org/jira/browse/HIVE-11703 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.0.0 > > > RegExp and RLike are treated as "precedenceEqualNegatableOperator" in Hive. > They actually come from MySQL. Both of them are not keywords in SQL2011. But > they are reserved keywords in MySQL. Making them reserved ones can eliminate > the current 14 ambiguities that we have in current Hive. If users still would > like to use them as identifiers/function names, users can "set > hive.support.sql11.reserved.keywords=false;" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11703) Make RegExp and RLike reserved keywords
[ https://issues.apache.org/jira/browse/HIVE-11703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-11703: -- Fix Version/s: 2.0.0 > Make RegExp and RLike reserved keywords > --- > > Key: HIVE-11703 > URL: https://issues.apache.org/jira/browse/HIVE-11703 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.0.0 > > > RegExp and RLike are treated as "precedenceEqualNegatableOperator" in Hive. > They actually come from MySQL. Both of them are not keywords in SQL2011. But > they are reserved keywords in MySQL. Making them reserved ones can eliminate > the current 14 ambiguities that we have in current Hive. If users still would > like to use them as identifiers/function names, users can "set > hive.support.sql11.reserved.keywords=false;" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11743) HBase Port conflict for MiniHBaseCluster
[ https://issues.apache.org/jira/browse/HIVE-11743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai resolved HIVE-11743. --- Resolution: Fixed Hadoop Flags: Reviewed Patch committed to hbase-metastore. Thanks Thejas for review! > HBase Port conflict for MiniHBaseCluster > > > Key: HIVE-11743 > URL: https://issues.apache.org/jira/browse/HIVE-11743 > Project: Hive > Issue Type: Sub-task > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: hbase-metastore-branch > > Attachments: HIVE-11743.1.patch > > > HMaster http port conflict. Presumably a bug in HBaseTestingUtility. It > suppose not to use default port for everything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11711) Merge hbase-metastore branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-11711: -- Attachment: HIVE-11711.5.patch > Merge hbase-metastore branch to trunk > - > > Key: HIVE-11711 > URL: https://issues.apache.org/jira/browse/HIVE-11711 > Project: Hive > Issue Type: Sub-task > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 2.0.0 > > Attachments: HIVE-11711.1.patch, HIVE-11711.2.patch, > HIVE-11711.3.patch, HIVE-11711.4.patch, HIVE-11711.5.patch > > > Major development of hbase-metastore is done and it's time to merge the > branch back into master. > Currently hbase-metastore is only invoked when running TestMiniTezCliDriver. > The instruction for setting up hbase-metastore is captured in > https://cwiki.apache.org/confluence/display/Hive/HBaseMetastoreDevelopmentGuide. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11587) Fix memory estimates for mapjoin hashtable
[ https://issues.apache.org/jira/browse/HIVE-11587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732062#comment-14732062 ] Wei Zheng commented on HIVE-11587: -- The above mismatches run clean locally. > Fix memory estimates for mapjoin hashtable > -- > > Key: HIVE-11587 > URL: https://issues.apache.org/jira/browse/HIVE-11587 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.0, 1.2.1 >Reporter: Sergey Shelukhin >Assignee: Wei Zheng > Attachments: HIVE-11587.01.patch, HIVE-11587.02.patch, > HIVE-11587.03.patch, HIVE-11587.04.patch, HIVE-11587.05.patch, > HIVE-11587.06.patch, HIVE-11587.07.patch, HIVE-11587.08.patch > > > Due to the legacy in in-memory mapjoin and conservative planning, the memory > estimation code for mapjoin hashtable is currently not very good. It > allocates the probe erring on the side of more memory, not taking data into > account because unlike the probe, it's free to resize, so it's better for > perf to allocate big probe and hope for the best with regard to future data > size. It is not true for hybrid case. > There's code to cap the initial allocation based on memory available > (memUsage argument), but due to some code rot, the memory estimates from > planning are not even passed to hashtable anymore (there used to be two > config settings, hashjoin size fraction by itself, or hashjoin size fraction > for group by case), so it never caps the memory anymore below 1 Gb. > Initial capacity is estimated from input key count, and in hybrid join cache > can exceed Java memory due to number of segments. > There needs to be a review and fix of all this code. > Suggested improvements: > 1) Make sure "initialCapacity" argument from Hybrid case is correct given the > number of segments. See how it's calculated from keys for regular case; it > needs to be adjusted accordingly for hybrid case if not done already. > 1.5) Note that, knowing the number of rows, the maximum capacity one will > ever need for probe size (in longs) is row count (assuming key per row, i.e. > maximum possible number of keys) divided by load factor, plus some very small > number to round up. That is for flat case. For hybrid case it may be more > complex due to skew, but that is still a good upper bound for the total probe > capacity of all segments. > 2) Rename memUsage to maxProbeSize, or something, make sure it's passed > correctly based on estimates that take into account both probe and data size, > esp. in hybrid case. > 3) Make sure that memory estimation for hybrid case also doesn't come up with > numbers that are too small, like 1-byte hashtable. I am not very familiar > with that code but it has happened in the past. > Other issues we have seen: > 4) Cap single write buffer size to 8-16Mb. The whole point of WBs is that you > should not allocate large array in advance. Even if some estimate passes > 500Mb or 40Mb or whatever, it doesn't make sense to allocate that. > 5) For hybrid, don't pre-allocate WBs - only allocate on write. > 6) Change everywhere rounding up to power of two is used to rounding down, at > least for hybrid case (?) > I wanted to put all of these items in single JIRA so we could keep track of > fixing all of them. > I think there are JIRAs for some of these already, feel free to link them to > this one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11711) Merge hbase-metastore branch to trunk
[
https://issues.apache.org/jira/browse/HIVE-11711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732097#comment-14732097
]
Hive QA commented on HIVE-11711:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12754337/HIVE-11711.5.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9218 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.streaming.TestStreaming.testRemainingTransactions
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5186/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5186/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5186/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12754337 - PreCommit-HIVE-TRUNK-Build
> Merge hbase-metastore branch to trunk
> -
>
> Key: HIVE-11711
> URL: https://issues.apache.org/jira/browse/HIVE-11711
> Project: Hive
> Issue Type: Sub-task
> Components: HBase Metastore
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Fix For: 2.0.0
>
> Attachments: HIVE-11711.1.patch, HIVE-11711.2.patch,
> HIVE-11711.3.patch, HIVE-11711.4.patch, HIVE-11711.5.patch
>
>
> Major development of hbase-metastore is done and it's time to merge the
> branch back into master.
> Currently hbase-metastore is only invoked when running TestMiniTezCliDriver.
> The instruction for setting up hbase-metastore is captured in
> https://cwiki.apache.org/confluence/display/Hive/HBaseMetastoreDevelopmentGuide.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11746) Connect command should not to be allowed from user[beeline-cli branch]
[ https://issues.apache.org/jira/browse/HIVE-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-11746: Attachment: HIVE-11746.1-beeline-cli.patch This patch is based on HIVE-11640. > Connect command should not to be allowed from user[beeline-cli branch] > -- > > Key: HIVE-11746 > URL: https://issues.apache.org/jira/browse/HIVE-11746 > Project: Hive > Issue Type: Sub-task > Components: Beeline >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-11746.1-beeline-cli.patch > > > For new cli, user should not be allowed to connect a server or database. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11593) Add aes_encrypt and aes_decrypt UDFs
[ https://issues.apache.org/jira/browse/HIVE-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732169#comment-14732169 ] Alexander Pivovarov commented on HIVE-11593: Added functions descriptions to the wiki https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-Misc.Functions > Add aes_encrypt and aes_decrypt UDFs > > > Key: HIVE-11593 > URL: https://issues.apache.org/jira/browse/HIVE-11593 > Project: Hive > Issue Type: New Feature > Components: UDF >Reporter: Alexander Pivovarov >Assignee: Alexander Pivovarov > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11593.1.patch, HIVE-11593.2.patch > > > AES (Advanced Encryption Standard) algorithm. > Oracle JRE supports AES-128 out of the box > AES-192 and AES-256 are supported if Cryptography Extension (JCE) Unlimited > Strength Jurisdiction Policy Files are installed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11640) Shell command doesn't work for new CLI[Beeline-cli branch]
[ https://issues.apache.org/jira/browse/HIVE-11640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732156#comment-14732156 ] Ferdinand Xu commented on HIVE-11640: - Thanks [~xuefuz] for your review. Please see my inline comments on the RB. > Shell command doesn't work for new CLI[Beeline-cli branch] > -- > > Key: HIVE-11640 > URL: https://issues.apache.org/jira/browse/HIVE-11640 > Project: Hive > Issue Type: Sub-task > Components: CLI >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-11640.1-beeline-cli.patch, > HIVE-11640.2-beeline-cli.patch, HIVE-11640.3-beeline-cli.patch, > HIVE-11640.4-beeline-cli.patch, HIVE-11640.5-beeline-cli.patch, > HIVE-11640.7-beeline-cli.patch > > > The shell command doesn't work for the new CLI and "Error: Method not > supported (state=,code=0)" was thrown during the execution for option f and e. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
