[
https://issues.apache.org/jira/browse/HIVE-24493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongle Zhang updated HIVE-24493:
--------------------------------
Description:
We tried to upgrade a hive single node from 2.1.1 to 2.3.7 and use derby as
the database. After upgrade, Hive log keeps printing the following Error
message repeatedly:
{code:java}
// code placeholder
2020-12-05T02:47:26,573 ERROR [main] metastore.RetryingHMSHandler: HMSHandler
Fatal error: MetaException(message:Hive Schema version 2.3.0 does not match
metastore's schema version 2.1.0 Metastore is not upgraded or corrupt)
at
org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7825)
at
org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
at com.sun.proxy.$Proxy34.verifySchema(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:595)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:588)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:655)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:431)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:79)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6902)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1707)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3600)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3652)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3632)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3894)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248)
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:388)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:332)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:312)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
at
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:917)
at
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:881)
at
org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1687)
at
org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:130)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:114)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:142)
at
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607)
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100)
at
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153){code}
We did use schematool to upgrade Hive. The log also shows that upgrade
metastore is performed. Here are the steps to reproduce it:
1. Start up a standalone hive node of version 2.1.1 using embedded derby, and
use schematool to initialize the schema.
2. Use the following shell script to generate 20k data.
{code:java}
rm -rf ./data.txt
touch data.txt
for((i=0;i<20000;i++))
do
str=',name';
name=${i}${str}${i}
echo $name>> data.txt
done
{code}
3. Create a table and load data by using:
{code:java}
create table testTable(id int,name string) row format delimited fields
terminated by ',';
LOAD DATA LOCAL INPATH '/data.txt' OVERWRITE INTO TABLE testTable;
{code}
4. We stopped the nodes, and started a node running hive compiled from 2.3.7,
while keeping the node of hadoop running.
5. Run the following command to upgrade schema.
{code:java}
schematool -dbType derby -upgradeSchema
{code}
Log shows that metastore is being upgraded:
{code:java}
LF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:
jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting upgrade metastore schema from version 2.1.0 to 2.3.0
Upgrade script upgrade-2.1.0-to-2.2.0.derby.sql
Connecting to jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
Connected to: Apache Derby (version 10.10.2.0 - (1582446))
Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
.
.
.
beeline> Completed upgrade-2.1.0-to-2.2.0.derby.sql
Upgrade script upgrade-2.2.0-to-2.3.0.derby.sql
Connecting to jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
Connected to: Apache Derby (version 10.10.2.0 - (1582446))
Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
.
.
.
beeline> Completed upgrade-2.2.0-to-2.3.0.derby.sql
schemaTool completed
{code}
6. However, after upgrade, the log still keeps printing the previous Error
message with exception - "Hive Schema version 2.3.0 does not match metastore's
schema version 2.1.0”.
was:
We tried to upgrade a hive single node from 2.1.1 to 2.3.7 and use derby as
the database. After upgrade, Hive log keeps printing the following Error
message repeatedly:
{code:java}
// code placeholder
2020-12-05T02:47:26,573 ERROR [main] metastore.RetryingHMSHandler: HMSHandler
Fatal error: MetaException(message:Hive Schema version 2.3.0 does not match
metastore's schema version 2.1.0 Metastore is not upgraded or corrupt)
at
org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7825)
at
org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
at com.sun.proxy.$Proxy34.verifySchema(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:595)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:588)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:655)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:431)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:79)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6902)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1707)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3600)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3652)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3632)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3894)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248)
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:388)
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:332)
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:312)
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
at
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:917)
at
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:881)
at
org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1687)
at
org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:130)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:114)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:142)
at
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607)
at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100)
at
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153){code}
We did use schematool to upgrade Hive. The log also shows that upgrade
metastore is performed. Here are the steps to reproduce it:
1. Start up a standalone hive node of version 2.1.1 using embedded derby, and
use schematool to initialize the schema.
2. Use the following shell script to generate 20k data.
{code:java}
rm -rf ./data.txt
touch data.txt
for((i=0;i<20000;i++))
do
str=',name';
name=${i}${str}${i}
echo $name>> data.txt
done
{code}
3. Create a table and load data by using:
{code:java}
create table testTable(id int,name string) row format delimited fields
terminated by ',';
LOAD DATA LOCAL INPATH '/data.txt' OVERWRITE INTO TABLE testTable;
{code}
4. We stopped the nodes, and started a node running hive compiled from 2.3.7,
while keeping the node of hadoop running.
5. Run the following command to upgrade schema.
{code:java}
schematool -dbType derby -upgradeSchema
{code}
Log shows that metastore is being upgraded:
{code:java}
LF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:
jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting upgrade metastore schema from version 2.1.0 to 2.3.0
Upgrade script upgrade-2.1.0-to-2.2.0.derby.sql
Connecting to jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
Connected to: Apache Derby (version 10.10.2.0 - (1582446))
Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
.
.
.
beeline> Completed upgrade-2.1.0-to-2.2.0.derby.sql
Upgrade script upgrade-2.2.0-to-2.3.0.derby.sql
Connecting to jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
Connected to: Apache Derby (version 10.10.2.0 - (1582446))
Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
.
.
.
beeline> Completed upgrade-2.2.0-to-2.3.0.derby.sql
schemaTool completed
{code}
6. However, after upgrade, the log still keeps printing the previous Error
message with exception - "Hive Schema version 2.3.0 does not match metastore's
schema version 2.1.0”.
> after upgrading Hive from 2.1.1 to 2.3.7, log keeps printing "Hive Schema
> version 2.3.0 does not match metastore's schema version 2.1.0"
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-24493
> URL: https://issues.apache.org/jira/browse/HIVE-24493
> Project: Hive
> Issue Type: Bug
> Environment: {code:java}
> Linux version 4.15.0-112-generic (buildd@lcy01-amd64-027) (gcc version 7.5.0
> (Ubuntu 7.5.0-3ubuntu1~18.04)) #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
> {code}
> {code:java}
> openjdk version "1.8.0_252"
> OpenJDK Runtime Environment (build 1.8.0_252-b09)
> OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode){code}
> Reporter: Yongle Zhang
> Priority: Major
>
> We tried to upgrade a hive single node from 2.1.1 to 2.3.7 and use derby as
> the database. After upgrade, Hive log keeps printing the following Error
> message repeatedly:
>
> {code:java}
> // code placeholder
> 2020-12-05T02:47:26,573 ERROR [main] metastore.RetryingHMSHandler: HMSHandler
> Fatal error: MetaException(message:Hive Schema version 2.3.0 does not match
> metastore's schema version 2.1.0 Metastore is not upgraded or corrupt)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7825)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7788)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> at com.sun.proxy.$Proxy34.verifySchema(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:595)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:588)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:655)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:431)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:79)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6902)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:164)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1707)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3600)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3652)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3632)
> at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3894)
> at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
> at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:388)
> at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:332)
> at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:312)
> at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288)
> at
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:917)
> at
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:881)
> at
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1687)
> at
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:130)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:114)
> at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:142)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607)
> at
> org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855)
> at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153){code}
>
>
> We did use schematool to upgrade Hive. The log also shows that upgrade
> metastore is performed. Here are the steps to reproduce it:
>
> 1. Start up a standalone hive node of version 2.1.1 using embedded derby, and
> use schematool to initialize the schema.
> 2. Use the following shell script to generate 20k data.
>
> {code:java}
> rm -rf ./data.txt
> touch data.txt
> for((i=0;i<20000;i++))
> do
> str=',name';
> name=${i}${str}${i}
> echo $name>> data.txt
> done
> {code}
> 3. Create a table and load data by using:
> {code:java}
> create table testTable(id int,name string) row format delimited fields
> terminated by ',';
> LOAD DATA LOCAL INPATH '/data.txt' OVERWRITE INTO TABLE testTable;
> {code}
> 4. We stopped the nodes, and started a node running hive compiled from 2.3.7,
> while keeping the node of hadoop running.
> 5. Run the following command to upgrade schema.
>
> {code:java}
> schematool -dbType derby -upgradeSchema
> {code}
> Log shows that metastore is being upgraded:
> {code:java}
> LF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL:
> jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
> Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
> Metastore connection User: APP
> Starting upgrade metastore schema from version 2.1.0 to 2.3.0
> Upgrade script upgrade-2.1.0-to-2.2.0.derby.sql
> Connecting to jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
> Connected to: Apache Derby (version 10.10.2.0 - (1582446))
> Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
> .
> .
> .
> beeline> Completed upgrade-2.1.0-to-2.2.0.derby.sql
> Upgrade script upgrade-2.2.0-to-2.3.0.derby.sql
> Connecting to jdbc:derby:;databaseName=/hive/data/metastore_db;create=true
> Connected to: Apache Derby (version 10.10.2.0 - (1582446))
> Driver: Apache Derby Embedded JDBC Driver (version 10.10.2.0 - (1582446))
> .
> .
> .
> beeline> Completed upgrade-2.2.0-to-2.3.0.derby.sql
> schemaTool completed
> {code}
> 6. However, after upgrade, the log still keeps printing the previous Error
> message with exception - "Hive Schema version 2.3.0 does not match
> metastore's schema version 2.1.0”.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
