[ https://issues.apache.org/jira/browse/HIVE-24279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xianyin Xin updated HIVE-24279: ------------------------------- Description: I got some exception when i configured transaction, {code} 2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an exception in the main loop of compactor initiator, exiting MetaException(message:Unable to connect to transaction database java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not exist. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown Source) at com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117) at com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java) at org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646) at org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208) at org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74) Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist. {code} After some debugging, i found the conf that passed to the {{CompactorThread}} was: {code} public void setConf(Configuration configuration) { // TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to call other methods with // it. This should be changed to Configuration once everything that this calls that requires // HiveConf is moved to the standalone metastore. conf = (configuration instanceof HiveConf) ? (HiveConf)configuration : new HiveConf(configuration, HiveConf.class); } {code} However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit all the configs come from {{configuration}}, actually the {{configuration}} will be overwrite by the default values those are not nulls: {code} private void initialize(Class<?> cls) { hiveJar = (new JobConf(cls)).getJar(); // preserve the original configuration origProp = getAllProperties(); // Overlay the ConfVars. Note that this ignores ConfVars with null values addResource(getConfVarInputStream()); // Overlay hive-site.xml if it exists if (hiveSiteURL != null) { addResource(hiveSiteURL); } // if embedded metastore is to be used as per config so far // then this is considered like the metastore server case String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS); // This is hackery, but having hive-common depend on standalone-metastore is really bad // because it will pull all of the metastore code into every module. We need to check that // we aren't using the standalone metastore. If we are, we should treat it the same as a // remote metastore situation. if (msUri == null || msUri.isEmpty()) { msUri = this.get("metastore.thrift.uris"); } LOG.debug("Found metastore URI of " + msUri); if(HiveConfUtil.isEmbeddedMetaStore(msUri)){ setLoadMetastoreConfig(true); } // load hivemetastore-site.xml if this is metastore and file exists if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) { addResource(hivemetastoreSiteUrl); } {code} That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new {{hiveConf}} but the configs which have null default values will be overwrite by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except that it is an embedded metastore. If hive-site doesn't have the db connection info, {{CompactorThread}} would connect to the default derby. was: I got some exception when i configured transaction, {code} 2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an exception in the main loop of compactor initiator, exiting MetaException(message:Unable to connect to transaction database java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not exist. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown Source) at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown Source) at com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117) at com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java) at org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646) at org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208) at org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74) Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist. {code} After some debugging, i found the conf that passed to the {{CompactorThread}} was: {code} public void setConf(Configuration configuration) { // TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to call other methods with // it. This should be changed to Configuration once everything that this calls that requires // HiveConf is moved to the standalone metastore. conf = (configuration instanceof HiveConf) ? (HiveConf)configuration : new HiveConf(configuration, HiveConf.class); } {code} However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit all the configs come from {{configuration}}, actually the {{configuration}} will be overwrite by the default values those are not nulls: {code} private void initialize(Class<?> cls) { hiveJar = (new JobConf(cls)).getJar(); // preserve the original configuration origProp = getAllProperties(); // Overlay the ConfVars. Note that this ignores ConfVars with null values addResource(getConfVarInputStream()); // Overlay hive-site.xml if it exists if (hiveSiteURL != null) { addResource(hiveSiteURL); } // if embedded metastore is to be used as per config so far // then this is considered like the metastore server case String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS); // This is hackery, but having hive-common depend on standalone-metastore is really bad // because it will pull all of the metastore code into every module. We need to check that // we aren't using the standalone metastore. If we are, we should treat it the same as a // remote metastore situation. if (msUri == null || msUri.isEmpty()) { msUri = this.get("metastore.thrift.uris"); } LOG.debug("Found metastore URI of " + msUri); if(HiveConfUtil.isEmbeddedMetaStore(msUri)){ setLoadMetastoreConfig(true); } // load hivemetastore-site.xml if this is metastore and file exists if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) { addResource(hivemetastoreSiteUrl); } {code} That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new {{hiveConf}} but the configs which have null default values will be overwrite by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except that it is an embedded metastore. If hive-site doesn't have the db connection info, {{CompactorThread}} would connect to the default derby which causes the failure. > Hive CompactorThread fails to connect to metastore if the connectionURL was > only configs in metastore-site > ---------------------------------------------------------------------------------------------------------- > > Key: HIVE-24279 > URL: https://issues.apache.org/jira/browse/HIVE-24279 > Project: Hive > Issue Type: Improvement > Components: Metastore > Affects Versions: 3.1.2 > Reporter: Xianyin Xin > Priority: Major > > I got some exception when i configured transaction, > {code} > 2020-10-15T11:05:41,356 ERROR [Thread-7] compactor.Initiator: Caught an > exception in the main loop of compactor initiator, exiting > MetaException(message:Unable to connect to transaction database > java.sql.SQLSyntaxErrorException: Table/View 'COMPACTION_QUEUE' does not > exist. > at > org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) > at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown > Source) > at > org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown > Source) > at > org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown > Source) > at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown > Source) > at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown > Source) > at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source) > at > org.apache.derby.impl.jdbc.EmbedStatement.executeLargeUpdate(Unknown Source) > at org.apache.derby.impl.jdbc.EmbedStatement.executeUpdate(Unknown > Source) > at > com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:117) > at > com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java) > at > org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.revokeFromLocalWorkers(CompactionTxnHandler.java:646) > at > org.apache.hadoop.hive.ql.txn.compactor.Initiator.recoverFailedCompactions(Initiator.java:208) > at > org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:74) > Caused by: ERROR 42X05: Table/View 'COMPACTION_QUEUE' does not exist. > {code} > After some debugging, i found the conf that passed to the {{CompactorThread}} > was: > {code} > public void setConf(Configuration configuration) { > // TODO MS-SPLIT for now, keep a copy of HiveConf around as we need to > call other methods with > // it. This should be changed to Configuration once everything that this > calls that requires > // HiveConf is moved to the standalone metastore. > conf = (configuration instanceof HiveConf) ? (HiveConf)configuration : > new HiveConf(configuration, HiveConf.class); > } > {code} > However, {{new HiveConf(configuration, HiveConf.class)}} would not inherit > all the configs come from {{configuration}}, actually the {{configuration}} > will be overwrite by the default values those are not nulls: > {code} > private void initialize(Class<?> cls) { > hiveJar = (new JobConf(cls)).getJar(); > // preserve the original configuration > origProp = getAllProperties(); > // Overlay the ConfVars. Note that this ignores ConfVars with null values > addResource(getConfVarInputStream()); > // Overlay hive-site.xml if it exists > if (hiveSiteURL != null) { > addResource(hiveSiteURL); > } > // if embedded metastore is to be used as per config so far > // then this is considered like the metastore server case > String msUri = this.getVar(HiveConf.ConfVars.METASTOREURIS); > // This is hackery, but having hive-common depend on standalone-metastore > is really bad > // because it will pull all of the metastore code into every module. We > need to check that > // we aren't using the standalone metastore. If we are, we should treat > it the same as a > // remote metastore situation. > if (msUri == null || msUri.isEmpty()) { > msUri = this.get("metastore.thrift.uris"); > } > LOG.debug("Found metastore URI of " + msUri); > if(HiveConfUtil.isEmbeddedMetaStore(msUri)){ > setLoadMetastoreConfig(true); > } > // load hivemetastore-site.xml if this is metastore and file exists > if (isLoadMetastoreConfig() && hivemetastoreSiteUrl != null) { > addResource(hivemetastoreSiteUrl); > } > {code} > That is, {{new HiveConf(configuration, HiveConf.class)}} is merely a new > {{hiveConf}} but the configs which have null default values will be overwrite > by {{configuration}}. A {{hiveConf}} would not load hivemetastore-site except > that it is an embedded metastore. If hive-site doesn't have the db connection > info, {{CompactorThread}} would connect to the default derby. -- This message was sent by Atlassian Jira (v8.3.4#803005)