[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703714#comment-16703714 ] Naveen Gangam commented on HIVE-20740: -- Looks good to me. +1 > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch, > HIVE-20740.14.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700701#comment-16700701 ] Andrew Sherman commented on HIVE-20740: --- +1 LGTM > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch, > HIVE-20740.14.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700013#comment-16700013 ] Vihang Karajgaonkar commented on HIVE-20740: Finally a green run. [~asherman] Updated the RB with the latest patch. There is no real code change since you last reviewed on the RB except for the fact that I rebased and was juggling through many unrelated failures on precommit. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch, > HIVE-20740.14.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699886#comment-16699886 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12949566/HIVE-20740.14.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15543 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15066/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15066/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15066/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12949566 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch, > HIVE-20740.14.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699869#comment-16699869 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 4s{color} | {color:blue} standalone-metastore/metastore-server in master has 185 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 46s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608 unchanged - 0 fixed = 609 total (was 608) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 15s{color} | {color:red} standalone-metastore/metastore-server generated 1 new + 183 unchanged - 2 fixed = 184 total (was 185) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 32m 47s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration) does not release lock on all paths At PersistenceManagerProvider.java:on all paths At PersistenceManagerProvider.java:[line 152] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15066/dev-support/hive-personality.sh | | git revision | master / 56625f3 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15066/yetus/diff-checkstyle-itests_hive-unit.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-15066/yetus/new-findbugs-standalone-metastore_metastore-server.html | | modules | C: standalone-metastore/metastore-server ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15066/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699624#comment-16699624 ] Vihang Karajgaonkar commented on HIVE-20740: cbo_limit.q failure is unrelated to this patch. The failure message is weird, could be extra space somewhere in the q.out [INFO] Running org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver [ERROR] Tests run: 30, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 405.018 s <<< FAILURE! - in org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver [ERROR] testCliDriver[cbo_limit](org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver) Time elapsed: 9.876 s <<< FAILURE! java.lang.AssertionError: Client Execution succeeded but contained differences (error code = 1) after executing cbo_limit.q 11c11 < 14 2 --- > 14 2 > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699601#comment-16699601 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12949531/HIVE-20740.13.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15540 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit] (batchId=182) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15061/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15061/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15061/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12949531 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699576#comment-16699576 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 35s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 6s{color} | {color:blue} standalone-metastore/metastore-server in master has 185 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 42s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 6s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608 unchanged - 0 fixed = 609 total (was 608) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 17s{color} | {color:red} standalone-metastore/metastore-server generated 1 new + 183 unchanged - 2 fixed = 184 total (was 185) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 33m 1s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration) does not release lock on all paths At PersistenceManagerProvider.java:on all paths At PersistenceManagerProvider.java:[line 152] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15061/dev-support/hive-personality.sh | | git revision | master / 0fee288 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15061/yetus/diff-checkstyle-itests_hive-unit.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-15061/yetus/new-findbugs-standalone-metastore_metastore-server.html | | modules | C: standalone-metastore/metastore-server ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15061/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699559#comment-16699559 ] Vihang Karajgaonkar commented on HIVE-20740: The test failures occur only on the precommit job. The logs do not have enough information to debug these failures. I will try to observer on the ptest server itself while the batch containing {{TestObjectStore}} test is running. Batch 230 has {{TestObjectStore}} {noformat} 2018-11-26 20:25:19,796 DEBUG [TestExecutor] ExecutionPhase.execute:98 PBatch: UnitTestBatch [name=230_UTBatch_standalone-metastore__metastore-server_20_tests, id=230, moduleName=standalone-metastore/metastore-server, batchSize=20, isParallel=true, testList=[TestMetaStoreConnectionUrlHook, TestSchemaToolForMetastore, TestMetastoreSchemaTool, TestMetaStoreSchemaFactory, TestRetryingHMSHandler, TestAdminUser, TestJSONMessageDeserializer, TestCatalogNonDefaultSvr, TestObjectStoreInitRetry, TestHdfsUtils, TestMetaStoreServerUtils, TestHiveMetaStoreSchemaMethods, TestOldSchema, TestCachedStore, TestCatalogCaching, TestDeadline, TestMetaStoreListenersError, TestMetaStoreEventListenerOnlyOnCommit, TestMetaStoreSchemaInfo, TestMarkPartition]] {noformat} > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684538#comment-16684538 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12947891/HIVE-20740.12.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15312 tests executed *Failed tests:* {noformat} TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestFMSketchSerialization - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestHLLNoBias - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestHLLSerialization - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestHyperLogLog - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestHyperLogLogDense - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestHyperLogLogMerge - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestHyperLogLogSparse - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestObjectStore - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestSparseEncodeHash - did not produce a TEST-*.xml file (likely timed out) (batchId=231) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14896/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14896/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14896/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12947891 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch, > HIVE-20740.11.patch, HIVE-20740.12.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684487#comment-16684487 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 42s{color} | {color:blue} ql in master has 2316 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 4s{color} | {color:blue} standalone-metastore/metastore-server in master has 185 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 33s{color} | {color:red} hive-unit in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 31s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608 unchanged - 0 fixed = 609 total (was 608) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 10s{color} | {color:red} standalone-metastore/metastore-server generated 1 new + 183 unchanged - 2 fixed = 184 total (was 185) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 54s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration) does not release lock on all paths At PersistenceManagerProvider.java:on all paths At PersistenceManagerProvider.java:[line 152] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14896/dev-support/hive-personality.sh | | git revision | master / bc39c49 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus/patch-mvninstall-itests_hive-unit.txt | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus/diff-checkstyle-itests_hive-unit.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus/new-findbugs-standalone-metastore_metastore-server.html | | modules | C: itests/hive-unit ql standalone-metastore/metastore-server U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14896/yetus.txt | | Powered by | A
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682159#comment-16682159 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12947502/HIVE-20740.11.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14847/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14847/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14847/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-11-10 02:42:25.172 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-14847/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-11-10 02:42:25.175 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 7ae4a2c HIVE-20833: package.jdo needs to be updated to conform with HIVE-20221 changes (Vineet Garg, reviewed by Jason Dere) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 7ae4a2c HIVE-20833: package.jdo needs to be updated to conform with HIVE-20221 changes (Vineet Garg, reviewed by Jason Dere) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-11-10 02:42:25.935 + rm -rf ../yetus_PreCommit-HIVE-Build-14847 + mkdir ../yetus_PreCommit-HIVE-Build-14847 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-14847 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14847/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java: does not exist in index error: a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java: does not exist in index error: a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java: does not exist in index error: a/standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java: does not exist in index error: patch failed: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java:86 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java' cleanly. Going to apply patch with: git apply -p1 error: patch failed: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java:86 Falling back to three-way merge... Applied patch to 'standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java' cleanly. + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc7010054971666294437.exe, --version] protoc-jar: executing: [/tmp/protoc7010054971666294437.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-s
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680304#comment-16680304 ] Vihang Karajgaonkar commented on HIVE-20740: The error logs say this :( {noformat} [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hive.cli.TestMiniDruidCliDriver [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 47.068s [INFO] Finished at: Wed Nov 07 17:57:04 UTC 2018 [INFO] Final Memory: 61M/510M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project hive-it-qfile: There are test failures. [ERROR] [ERROR] Please refer to /home/hiveptest/35.232.67.223-hiveptest-0/apache-github-source-source/itests/qtest/target/surefire-reports for the individual test results. [ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream. [ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /home/hiveptest/35.232.67.223-hiveptest-0/apache-github-source-source/itests/qtest && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx2048m -jar /home/hiveptest/35.232.67.223-hiveptest-0/apache-github-source-source/itests/qtest/target/surefire/surefirebooter2827692084749430164.jar /home/hiveptest/35.232.67.223-hiveptest-0/apache-github-source-source/itests/qtest/target/surefire 2018-11-07T17-56-50_252-jvmRun1 surefire203521922645361632tmp surefire_04497948723045853634tmp [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was /bin/sh -c cd /home/hiveptest/35.232.67.223-hiveptest-0/apache-github-source-source/itests/qtest && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx2048m -jar /home/hiveptest/35.232.67.223-hiveptest-0/apache-github-source-source/itests/qtest/target/surefire/surefirebooter2827692084749430164.jar /home/hiveptest/35.232.67.223-hiveptest-0/apache-github-source-source/itests/qtest/target/surefire 2018-11-07T17-56-50_252-jvmRun1 surefire203521922645361632tmp surefire_04497948723045853634tmp [ERROR] Error occurred in starting fork, check output in log [ERROR] Process Exit Code: 1 {noformat} > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionS
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16678616#comment-16678616 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12947160/HIVE-20740.10.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15525 tests executed *Failed tests:* {noformat} TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=196) [druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q] {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14793/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14793/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14793/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12947160 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This me
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16678580#comment-16678580 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 39s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 4s{color} | {color:blue} standalone-metastore/metastore-server in master has 185 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 35s{color} | {color:red} hive-unit in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 31s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608 unchanged - 0 fixed = 609 total (was 608) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 9s{color} | {color:red} standalone-metastore/metastore-server generated 1 new + 183 unchanged - 2 fixed = 184 total (was 185) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 32s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration) does not release lock on all paths At PersistenceManagerProvider.java:on all paths At PersistenceManagerProvider.java:[line 152] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14793/dev-support/hive-personality.sh | | git revision | master / 6d713b6 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14793/yetus/patch-mvninstall-itests_hive-unit.txt | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14793/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14793/yetus/diff-checkstyle-itests_hive-unit.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14793/yetus/new-findbugs-standalone-metastore_metastore-server.html | | modules | C: itests/hive-unit ql standalone-metastore/metastore-server U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14793/yetus.txt | | Powered by | A
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677490#comment-16677490 ] Vihang Karajgaonkar commented on HIVE-20740: These tests are working for me locally and unfortunately, the logs are gone now. Rebased the patch and reattached. I will keep an eye on the precommit job results this time. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674205#comment-16674205 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12946744/HIVE-20740.09.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15524 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testAlterPartitionSetLocationNonDefaultNameNode (batchId=244) org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateDatabaseWithTableNonDefaultNameNode (batchId=244) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14730/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14730/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14730/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12946744 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch, HIVE-20740.09.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very s
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16674188#comment-16674188 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 19s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 45s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 4s{color} | {color:blue} standalone-metastore/metastore-server in master has 185 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 34s{color} | {color:red} hive-unit in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 31s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608 unchanged - 0 fixed = 609 total (was 608) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 8s{color} | {color:red} standalone-metastore/metastore-server generated 2 new + 183 unchanged - 2 fixed = 185 total (was 185) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | Dead store to thislimit in org.apache.hadoop.hive.metastore.ObjectStore.getCallStack() At ObjectStore.java:org.apache.hadoop.hive.metastore.ObjectStore.getCallStack() At ObjectStore.java:[line 9522] | | | org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration) does not release lock on all paths At PersistenceManagerProvider.java:on all paths At PersistenceManagerProvider.java:[line 151] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14730/dev-support/hive-personality.sh | | git revision | master / ae1eb15 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14730/yetus/patch-mvninstall-itests_hive-unit.txt | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14730/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14730/yetus/diff-checkstyle-itests_hive-unit.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14730/yetus/new-findbugs-standalone-metasto
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673729#comment-16673729 ] Vihang Karajgaonkar commented on HIVE-20740: Test is working for me locally. I will create a JIRA to disable/fix this test. [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.367 s - in org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673025#comment-16673025 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12946583/HIVE-20740.08.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15521 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit] (batchId=182) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14702/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14702/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14702/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12946583 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673002#comment-16673002 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 39s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 3s{color} | {color:blue} standalone-metastore/metastore-server in master has 185 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 35s{color} | {color:red} hive-unit in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 29s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608 unchanged - 0 fixed = 609 total (was 608) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 13s{color} | {color:red} standalone-metastore/metastore-server generated 2 new + 183 unchanged - 2 fixed = 185 total (was 185) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 31m 36s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | Dead store to thislimit in org.apache.hadoop.hive.metastore.ObjectStore.getCallStack() At ObjectStore.java:org.apache.hadoop.hive.metastore.ObjectStore.getCallStack() At ObjectStore.java:[line 9522] | | | org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration) does not release lock on all paths At PersistenceManagerProvider.java:on all paths At PersistenceManagerProvider.java:[line 151] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14702/dev-support/hive-personality.sh | | git revision | master / f3fab45 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14702/yetus/patch-mvninstall-itests_hive-unit.txt | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14702/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14702/yetus/diff-checkstyle-itests_hive-unit.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14702/yetus/new-findbugs-standalone-metasto
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671953#comment-16671953 ] Vihang Karajgaonkar commented on HIVE-20740: Reattached the last patch since precommit queue was drained due to infrastructure issues. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch, > HIVE-20740.08.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667061#comment-16667061 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12946003/HIVE-20740.06.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15509 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.TestObjectStoreInitRetry.testObjStoreRetry (batchId=229) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testDataTypes (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testEscapedStrings (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testLlapInputFormatEndToEnd (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testNonAsciiStrings (batchId=257) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14679/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14679/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14679/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12946003 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unloc
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667016#comment-16667016 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 50s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 1s{color} | {color:blue} standalone-metastore/metastore-server in master has 181 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 32s{color} | {color:red} hive-unit in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 30s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 608 unchanged - 0 fixed = 609 total (was 608) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 13s{color} | {color:red} standalone-metastore/metastore-server generated 2 new + 179 unchanged - 2 fixed = 181 total (was 181) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 31m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:standalone-metastore/metastore-server | | | Dead store to thislimit in org.apache.hadoop.hive.metastore.ObjectStore.getCallStack() At ObjectStore.java:org.apache.hadoop.hive.metastore.ObjectStore.getCallStack() At ObjectStore.java:[line 9588] | | | org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(Configuration) does not release lock on all paths At PersistenceManagerProvider.java:on all paths At PersistenceManagerProvider.java:[line 118] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14679/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14679/yetus/patch-mvninstall-itests_hive-unit.txt | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14679/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14679/yetus/diff-checkstyle-itests_hive-unit.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14679/yetus/new-findbugs-standalone-metasto
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1680#comment-1680 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945944/HIVE-20740.05.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15509 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar (batchId=240) org.apache.hadoop.hive.metastore.TestObjectStoreInitRetry.testObjStoreRetry (batchId=229) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching (batchId=260) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14675/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14675/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14675/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945944 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very sm
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1647#comment-1647 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 14s{color} | {color:red} /data/hiveptest/logs/PreCommit-HIVE-Build-14675/patches/PreCommit-HIVE-Build-14675.patch does not apply to master. Rebase required? Wrong Branch? See http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14675/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1237#comment-1237 ] Vihang Karajgaonkar commented on HIVE-20740: Spent some more time on this. Based on what I understand, there is a race in the existing code which could lead to a NPE. Eg. Thread A calls setConf method with a new datasource properties. Thread A acquires {{proplock}} and sets {{pmf = null}} Meanwhile Thread B is in {{getPMF}} method which even though is static synchronized doesn't know if pmf is being updated by some other thread in {{setConf}} object. This could lead to a NPE on Thread B. Or some error on Thread A when it tries to get {{PersistenceManager}} from unitialized pmf. The good news this is very unlikely to happen. When datasource properties are changed, it needs a restart of HMS service which means there are no other threads requesting {{pmf}} until HMS up and running. But still having {{pmf}} and {{props}} objects as private members which need to be synchronized when updating can be handled more cleaning if we move them to a separate wrapper class which deals with these details. Latest patch refactors some of this code and hopefully makes is cleaner than the current implementation. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659247#comment-16659247 ] Vihang Karajgaonkar commented on HIVE-20740: Okay, so looks like there is more to this than just protecting pmf for read/write access because when you create a {{PersistenceManager}} using {{PersistenceManagerFactory}} it modifies its internal state (by taking multiple locks) internally. I am not sure how it is not showing a race even with the existing code without the patch. Will investigate more. Also, will do a simple perf test. May be all we really need to do is to add a {{static synchronized}} around {{getPMF}} and a {{synchronized}} block using {{ObjectStore.class}} in the {{setConf}} method. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657839#comment-16657839 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12944804/HIVE-20740.04.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15105 tests executed *Failed tests:* {noformat} TestSSL - did not produce a TEST-*.xml file (likely timed out) (batchId=255) org.apache.hadoop.hive.metastore.TestObjectStore.testConcurrentPMFInitialize (batchId=231) org.apache.hadoop.hive.metastore.TestObjectStoreInitRetry.testObjStoreRetry (batchId=229) org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir (batchId=244) org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths (batchId=244) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14589/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14589/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14589/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12944804 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to inst
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657829#comment-16657829 ] Hive QA commented on HIVE-20740: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 15s{color} | {color:blue} standalone-metastore/metastore-server in master has 182 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 22s{color} | {color:green} standalone-metastore/metastore-server generated 0 new + 180 unchanged - 2 fixed = 180 total (was 182) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 47s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14589/dev-support/hive-personality.sh | | git revision | master / 7ed8bf4 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: standalone-metastore/metastore-server U: standalone-metastore/metastore-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14589/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > }
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654441#comment-16654441 ] Vihang Karajgaonkar commented on HIVE-20740: Test failures may be related. Taking a look. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653606#comment-16653606 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12944230/HIVE-20740.02.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14520/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14520/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14520/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12944230/HIVE-20740.02.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12944230 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653146#comment-16653146 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12944230/HIVE-20740.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 15097 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2] (batchId=85) org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver[url_hook] (batchId=262) org.apache.hadoop.hive.metastore.TestObjectStoreInitRetry.testObjStoreRetry (batchId=229) org.apache.hadoop.hive.metastore.TestTransactionalValidationListener.testCreateAsAcid[Remote] (batchId=240) org.apache.hadoop.hive.metastore.client.TestAddPartitions.noSuchCatalog[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestAlterPartitions.deprecatedCalls[Remote] (batchId=223) org.apache.hadoop.hive.metastore.client.TestAppendPartitions.otherCatalog[Remote] (batchId=223) org.apache.hadoop.hive.metastore.client.TestDatabases.databasesInCatalogs[Remote] (batchId=223) org.apache.hadoop.hive.metastore.client.TestDatabases.testDefaultDatabaseData[Embedded] (batchId=223) org.apache.hadoop.hive.metastore.client.TestDropPartitions.otherCatalog[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestFunctions.otherCatalog[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestGetPartitions.otherCatalog[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestGetTableMeta.tablesInDifferentCatalog[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestListPartitions.otherCatalog[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestRuntimeStats.testRuntimeStatHandling[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.moveTablesBetweenCatalogsOnAlter[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.tablesInOtherCatalogs[Remote] (batchId=221) org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableChangingDatabase[Embedded] (batchId=221) org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableRename[Embedded] (batchId=221) org.apache.hadoop.hive.metastore.client.TestTablesGetExists.otherCatalog[Remote] (batchId=223) org.apache.hadoop.hive.metastore.client.TestTablesList.otherCatalogs[Remote] (batchId=221) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=243) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalRepeatEventOnExistingObject (batchId=243) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTaskCreationOptimization (batchId=243) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testViewsReplication (batchId=243) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=205) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=205) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14516/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14516/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14516/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 27 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12944230 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSu
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653078#comment-16653078 ] Hive QA commented on HIVE-20740: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 55s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 7s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 59s{color} | {color:blue} standalone-metastore/metastore-server in master has 182 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s{color} | {color:green} standalone-metastore/metastore-server generated 0 new + 180 unchanged - 2 fixed = 180 total (was 182) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 42s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14516/dev-support/hive-personality.sh | | git revision | master / fa97c67 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: standalone-metastore/metastore-server U: standalone-metastore/metastore-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14516/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; >
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652889#comment-16652889 ] Alan Gates commented on HIVE-20740: --- Will be a couple of days before I have time to look at it. cc [~thejas] > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652579#comment-16652579 ] Vihang Karajgaonkar commented on HIVE-20740: v2 patch includes suggested changes from [~asherman] on review board > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652576#comment-16652576 ] Vihang Karajgaonkar commented on HIVE-20740: Added review board link. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652401#comment-16652401 ] Vihang Karajgaonkar commented on HIVE-20740: [~asherman] [~pvary] Can you please help review? > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652282#comment-16652282 ] Vihang Karajgaonkar commented on HIVE-20740: Turns out there was a race condition as well in the code. The {{getPMF}} is a {{static synchronized}} method (takes a lock on ObjectStore.class) while the {{setConf}} method takes a lock on {{pmfPropLock}} method. So in theory it is possible that some thread which calls {{getPMF}} is updating {{pmf}} instance while another thread which calls {{setConf}} is also updating {{pmf}} as can be seen from the code above. The patch moves all the updates to the {{pmf}} and {{prop}} static variables into one method where they are protected using read locks and write locks to optimize for the common case of datasource properties not getting updated often. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)