[jira] [Commented] (HIVE-2029) MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13133723#comment-13133723 ] Ashutosh Chauhan commented on HIVE-2029: @Paul, From the url I quoted above: bq. DataNucleus has the capability to switch to between DataSources upon failure of one while obtaining a datastore connection. The failover mechanism is useful for applications with multiple database nodes when the data is actually replicated/synchronized by the underlying database. So, DN will try to connect to a slave automatically when it can't connect with master. But it won't promote the slave to master so metastore can proceed with read only queries on DB. This falls short of what you have described, but once we upgrade to 3.x via HIVE-2084 we can atleast simplify the reconnect logic. MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist -- Key: HIVE-2029 URL: https://issues.apache.org/jira/browse/HIVE-2029 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Carl Steinbach Attachments: hive_2029.patch HIVE-1219 defined the JDOConnectionURLHook plugin, and integrated this feature into HiveMetaStore. On MetaStore operation failures, this plugin is used to update the metastore ConnectionURL configuration property. Currently this update triggers the reinitialization of the underlying JDO PersistenceManager, but it does not trigger checks to see if the default database exists, nor will it create the default database if it does not exist. It needs to do both. This ticket also covers removing the 'hive.metastore.force.reload.conf' property from HiveConf and HiveMetaStore. This property should not have been added in the first place since its sole purpose is to facilitate testing of the JDOConnectionURLHook mechanism by unnaturally forcing reinitialization of the PersistenceManager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2029) MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13043434#comment-13043434 ] Ashutosh Chauhan commented on HIVE-2029: Does anyone have any opinion on this? MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist -- Key: HIVE-2029 URL: https://issues.apache.org/jira/browse/HIVE-2029 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Carl Steinbach Attachments: hive_2029.patch HIVE-1219 defined the JDOConnectionURLHook plugin, and integrated this feature into HiveMetaStore. On MetaStore operation failures, this plugin is used to update the metastore ConnectionURL configuration property. Currently this update triggers the reinitialization of the underlying JDO PersistenceManager, but it does not trigger checks to see if the default database exists, nor will it create the default database if it does not exist. It needs to do both. This ticket also covers removing the 'hive.metastore.force.reload.conf' property from HiveConf and HiveMetaStore. This property should not have been added in the first place since its sole purpose is to facilitate testing of the JDOConnectionURLHook mechanism by unnaturally forcing reinitialization of the PersistenceManager. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2029) MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13044032#comment-13044032 ] Paul Yang commented on HIVE-2029: - Can you elaborate on how this retry feature works in datanucleus 3.0? The case that could be handled with the URL hook is as follows - a db host goes down. A failover is performed and a replica on a different host is promoted to be the new master. Using the hook, the client is able to re-execute the query on the new host and the Hive query succeeds without failure. Would it be possible to implement something similar in datanucleus 3.0? MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist -- Key: HIVE-2029 URL: https://issues.apache.org/jira/browse/HIVE-2029 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Carl Steinbach Attachments: hive_2029.patch HIVE-1219 defined the JDOConnectionURLHook plugin, and integrated this feature into HiveMetaStore. On MetaStore operation failures, this plugin is used to update the metastore ConnectionURL configuration property. Currently this update triggers the reinitialization of the underlying JDO PersistenceManager, but it does not trigger checks to see if the default database exists, nor will it create the default database if it does not exist. It needs to do both. This ticket also covers removing the 'hive.metastore.force.reload.conf' property from HiveConf and HiveMetaStore. This property should not have been added in the first place since its sole purpose is to facilitate testing of the JDOConnectionURLHook mechanism by unnaturally forcing reinitialization of the PersistenceManager. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2029) MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13034294#comment-13034294 ] Ashutosh Chauhan commented on HIVE-2029: My question didn't get any answers. So, let me ask another question. I think failover is present in datanucleus 3.0 and Hive is still on 2.x. So, that may be the reason that Hive implemented its own failover. If this is correct, shall we first upgrade Datanucleus version to 3.x and then switch to Datanucleus functionality instead of our own retrials. If there is an agreement then patch on this ticket can be committed as is and I will open tickets for followup work. MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist -- Key: HIVE-2029 URL: https://issues.apache.org/jira/browse/HIVE-2029 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Carl Steinbach Attachments: hive_2029.patch HIVE-1219 defined the JDOConnectionURLHook plugin, and integrated this feature into HiveMetaStore. On MetaStore operation failures, this plugin is used to update the metastore ConnectionURL configuration property. Currently this update triggers the reinitialization of the underlying JDO PersistenceManager, but it does not trigger checks to see if the default database exists, nor will it create the default database if it does not exist. It needs to do both. This ticket also covers removing the 'hive.metastore.force.reload.conf' property from HiveConf and HiveMetaStore. This property should not have been added in the first place since its sole purpose is to facilitate testing of the JDOConnectionURLHook mechanism by unnaturally forcing reinitialization of the PersistenceManager. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2029) MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027219#comment-13027219 ] Ashutosh Chauhan commented on HIVE-2029: +1 for removing 'hive.metastore.force.reload.conf'. Tests should not be written in such a manner that they need to modify the code. This has opened up multiple paths in Metastore code which needs to be reasoned about. Is there an agreement on whether connectionURL hook should look for default DB and create it in case it doesn't exist? MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist -- Key: HIVE-2029 URL: https://issues.apache.org/jira/browse/HIVE-2029 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Carl Steinbach HIVE-1219 defined the JDOConnectionURLHook plugin, and integrated this feature into HiveMetaStore. On MetaStore operation failures, this plugin is used to update the metastore ConnectionURL configuration property. Currently this update triggers the reinitialization of the underlying JDO PersistenceManager, but it does not trigger checks to see if the default database exists, nor will it create the default database if it does not exist. It needs to do both. This ticket also covers removing the 'hive.metastore.force.reload.conf' property from HiveConf and HiveMetaStore. This property should not have been added in the first place since its sole purpose is to facilitate testing of the JDOConnectionURLHook mechanism by unnaturally forcing reinitialization of the PersistenceManager. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
