[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279674#comment-15279674 ] Lefty Leverenz commented on HIVE-13596: --- Removed the TODOC2.1 label. Thanks, [~asears]. > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.03.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279417#comment-15279417 ] Sergey Shelukhin commented on HIVE-13596: - Thanks! > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.03.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279362#comment-15279362 ] Andrew Sears commented on HIVE-13596: - Doc'd! > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.03.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265227#comment-15265227 ] Lefty Leverenz commented on HIVE-13596: --- Doc note: This adds *hive.allow.udf.load.on.demand* to HiveConf.java so it will need to be documented in the wiki for release 2.1.0. * [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution] TODOC2.1 label added to this issue. > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Labels: TODOC2.1 > Fix For: 2.1.0 > > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.03.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264438#comment-15264438 ] Sergey Shelukhin commented on HIVE-13596: - Test failures are the same as in other recent JIRAs... metastore problems on the server, or broken by other jiras > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.03.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263429#comment-15263429 ] Hive QA commented on HIVE-13596: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12801156/HIVE-13596.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 42 failed/errored test(s), 9961 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestMiniTezCliDriver-vector_non_string_partition.q-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccessWithReadOnly org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.jdbc.TestSSL.testSSLFetchHttp org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/111/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/111/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-111/ Messages: {noformat} Executing
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261303#comment-15261303 ] Jason Dere commented on HIVE-13596: --- ok, i think this looks good. +1 if tests look good. > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.03.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15259139#comment-15259139 ] Sergey Shelukhin commented on HIVE-13596: - RB at https://reviews.apache.org/r/46715/ > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13596.01.patch, HIVE-13596.02.patch, > HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1525#comment-1525 ] Jason Dere commented on HIVE-13596: --- Ok, looking at the code again, I see why checkFunctionClass() is no longer called - there is a separate registerToSessionRegistry() method to add a UDF from the system registry to the session registry. It's not bad to have it in the session registry, since the UDF does eventually need to get added there, though it is a bit different from the UDFs added via reloadFunctions() since those are added to the system registry. > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13596.01.patch, HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258724#comment-15258724 ] Sergey Shelukhin commented on HIVE-13596: - {noformat} Should this be a settable option (as opposed to always on)? And why default to false? {noformat} It's settable per session. Defaulting to false because that was the current behavior for a while. {noformat} Which Registry is performing the UDF lookup, the system registry or the session registry? If it is the system registry, then we may run into HIVE-6672 again. checkFunctionClass() (removed in your patch) was added for this purpose. {noformat} Session registry. I removed the method because it was unused... {noformat} If the functions are being looked up/added to the session registry, then this may not be an issue because every session would need to lookup UDF/load JARs. Actually I see that the permanent UDFs by Hive.reloadFunctions() (at initialize time) are added to the system registry .. I suspect Hive probably has class loading issues if we ever use "RELOAD FUNCTIONS" to pick up new UDFs, since Hive no longer seems to be calling checkFunctionClass(). {noformat} Hmm... should this be done in the system registry then? Does it hurt to have them in the session registry? Also, does checkFunctionClass need to be reinstated here or in a separate JIRA. {noformat} public Registry(boolean isNative, HiveConf conf): conf needs a null check before it's used {noformat} Implied that it's not used for a native registry; I will add an explicit check. > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13596) HS2 should be able to get UDFs on demand from metastore
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15258709#comment-15258709 ] Jason Dere commented on HIVE-13596: --- - Should this be a settable option (as opposed to always on)? And why default to false? - Which Registry is performing the UDF lookup, the system registry or the session registry? If it is the system registry, then we may run into HIVE-6672 again. checkFunctionClass() (removed in your patch) was added for this purpose. If the functions are being looked up/added to the session registry, then this may not be an issue because every session would need to lookup UDF/load JARs. Actually I see that the permanent UDFs by Hive.reloadFunctions() (at initialize time) are added to the system registry .. I suspect Hive probably has class loading issues if we ever use "RELOAD FUNCTIONS" to pick up new UDFs, since Hive no longer seems to be calling checkFunctionClass(). - public Registry(boolean isNative, HiveConf conf): - conf needs a null check before it's used - private FunctionInfo getQualifiedFunctionInfo(): - Wrap if/then with braces - private boolean refreshFunctionInfoFromMetastore(String functionName) - line 629: wrap if/then with braces > HS2 should be able to get UDFs on demand from metastore > --- > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13596.patch > > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We should restore the pre-HIVE-2573 behavior of being able to refresh the > UDFs on demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)