[jira] [Commented] (HIVE-14609) HS2 cannot drop a function whose associated jar file has been removed
[
https://issues.apache.org/jira/browse/HIVE-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770745#comment-16770745
]
Hui Huang commented on HIVE-14609:
--
Maybe we can get the function info directly from metastore inside the method
`FunctionSemanticAnalyzer.analyzeDropFunction`:
{code:java}
FunctionInfo info = FunctionRegistry.getFunctionInfo(functionName);
if (info == null) {
try {
String[] parts =
FunctionUtils.getQualifiedFunctionNameParts(functionName);
Function func = Hive.get(conf).getFunction(parts[0].toLowerCase(),
parts[1]);
if (func != null) {
info = new FunctionInfo(functionName, func.getClassName(),
FunctionTask.toFunctionResource(func.getResourceUris()));
}
} catch (HiveException e) {
e.printStackTrace();
}
}
if (info == null) {
if (throwException) {
throw new
SemanticException(ErrorMsg.INVALID_FUNCTION.getMsg(functionName));
} else {
// Fail silently
return;
}
} else if (info.isBuiltIn()) {
throw new
SemanticException(ErrorMsg.DROP_NATIVE_FUNCTION.getMsg(functionName));
}
{code}
> HS2 cannot drop a function whose associated jar file has been removed
> -
>
> Key: HIVE-14609
> URL: https://issues.apache.org/jira/browse/HIVE-14609
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Chaoyu Tang
>Priority: Major
>
> Create a permanent function with below command:
> {code:sql}
> create function yshi.dummy as 'com.yshi.hive.udf.DummyUDF' using jar
> 'hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar';
> {code}
> After that, delete the HDFS file
> {{hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar}}, and
> *restart HS2 to remove the loaded class*.
> Now the function cannot be dropped:
> {noformat}
> 0: jdbc:hive2://10.17.81.144:1/default> show functions yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 1.259 seconds
> INFO : Executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO : Completed executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 0.024 seconds
> INFO : OK
> +-+--+
> | tab_name |
> +-+--+
> | yshi.dummy |
> +-+--+
> 1 row selected (3.877 seconds)
> 0: jdbc:hive2://10.17.81.144:1/default> drop function yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc):
> drop function yshi.dummy
> INFO : converting to local
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> ERROR : Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1200)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1136)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1126)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:470)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:456)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:245)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:455)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:99)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:61)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:451)
>
[jira] [Commented] (HIVE-14609) HS2 cannot drop a function whose associated jar file has been removed
[
https://issues.apache.org/jira/browse/HIVE-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624170#comment-16624170
]
BELUGA BEHR commented on HIVE-14609:
By the same token, I cannot {{describe function}} either to figure out where
the missing JAR file is.
{code}
0: jdbc:hive2://host> describe function row_sequence;
INFO : Compiling
command(queryId=hive_2018092113_3c26b2ae-9f0a-4a80-ba3c-a96b23fe8f9d):
describe function row_sequence
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_2018092113_3c26b2ae-9f0a-4a80-ba3c-a96b23fe8f9d); Time
taken: 0.286 seconds
INFO : Executing
command(queryId=hive_2018092113_3c26b2ae-9f0a-4a80-ba3c-a96b23fe8f9d):
describe function row_sequence
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : converting to local hdfs://ns1/tmp/hive-contrib-1.1.0.jar
ERROR : Failed to read external resource hdfs://ns1/tmp/hive-contrib-1.1.0.jar
java.lang.RuntimeException: Failed to read external resource
hdfs://ns1/tmp/hive-contrib-1.1.0.jar
at
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1442)
at
org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1398)
at
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1322)
at
org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1308)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
at
org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:570)
at
org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:556)
at
org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:308)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:471)
at
org.apache.hadoop.hive.ql.exec.DDLTask.describeFunction(DDLTask.java:2907)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:385)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2054)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1750)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1503)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1287)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1282)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
at
org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
at
org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
at
org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File does not exist:
hdfs://ns1/tmp/hive-contrib-1.1.0.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1270)
at
org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1262)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2123)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2092)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2068)
at
org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1428)
... 29 more
INFO : Completed executing
command(queryId=hive_2018092113_3c26b2ae-9f0a-4a80-ba3c-a96b23fe8f9d); Time
taken: 0.383 seconds
INFO : OK
+--+--+
|
[jira] [Commented] (HIVE-14609) HS2 cannot drop a function whose associated jar file has been removed
[
https://issues.apache.org/jira/browse/HIVE-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435978#comment-15435978
]
Yibing Shi commented on HIVE-14609:
---
To drop a function, Hive first gets the function definition:
https://github.com/cloudera/hive/blob/cdh5-1.1.0_5.8.0/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java#L99
{code}
FunctionInfo info = FunctionRegistry.getFunctionInfo(functionName);
if (info == null) {
if (throwException) {
throw new
SemanticException(ErrorMsg.INVALID_FUNCTION.getMsg(functionName));
} else {
// Fail silently
return;
}
} else if (info.isBuiltIn()) {
throw new
SemanticException(ErrorMsg.DROP_NATIVE_FUNCTION.getMsg(functionName));
}
{code}
Unfortunately {{FunctionRegistry.getFunctionInfo}} tries to load the function
into registry after gets its definition, which includes the step of downloading
jars and causes the failure. We should be able to fix this by adding one
parameter to the getFunctionInfo method to control whether to adds the function
to registry.
And for the reason why Hive fails silently, it is because
"hive.exec.drop.ignorenonexistent" is set to true by default, and thus Hive
doesn't throw any exception when the failure happens.
> HS2 cannot drop a function whose associated jar file has been removed
> -
>
> Key: HIVE-14609
> URL: https://issues.apache.org/jira/browse/HIVE-14609
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Chaoyu Tang
>
> Create a permanent function with below command:
> {code:sql}
> create function yshi.dummy as 'com.yshi.hive.udf.DummyUDF' using jar
> 'hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar';
> {code}
> After that, delete the HDFS file
> {{hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar}}, and
> *restart HS2 to remove the loaded class*.
> Now the function cannot be dropped:
> {noformat}
> 0: jdbc:hive2://10.17.81.144:1/default> show functions yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 1.259 seconds
> INFO : Executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO : Completed executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 0.024 seconds
> INFO : OK
> +-+--+
> | tab_name |
> +-+--+
> | yshi.dummy |
> +-+--+
> 1 row selected (3.877 seconds)
> 0: jdbc:hive2://10.17.81.144:1/default> drop function yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc):
> drop function yshi.dummy
> INFO : converting to local
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> ERROR : Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1200)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1136)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1126)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:470)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:456)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:245)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:455)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:99)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:61)
> at
>
[jira] [Commented] (HIVE-14609) HS2 cannot drop a function whose associated jar file has been removed
[
https://issues.apache.org/jira/browse/HIVE-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435536#comment-15435536
]
Sergey Shelukhin commented on HIVE-14609:
-
Why is it even trying to download on drop function in the first place?
> HS2 cannot drop a function whose associated jar file has been removed
> -
>
> Key: HIVE-14609
> URL: https://issues.apache.org/jira/browse/HIVE-14609
> Project: Hive
> Issue Type: Bug
>Reporter: Yibing Shi
>Assignee: Chaoyu Tang
>
> Create a permanent function with below command:
> {code:sql}
> create function yshi.dummy as 'com.yshi.hive.udf.DummyUDF' using jar
> 'hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar';
> {code}
> After that, delete the HDFS file
> {{hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar}}, and
> *restart HS2 to remove the loaded class*.
> Now the function cannot be dropped:
> {noformat}
> 0: jdbc:hive2://10.17.81.144:1/default> show functions yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 1.259 seconds
> INFO : Executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO : Completed executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 0.024 seconds
> INFO : OK
> +-+--+
> | tab_name |
> +-+--+
> | yshi.dummy |
> +-+--+
> 1 row selected (3.877 seconds)
> 0: jdbc:hive2://10.17.81.144:1/default> drop function yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc):
> drop function yshi.dummy
> INFO : converting to local
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> ERROR : Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1200)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1136)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1126)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:470)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:456)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:245)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:455)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:99)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:61)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:451)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:316)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
>
