[
https://issues.apache.org/jira/browse/HIVE-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435978#comment-15435978
]
Yibing Shi commented on HIVE-14609:
-----------------------------------
To drop a function, Hive first gets the function definition:
https://github.com/cloudera/hive/blob/cdh5-1.1.0_5.8.0/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java#L99
{code}
FunctionInfo info = FunctionRegistry.getFunctionInfo(functionName);
if (info == null) {
if (throwException) {
throw new
SemanticException(ErrorMsg.INVALID_FUNCTION.getMsg(functionName));
} else {
// Fail silently
return;
}
} else if (info.isBuiltIn()) {
throw new
SemanticException(ErrorMsg.DROP_NATIVE_FUNCTION.getMsg(functionName));
}
{code}
Unfortunately {{FunctionRegistry.getFunctionInfo}} tries to load the function
into registry after gets its definition, which includes the step of downloading
jars and causes the failure. We should be able to fix this by adding one
parameter to the getFunctionInfo method to control whether to adds the function
to registry.
And for the reason why Hive fails silently, it is because
"hive.exec.drop.ignorenonexistent" is set to true by default, and thus Hive
doesn't throw any exception when the failure happens.
> HS2 cannot drop a function whose associated jar file has been removed
> ---------------------------------------------------------------------
>
> Key: HIVE-14609
> URL: https://issues.apache.org/jira/browse/HIVE-14609
> Project: Hive
> Issue Type: Bug
> Reporter: Yibing Shi
> Assignee: Chaoyu Tang
>
> Create a permanent function with below command:
> {code:sql}
> create function yshi.dummy as 'com.yshi.hive.udf.DummyUDF' using jar
> 'hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar';
> {code}
> After that, delete the HDFS file
> {{hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar}}, and
> *restart HS2 to remove the loaded class*.
> Now the function cannot be dropped:
> {noformat}
> 0: jdbc:hive2://10.17.81.144:10000/default> show functions yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 1.259 seconds
> INFO : Executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded):
> show functions yshi.dummy
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO : Completed executing
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded);
> Time taken: 0.024 seconds
> INFO : OK
> +-------------+--+
> | tab_name |
> +-------------+--+
> | yshi.dummy |
> +-------------+--+
> 1 row selected (3.877 seconds)
> 0: jdbc:hive2://10.17.81.144:10000/default> drop function yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc):
> drop function yshi.dummy
> INFO : converting to local
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> ERROR : Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> java.lang.RuntimeException: Failed to read external resource
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1200)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1136)
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1126)
> at
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:470)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:456)
> at
> org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:245)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:455)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:99)
> at
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:61)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:451)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1181)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:316)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:718)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1219)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
> at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2014)
> at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1983)
> at
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1959)
> at
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1186)
> ... 30 more
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc);
> Time taken: 0.297 seconds
> INFO : Executing
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc):
> drop function yshi.dummy
> INFO : Completed executing
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc);
> Time taken: 0.003 seconds
> INFO : OK
> No rows affected (0.324 seconds)
> 0: jdbc:hive2://10.17.81.144:10000/default> show functions yshi.dummy;
> INFO : Compiling
> command(queryId=hive_20160821213434_b69fd2a2-ec0b-463c-821c-3273d834c0ab):
> show functions yshi.dummy
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from
> deserializer)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20160821213434_b69fd2a2-ec0b-463c-821c-3273d834c0ab);
> Time taken: 0.123 seconds
> INFO : Executing
> command(queryId=hive_20160821213434_b69fd2a2-ec0b-463c-821c-3273d834c0ab):
> show functions yshi.dummy
> INFO : Starting task [Stage-0:DDL] in serial mode
> INFO : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO : Completed executing
> command(queryId=hive_20160821213434_b69fd2a2-ec0b-463c-821c-3273d834c0ab);
> Time taken: 0.004 seconds
> INFO : OK
> +-------------+--+
> | tab_name |
> +-------------+--+
> | yshi.dummy |
> +-------------+--+
> 1 row selected (0.15 seconds)
> 0: jdbc:hive2://10.17.81.144:10000/default>
> {noformat}
> There are 2 problem to fix here:
> # Returns an error to client if the dop function operation fails.
> # Drop the function definition even though the associated jar file cannot be
> downloaded to local file system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)