[ 
https://issues.apache.org/jira/browse/HIVE-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435978#comment-15435978
 ] 

Yibing Shi commented on HIVE-14609:
-----------------------------------

To drop a function, Hive first gets the function definition:
https://github.com/cloudera/hive/blob/cdh5-1.1.0_5.8.0/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java#L99
{code}
    FunctionInfo info = FunctionRegistry.getFunctionInfo(functionName);
    if (info == null) {
      if (throwException) {
        throw new 
SemanticException(ErrorMsg.INVALID_FUNCTION.getMsg(functionName));
      } else {
        // Fail silently
        return;
      }
    } else if (info.isBuiltIn()) {
      throw new 
SemanticException(ErrorMsg.DROP_NATIVE_FUNCTION.getMsg(functionName));
    }
{code}

Unfortunately {{FunctionRegistry.getFunctionInfo}} tries to load the function 
into registry after gets its definition, which includes the step of downloading 
jars and causes the failure. We should be able to fix this by adding one 
parameter to the getFunctionInfo method to control whether to adds the function 
to registry.

And for the reason why Hive fails silently, it is because 
"hive.exec.drop.ignorenonexistent" is set to true by default, and thus Hive 
doesn't throw any exception when the failure happens.

> HS2 cannot drop a function whose associated jar file has been removed
> ---------------------------------------------------------------------
>
>                 Key: HIVE-14609
>                 URL: https://issues.apache.org/jira/browse/HIVE-14609
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Yibing Shi
>            Assignee: Chaoyu Tang
>
> Create a permanent function with below command:
> {code:sql}
> create function yshi.dummy as 'com.yshi.hive.udf.DummyUDF' using jar 
> 'hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar';
> {code}
> After that, delete the HDFS file 
> {{hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar}}, and 
> *restart HS2 to remove the loaded class*.
> Now the function cannot be dropped:
> {noformat}
> 0: jdbc:hive2://10.17.81.144:10000/default> show functions yshi.dummy;
> INFO  : Compiling 
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded): 
> show functions yshi.dummy
> INFO  : Semantic Analysis Completed
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded); 
> Time taken: 1.259 seconds
> INFO  : Executing 
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded): 
> show functions yshi.dummy
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO  : Completed executing 
> command(queryId=hive_20160821213434_d0271d77-84d8-45ba-8d92-3da1c143bded); 
> Time taken: 0.024 seconds
> INFO  : OK
> +-------------+--+
> |  tab_name   |
> +-------------+--+
> | yshi.dummy  |
> +-------------+--+
> 1 row selected (3.877 seconds)
> 0: jdbc:hive2://10.17.81.144:10000/default> drop function yshi.dummy;
> INFO  : Compiling 
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc): 
> drop function yshi.dummy
> INFO  : converting to local 
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> ERROR : Failed to read external resource 
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
> java.lang.RuntimeException: Failed to read external resource 
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
>               at 
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1200)
>               at 
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1136)
>               at 
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1126)
>               at 
> org.apache.hadoop.hive.ql.exec.FunctionTask.addFunctionResources(FunctionTask.java:304)
>               at 
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:470)
>               at 
> org.apache.hadoop.hive.ql.exec.Registry.getQualifiedFunctionInfo(Registry.java:456)
>               at 
> org.apache.hadoop.hive.ql.exec.Registry.getFunctionInfo(Registry.java:245)
>               at 
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionInfo(FunctionRegistry.java:455)
>               at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeDropFunction(FunctionSemanticAnalyzer.java:99)
>               at 
> org.apache.hadoop.hive.ql.parse.FunctionSemanticAnalyzer.analyzeInternal(FunctionSemanticAnalyzer.java:61)
>               at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>               at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:451)
>               at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
>               at 
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
>               at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1181)
>               at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:134)
>               at 
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:206)
>               at 
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:316)
>               at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
>               at 
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:401)
>               at 
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
>               at 
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
>               at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
>               at 
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
>               at 
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>               at 
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>               at 
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:718)
>               at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
>               at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>               at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>               at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: File does not exist: 
> hdfs://host-10-17-81-142.coe.cloudera.com:8020/hive/jars/yshi.jar
>               at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1219)
>               at 
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
>               at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>               at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
>               at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340)
>               at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
>               at 
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2014)
>               at 
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1983)
>               at 
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1959)
>               at 
> org.apache.hadoop.hive.ql.session.SessionState.downloadResource(SessionState.java:1186)
>               ... 30 more
> INFO  : Semantic Analysis Completed
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc); 
> Time taken: 0.297 seconds
> INFO  : Executing 
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc): 
> drop function yshi.dummy
> INFO  : Completed executing 
> command(queryId=hive_20160821213434_47d14df5-59b3-4ebc-9a48-5e1d9c60c1fc); 
> Time taken: 0.003 seconds
> INFO  : OK
> No rows affected (0.324 seconds)
> 0: jdbc:hive2://10.17.81.144:10000/default> show functions yshi.dummy;
> INFO  : Compiling 
> command(queryId=hive_20160821213434_b69fd2a2-ec0b-463c-821c-3273d834c0ab): 
> show functions yshi.dummy
> INFO  : Semantic Analysis Completed
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20160821213434_b69fd2a2-ec0b-463c-821c-3273d834c0ab); 
> Time taken: 0.123 seconds
> INFO  : Executing 
> command(queryId=hive_20160821213434_b69fd2a2-ec0b-463c-821c-3273d834c0ab): 
> show functions yshi.dummy
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : SHOW FUNCTIONS is deprecated, please use SHOW FUNCTIONS LIKE instead.
> INFO  : Completed executing 
> command(queryId=hive_20160821213434_b69fd2a2-ec0b-463c-821c-3273d834c0ab); 
> Time taken: 0.004 seconds
> INFO  : OK
> +-------------+--+
> |  tab_name   |
> +-------------+--+
> | yshi.dummy  |
> +-------------+--+
> 1 row selected (0.15 seconds)
> 0: jdbc:hive2://10.17.81.144:10000/default>
> {noformat}
> There are 2 problem to fix here:
> # Returns an error to client if the dop function operation fails.
> # Drop the function definition even though the associated jar file cannot be 
> downloaded to local file system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to