opera33 opened a new issue #7650:
URL: https://github.com/apache/pulsar/issues/7650
**Describe the bug**
Randomly getting IO exception with Pulsar Function, We have about 500 of the
functions running on 1 cluster with 3 nodes, using same jar (fat jar) file with
2 brokers with function workers, and 1 broker without function worker. (all
running 2.5.2)
Some of the function may randomly hit IO Exception and upon investigating,
this is the error message we got in the log.
**Expected behavior**
Function run as usual
**Desktop (please complete the following information):**
- Ubuntu 20.04
- Pulsar 2.5.2
**Additional context**
`01:33:40.240 [public/default/logofb20337-0] INFO
org.apache.pulsar.functions.instance.JavaInstanceRunnable - Load JAR:
/home/ubuntu/apache-pulsar-2.5.2/functions/pulsar-1.0-jar-with-dependencies.jar
01:33:42.922 [public/default/logofb20337-0] ERROR
org.apache.pulsar.functions.instance.JavaInstanceRunnable -
[public/default/logofb20337:0] Uncaught exception in Java Instance
java.io.IOException: Unable to delete
/tmp/pulsar-nar/pulsar-1.0-jar-with-dependencies.jar-unpacked/org/apache/pulsar/shade/org/asynchttpclient/netty/channel/DefaultChannelPool$PoolLeaseStrategy$2.class
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:213)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFiles(FileUtils.java:202)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.FileUtils.deleteFile(FileUtils.java:209)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.NarUnpacker.unpackNar(NarUnpacker.java:73)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.common.nar.NarClassLoader.getFromArchive(NarClassLoader.java:151)
~[org.apache.pulsar-pulsar-common-2.5.2.jar:2.5.2]
at
org.apache.pulsar.functions.utils.functioncache.FunctionCacheEntry.<init>(FunctionCacheEntry.java:75)
~[org.apache.pulsar-pulsar-functions-utils-2.5.2.jar:2.5.2]
at
org.apache.pulsar.functions.utils.functioncache.FunctionCacheManagerImpl.registerFunctionInstanceWithArchive(FunctionCacheManagerImpl.java:128)
~[org.apache.pulsar-pulsar-functions-utils-2.5.2.jar:2.5.2]
at
org.apache.pulsar.functions.instance.JavaInstanceRunnable.loadJars(JavaInstanceRunnable.java:306)
~[org.apache.pulsar-pulsar-functions-instance-2.5.2.jar:?]
at
org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupJavaInstance(JavaInstanceRunnable.java:190)
~[org.apache.pulsar-pulsar-functions-instance-2.5.2.jar:?]
at
org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:245)
[org.apache.pulsar-pulsar-functions-instance-2.5.2.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
