lhotari commented on pull request #9638:
URL: https://github.com/apache/pulsar/pull/9638#issuecomment-793730034
>Again, this not true or not suppose to be true. I wrote and designed the
classloading mechanism in functions so at least i know the intention if for
user code to be loaded in a separate classloader than doesn't contain all of
pulsar dependencies. However, currently because of issue such as #9246 , the
classloader for user code maybe polluted with unnecessary deps.
@jerrypeng You are right that the problem is caused by PR #9246 (open issue
#9640) . I ran some tests with a k8s cluster with Pulsar 2.7.0 . There, the
classloaders are properly isolated and the classloaders contain only the
minimum dependencies.
Some output from my experiment:
```
kubectl --namespace=pulsar exec -it
pf-public-default-test-pulsar-inspector-0 -- jar tvf
/pulsar/instances/java-instance.jar |grep pom.xml
1495 Tue Dec 01 11:25:22 UTC 2020
META-INF/maven/org.apache.pulsar/pulsar-io-core/pom.xml
1793 Tue Dec 01 11:25:22 UTC 2020
META-INF/maven/org.apache.pulsar/pulsar-functions-api/pom.xml
2689 Tue Dec 01 11:25:22 UTC 2020
META-INF/maven/org.apache.pulsar/pulsar-client-api/pom.xml
3833 Thu Mar 16 16:53:42 UTC 2017
META-INF/maven/org.slf4j/slf4j-api/pom.xml
7718 Sun Nov 19 00:48:12 UTC 2017
META-INF/maven/org.apache.logging.log4j/log4j-slf4j-impl/pom.xml
12371 Sun Nov 19 00:48:10 UTC 2017
META-INF/maven/org.apache.logging.log4j/log4j-api/pom.xml
20371 Sun Nov 19 00:48:10 UTC 2017
META-INF/maven/org.apache.logging.log4j/log4j-core/pom.xml
3281 Tue Dec 01 11:25:22 UTC 2020
META-INF/maven/org.apache.pulsar/pulsar-functions-runtime-all/pom.xml
```
```
cat classloader_report.json|jq -r .classloader_report
Classloader:
org.apache.pulsar.common.nar.NarClassLoader[/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked]
identityHashCode:1618560520
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/jackson-annotations-2.11.2.jar
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/jackson-core-2.11.2.jar
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/jackson-databind-2.11.2.jar
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/pulsar-client-api-2.7.0.jar
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/pulsar-functions-api-2.7.0.jar
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/pulsar-io-core-2.7.0.jar
file:/tmp/pulsar-nar/pulsar-inspector-connector-1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/slf4j-api-1.7.25.jar
Parent:
Classloader: sun.misc.Launcher$AppClassLoader@7852e922
identityHashCode:2018699554
file:/pulsar/instances/java-instance.jar
file:/pulsar/instances/deps/*
Parent:
Classloader: sun.misc.Launcher$ExtClassLoader@4ac68d3e
identityHashCode:1254526270
file:/usr/local/openjdk-8/jre/lib/ext/cldrdata.jar
file:/usr/local/openjdk-8/jre/lib/ext/localedata.jar
file:/usr/local/openjdk-8/jre/lib/ext/sunpkcs11.jar
file:/usr/local/openjdk-8/jre/lib/ext/nashorn.jar
file:/usr/local/openjdk-8/jre/lib/ext/zipfs.jar
file:/usr/local/openjdk-8/jre/lib/ext/jaccess.jar
file:/usr/local/openjdk-8/jre/lib/ext/dnsns.jar
file:/usr/local/openjdk-8/jre/lib/ext/sunec.jar
file:/usr/local/openjdk-8/jre/lib/ext/sunjce_provider.jar
```
(btw. I created a full blown experiment in
https://github.com/lhotari/pulsar-inspector-connector which created the
classloader report above. I used as a tool to learn how to debug Pulsar
Functions in a local k8s cluster since I didn't have the development
environment setup for that previously.)
On 2.8.0-SNAPSHOT, java-instance.jar contains:
```
jar tvf pulsar-functions/runtime-all/target/java-instance.jar |grep pom.xml
2122 Wed Mar 03 14:02:44 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-io-core/pom.xml
2625 Mon Feb 15 12:51:24 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-functions-api/pom.xml
2590 Mon Feb 15 12:28:36 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-client-admin-api/pom.xml
6882 Wed Mar 03 14:02:44 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-common/pom.xml
6246 Wed Jul 01 11:07:32 EEST 2020
META-INF/maven/io.swagger/swagger-annotations/pom.xml
7375 Thu Jun 25 12:44:54 EEST 2020
META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml
3544 Thu Jun 25 12:29:14 EEST 2020
META-INF/maven/com.fasterxml.jackson.core/jackson-annotations/pom.xml
4860 Thu Jun 25 12:34:42 EEST 2020
META-INF/maven/com.fasterxml.jackson.core/jackson-core/pom.xml
19342 Thu Jul 09 12:44:34 EEST 2020
META-INF/maven/io.netty/netty-transport-native-unix-common/pom.xml
5208 Mon Oct 14 19:00:34 EEST 2019
META-INF/maven/io.airlift/aircompressor/pom.xml
1820 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-codec-haproxy/pom.xml
4084 Fri Nov 20 15:05:42 EET 2020
META-INF/maven/org.eclipse.jetty/jetty-util/pom.xml
2278 Thu Jun 25 14:09:32 EEST 2020
META-INF/maven/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml/pom.xml
37848 Fri Feb 28 09:06:20 EET 2020 META-INF/maven/org.yaml/snakeyaml/pom.xml
30198 Fri Aug 04 15:01:00 EEST 2017
META-INF/maven/javax.ws.rs/javax.ws.rs-api/pom.xml
7155 Wed Mar 03 14:02:44 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-client-original/pom.xml
2910 Wed Mar 03 14:02:44 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-transaction-common/pom.xml
4775 Fri Feb 14 21:06:54 EET 2020
META-INF/maven/com.google.protobuf/protobuf-java-util/pom.xml
2609 Mon Feb 15 12:26:12 EET 2021
META-INF/maven/org.apache.pulsar/bouncy-castle-bc/pom.xml
2402 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-codec-http/pom.xml
3000 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-resolver-dns/pom.xml
2143 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-codec-dns/pom.xml
2701 Wed Apr 08 07:38:40 EEST 2020
META-INF/maven/org.asynchttpclient/async-http-client/pom.xml
757 Wed Apr 08 07:38:40 EEST 2020
META-INF/maven/org.asynchttpclient/async-http-client-netty-utils/pom.xml
2150 Mon Nov 18 22:45:48 EET 2019
META-INF/maven/com.typesafe.netty/netty-reactive-streams/pom.xml
6515 Fri Sep 01 16:13:04 EEST 2017
META-INF/maven/com.sun.activation/javax.activation/pom.xml
2183 Sun Nov 20 15:37:30 EET 2016
META-INF/maven/com.yahoo.datasketches/sketches-core/pom.xml
807 Sun Nov 20 15:37:30 EET 2016
META-INF/maven/com.yahoo.datasketches/memory/pom.xml
2527 Fri Oct 04 11:54:30 EEST 2019
META-INF/maven/com.google.code.gson/gson/pom.xml
6380 Wed Aug 28 09:16:18 EEST 2019
META-INF/maven/org.apache.avro/avro/pom.xml
18227 Sat Aug 24 13:29:46 EEST 2019
META-INF/maven/org.apache.commons/commons-compress/pom.xml
4057 Wed Aug 28 09:16:18 EEST 2019
META-INF/maven/org.apache.avro/avro-protobuf/pom.xml
2949 Thu Jun 25 13:58:40 EEST 2020
META-INF/maven/com.fasterxml.jackson.module/jackson-module-jsonSchema/pom.xml
7860 Wed Apr 10 15:02:26 EEST 2013
META-INF/maven/javax.validation/validation-api/pom.xml
2673 Mon Feb 15 12:26:12 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-package-core/pom.xml
2225 Mon Feb 15 12:26:12 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-client-api/pom.xml
3833 Thu Mar 16 16:53:42 EET 2017
META-INF/maven/org.slf4j/slf4j-api/pom.xml
11507 Fri Nov 06 14:02:20 EET 2020
META-INF/maven/org.apache.logging.log4j/log4j-slf4j-impl/pom.xml
14045 Fri Nov 06 14:02:18 EET 2020
META-INF/maven/org.apache.logging.log4j/log4j-api/pom.xml
23342 Fri Nov 06 14:02:18 EET 2020
META-INF/maven/org.apache.logging.log4j/log4j-core/pom.xml
12208 Mon Dec 14 10:51:40 EET 2020
META-INF/maven/com.google.guava/guava/pom.xml
2413 Mon Nov 19 12:30:00 EET 2018
META-INF/maven/com.google.guava/failureaccess/pom.xml
2278 Tue Jan 01 03:00:00 EET 1980
META-INF/maven/com.google.guava/listenablefuture/pom.xml
4286 Fri Mar 31 10:21:36 EEST 2017
META-INF/maven/com.google.code.findbugs/jsr305/pom.xml
2111 Mon Dec 02 11:00:20 EET 2019
META-INF/maven/com.google.errorprone/error_prone_annotations/pom.xml
2762 Wed Jan 18 15:09:46 EET 2017
META-INF/maven/com.google.j2objc/j2objc-annotations/pom.xml
7575 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-common/pom.xml
4193 Fri Jan 03 10:59:38 EET 2020
META-INF/maven/org.jctools/jctools-core/pom.xml
2247 Thu Feb 18 00:45:14 EET 2021
META-INF/maven/org.apache.bookkeeper/bookkeeper-common-allocator/pom.xml
1578 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-buffer/pom.xml
5564 Fri Feb 14 21:06:54 EET 2020
META-INF/maven/com.google.protobuf/protobuf-java/pom.xml
17494 Thu Jan 13 23:05:12 EET 2011
META-INF/maven/commons-lang/commons-lang/pom.xml
3556 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-handler/pom.xml
1585 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-resolver/pom.xml
1924 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-transport/pom.xml
3587 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-codec/pom.xml
17788 Thu Jul 09 12:28:38 EEST 2020
META-INF/maven/io.netty/netty-transport-native-epoll/pom.xml
46166 Tue Aug 18 13:10:16 EEST 2020
META-INF/maven/io.netty/netty-tcnative-boringssl-static/pom.xml
11609 Wed Nov 05 23:26:00 EET 2014
META-INF/maven/commons-codec/commons-codec/pom.xml
13290 Thu Apr 14 09:17:56 EEST 2016
META-INF/maven/commons-io/commons-io/pom.xml
27495 Fri Jun 09 11:38:14 EEST 2017
META-INF/maven/org.apache.commons/commons-lang3/pom.xml
19206 Sat Jul 05 20:11:36 EEST 2014
META-INF/maven/commons-logging/commons-logging/pom.xml
20716 Thu Oct 24 01:20:20 EEST 2013
META-INF/maven/commons-configuration/commons-configuration/pom.xml
3854 Mon Feb 15 12:26:12 EET 2021
META-INF/maven/org.apache.pulsar/pulsar-functions-runtime-all/pom.xml
```
I hope #9640 gets addressed asap.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
