Mortom123 opened a new issue, #21069: URL: https://github.com/apache/pulsar/issues/21069
### Search before asking - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Version OS: Win10 (Dockerhost), RHEL 8.8 (Container) Pulsar Container Version: "Latest" (as of right now, this points to 3.0.1, running in the container) ### Minimal reproduce step My POM looks somewhat like this: ```xml <?xml version="1.0" encoding="UTF-8"?> <project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xmlns="http://maven.apache.org/POM/4.0.0"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> <modelVersion>4.0.0</modelVersion> <artifactId>example</artifactId> <groupId>com.example</groupId> <version>0.0.1</version> <name>example</name> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <kotlin.code.style>official</kotlin.code.style> <java.version>17</java.version> <kotlin.version>1.7.0</kotlin.version> <kotlin.compiler.jvmTarget>${java.version}</kotlin.compiler.jvmTarget> <jackson.version>2.15.2</jackson.version> <pulsar.version>3.0.1</pulsar.version> </properties> <dependencies> <dependency> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-stdlib</artifactId> <version>${kotlin.version}</version> </dependency> <dependency> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-reflect</artifactId> <version>${kotlin.version}</version> </dependency> <dependency> <groupId>org.jetbrains.kotlinx</groupId> <artifactId>kotlinx-coroutines-core</artifactId> <version>${kotlin.version}</version> </dependency> <dependency> <groupId>org.apache.pulsar</groupId> <artifactId>pulsar-io-core</artifactId> <version>${pulsar.version}</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>${jackson.version}</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.module</groupId> <artifactId>jackson-module-kotlin</artifactId> <version>${jackson.version}</version> </dependency> </dependencies> <build> <sourceDirectory>src/main/kotlin</sourceDirectory> <testSourceDirectory>src/test/kotlin</testSourceDirectory> <plugins> <!-- kotlin to java --> <plugin> <groupId>org.jetbrains.kotlin</groupId> <artifactId>kotlin-maven-plugin</artifactId> <version>${kotlin.version}</version> <executions> <execution> <id>compile</id> <phase>compile</phase> <goals> <goal>compile</goal> </goals> <configuration> <sourceDirs> <source>src/main/kotlin</source> <source>target/generated-sources/annotations</source> </sourceDirs> </configuration> </execution> <execution> <id>test-compile</id> <phase>test-compile</phase> <goals> <goal>test-compile</goal> </goals> <configuration> <sourceDirs> <source>src/test/kotlin</source> <source>target/generated-test-sources/test-annotations</source> </sourceDirs> </configuration> </execution> </executions> </plugin> <!-- java to class --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>${java.version}</source> <target>${java.version}</target> </configuration> </plugin> <!-- package nar --> <plugin> <groupId>org.apache.nifi</groupId> <artifactId>nifi-nar-maven-plugin</artifactId> <version>1.5.0</version> <executions> <execution> <id>default-nar</id> <phase>package</phase> <goals> <goal>nar</goal> </goals> </execution> </executions> </plugin> </plugins> </build> </project> ``` The Sink-Implementation looks like this: ```kotlin val mapper: ObjectMapper by lazy { val tmp = jacksonObjectMapper() .enable(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT) // ignore getters / setters (we use kotlin data classes with constructors) tmp.setVisibility( tmp.serializationConfig.defaultVisibilityChecker .withFieldVisibility(JsonAutoDetect.Visibility.ANY) .withGetterVisibility(JsonAutoDetect.Visibility.NONE) .withSetterVisibility(JsonAutoDetect.Visibility.NONE) ) tmp } inline fun <reified T : Any> parseFromMap(conf: Map<String, Any>): T { return mapper.readValue(mapper.writeValueAsString(conf), T::class.java) } class ExampleSink: Sink<Map<String, Any>> { private lateinit var sinkConfiguration: SinkConfiguration override fun open(config: MutableMap<String, Any>, sinkContext: SinkContext?) { sinkConfiguration = parseFromMap(config) println("HELLO") // just here as an example } override fun write(record: Record<Map<String, Any>>) { // write something } override fun close() { // close something } } ``` The whole pulsar setup is containerized and ran using docker. The project is built using `mvn clean package` and the sink is started using `./pulsar-admin sinks localrun --tenant public --namespace default --sink-type example-archive.nar --name exampleSinkTest --sink-config-file sink-config.yml --inputs test` It is important to use localrun because the sink itself will run in container and requires some native C-Libraries to bind against (not shown here), and I don't want to clutter the pulsar broker container with that. ### What did you expect to see? When providing the sink with a correct configuration, I would expect it to parse this configuration and print "HELLO" afterwards. ### What did you see instead? [root@5a8dbfb1b0f2 bin]# ./pulsar-admin sinks localrun --tenant public --namespace default --sink-type /app/pulsar/bin/*.nar --name exampleSinkTest --sink-config-file /app/pulsar/bin/sink-config.yml --inputs test 2023-08-25T12:41:59,104+0000 [main] INFO org.apache.pulsar.functions.utils.io.ConnectorUtils - Searching for connectors in /app/pulsar/connectors 2023-08-25T12:41:59,110+0000 [main] WARN org.apache.pulsar.functions.utils.io.ConnectorUtils - Connectors archive directory not found 2023-08-25T12:41:59,119+0000 [main] INFO org.apache.pulsar.common.nar.NarUnpacker - Created directory /tmp/pulsar_localrunner_nars_16906245267260121651/pulsar-nar/pulsar-example-connector-0.0.1-DEV.nar-u npacked 2023-08-25T12:42:01,215+0000 [main] INFO org.apache.pulsar.common.nar.NarUnpacker - Extracting /app/pulsar/bin/pulsar-example-connector-0.0.1-DEV.nar to /tmp/pulsar_localrunner_nars_16906245267260121651/ pulsar-nar/pulsar-example-connector-0.0.1-DEV.nar-unpacked/IvmEgg1ISuBuQBaI1r_s2A 2023-08-25T12:42:01,725+0000 [main] INFO org.apache.pulsar.functions.runtime.RuntimeSpawner - public/default/exampleSinkTest-0 RuntimeSpawner starting function 2023-08-25T12:42:01,729+0000 [main] INFO org.apache.pulsar.common.nar.FileUtils - Jar file /app/pulsar/bin/pulsar-example-connector-0.0.1-DEV.nar contains META-INF/bundled-dependencies, it may be a NAR f ile 2023-08-25T12:42:01,730+0000 [main] INFO org.apache.pulsar.functions.runtime.thread.ThreadRuntime - Trying Loading file as NAR file: /app/pulsar/bin/pulsar-example-connector-0.0.1-DEV.nar 2023-08-25T12:42:04,294+0000 [main] INFO org.apache.pulsar.functions.runtime.thread.ThreadRuntime - Initialize function class loader for function exampleSinkTest at function cache manager, functionClassL oader: org.apache.pulsar.common.nar.NarClassLoader[/tmp/pulsar_localrunner_nars_16906245267260121651/pulsar-nar/pulsar-example-connector-0.0.1-DEV.nar-unpacked/IvmEgg1ISuBuQBaI1r_s2A] 2023-08-25T12:42:04,300+0000 [main] INFO org.apache.pulsar.functions.runtime.thread.ThreadRuntime - ThreadContainer starting function with instanceId 0 functionId 1bc65d9d-1216-4c8f-9eca-e3cd849a4af1 n amespace default 2023-08-25T12:42:34,303+0000 [function-timer-thread-28-1] ERROR org.apache.pulsar.functions.runtime.RuntimeSpawner - public/default/exampleSinkTest Function Container is dead with following exception. Res tarting. kotlin.jvm.KotlinReflectionNotSupportedError: Kotlin reflection implementation is not found at runtime. Make sure you have kotlin-reflect.jar in the classpath at kotlin.jvm.internal.ClassReference.error(ClassReference.kt:88) ~[org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)] at kotlin.jvm.internal.ClassReference.getConstructors(ClassReference.kt:21) ~[org.jetbrains.kotlin-kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)] at kotlin.reflect.jvm.ReflectJvmMapping.getKotlinFunction(ReflectJvmMapping.kt:146) ~[?:?] at com.fasterxml.jackson.module.kotlin.KotlinNamesAnnotationIntrospector.findKotlinParameterName(KotlinNamesAnnotationIntrospector.kt:108) ~[?:?] at com.fasterxml.jackson.module.kotlin.KotlinNamesAnnotationIntrospector.findImplicitPropertyName(KotlinNamesAnnotationIntrospector.kt:57) ~[?:?] at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findImplicitPropertyName(AnnotationIntrospectorPair.java:502) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.1 4.2] at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addCreatorParam(POJOPropertiesCollector.java:643) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addCreators(POJOPropertiesCollector.java:623) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:431) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getPropertyMap(POJOPropertiesCollector.java:391) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getProperties(POJOPropertiesCollector.java:233) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.introspect.BasicBeanDescription._properties(BasicBeanDescription.java:164) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findProperties(BasicBeanDescription.java:239) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._findCreatorsFromProperties(BasicDeserializerFactory.java:328) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:272) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2. 14.2] at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:223) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:262) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:151) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:415) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:350) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:648) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4861) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4731) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3677) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3645) ~[com.fasterxml.jackson.core-jackson-databind-2.14.2.jar:2.14.2] at com.example.example.ExampleSink.open(ExampleSink.kt:62) ~[?:?] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupOutput(JavaInstanceRunnable.java:943) ~[org.apache.pulsar-pulsar-functions-instance-3.0.1.jar:3.0.1] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setup(JavaInstanceRunnable.java:251) ~[org.apache.pulsar-pulsar-functions-instance-3.0.1.jar:3.0.1] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:290) ~[org.apache.pulsar-pulsar-functions-instance-3.0.1.jar:3.0.1] at java.lang.Thread.run(Thread.java:833) ~[?:?] ### Anything else? The content of META-INF in the .nar archive looks like this: META-INF/bundled-dependencies/annotations-13.0.jar META-INF/bundled-dependencies/jackson-annotations-2.15.2.jar META-INF/bundled-dependencies/jackson-core-2.15.2.jar META-INF/bundled-dependencies/jackson-databind-2.15.2.jar META-INF/bundled-dependencies/jackson-module-kotlin-2.15.2.jar META-INF/bundled-dependencies/kotlin-reflect-1.7.0.jar META-INF/bundled-dependencies/kotlin-stdlib-1.7.0.jar META-INF/bundled-dependencies/kotlin-stdlib-common-1.7.0.jar META-INF/bundled-dependencies/kotlin-stdlib-jdk7-1.8.20.jar META-INF/bundled-dependencies/kotlin-stdlib-jdk8-1.8.20.jar META-INF/bundled-dependencies/kotlinx-coroutines-core-1.7.0.jar META-INF/bundled-dependencies/kotlinx-coroutines-core-jvm-1.7.0.jar META-INF/bundled-dependencies/pulsar-client-admin-api-3.0.1.jar META-INF/bundled-dependencies/pulsar-client-api-3.0.1.jar META-INF/bundled-dependencies/pulsar-functions-api-3.0.1.jar META-INF/bundled-dependencies/pulsar-io-core-3.0.1.jar META-INF/bundled-dependencies/slf4j-api-2.0.4.jar META-INF/pulsar-example-connector.kotlin_module META-INF/services/pulsar-io.yaml META-INF/maven/com.example/pulsar-example-connector/pom.xml META-INF/maven/com.example/pulsar-example-connector/pom.properties ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
