Hi Yang, thanks a lot for your help. It ended up being the case that my command in the initContainer was specified incorrectly.
On Thu, 7 Apr 2022 at 18:41, Yang Wang <danrtsey...@gmail.com> wrote: > It seems that you have a typo when specifying the pipeline classpath. > "file:///flink-jar/flink-connector-rabbitmq_2.12-1.14.4.jar" -> > "file:///flink-jars/flink-connector-rabbitmq_2.12-1.14.4.jar" > > If this is not the root cause, maybe you could have a try with downloading > the connector jars to /opt/flink/usrlib. The usrlib will be loaded to the > user classloader automatically without any configuration. > > BTW, I am not aware of any other bugs which will cause pipeline classpath > not take effect except FLINK-21289[1]. > > [1]. https://issues.apache.org/jira/browse/FLINK-21289 > > Best, > Yang > > Francis Conroy <francis.con...@switchdin.com> 于2022年4月7日周四 15:14写道： > >> Hi all, >> thanks in advance for any tips. >> >> I've been trying to specify some additional classpaths in my kubernetes >> yaml file when using the official flink operator and nothing seems to work. >> >> I know the technique for getting my job jar works fine since it's finding >> the class ok, but I cannot get the RabbitMQ connector jar to load. >> >> apiVersion: flink.apache.org/v1alpha1 >> kind: FlinkDeployment >> metadata: >> namespace: default >> name: http-over-mqtt >> spec: >> image: flink:1.14.4-scala_2.12-java11 >> flinkVersion: v1_14 >> flinkConfiguration: >> taskmanager.numberOfTaskSlots: "2" >> pipeline.classpaths: >> "file:///flink-jar/flink-connector-rabbitmq_2.12-1.14.4.jar" >> serviceAccount: flink >> jobManager: >> replicas: 1 >> resource: >> memory: "1024m" >> cpu: 1 >> taskManager: >> resource: >> memory: "1024m" >> cpu: 1 >> podTemplate: >> spec: >> serviceAccount: flink >> containers: >> - name: flink-main-container >> volumeMounts: >> - mountPath: /flink-job >> name: flink-jobs >> - mountPath: /flink-jars >> name: flink-jars >> initContainers: >> - name: grab-mqtt-over-http-jar >> image: busybox >> command: [ '/bin/sh', '-c', >> 'cd /tmp/job; wget >> https://jenkins/job/platform_flink/job/master/39/artifact/src-java/switchdin-topologies/target/switchdin-topologies-1.0-SNAPSHOT.jar >> --no-check-certificate;', >> 'cd /tmp/jar; wget >> https://repo1.maven.org/maven2/org/apache/flink/flink-connector-rabbitmq_2.12/1.14.4/flink-connector-rabbitmq_2.12-1.14.4.jar' >> ] >> volumeMounts: >> - name: flink-jobs >> mountPath: /tmp/job >> - name: flink-jars >> mountPath: /tmp/jar >> volumes: >> - name: flink-jobs >> emptyDir: { } >> - name: flink-jars >> emptyDir: { } >> job: >> jarURI: local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar >> entryClass: org.switchdin.HTTPOverMQTT >> parallelism: 1 >> upgradeMode: stateless >> state: running >> >> Any ideas? I've looked at the ConfigMaps that result and they also look >> fine. >> apiVersion: v1 >> data: >> flink-conf.yaml: "blob.server.port: 6124\nkubernetes.jobmanager.replicas: >> 1\njobmanager.rpc.address: >> http-over-mqtt.default\nkubernetes.taskmanager.cpu: 1.0\n >> kubernetes.service-account: >> flink\nkubernetes.cluster-id: http-over-mqtt\n >> $internal.application.program-args: >> \nkubernetes.container.image: flink:1.14.4-scala_2.12-java11\n >> parallelism.default: >> 1\nkubernetes.namespace: default\ntaskmanager.numberOfTaskSlots: 2\n >> kubernetes.rest-service.exposed.type: >> ClusterIP\n$internal.application.main: org.switchdin.HTTPOverMQTT\n >> taskmanager.memory.process.size: >> 1024m\nkubernetes.internal.jobmanager.entrypoint.class: >> org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint >> \nkubernetes.pod-template-file: >> /tmp/podTemplate_11292791104169595925.yaml\n >> kubernetes.pod-template-file.taskmanager: >> /tmp/podTemplate_17362225267763549900.yaml\nexecution.target: >> kubernetes-application\njobmanager.memory.process.size: >> 1024m\njobmanager.rpc.port: 6123\ntaskmanager.rpc.port: 6122\n >> internal.cluster.execution-mode: >> NORMAL\nqueryable-state.proxy.ports: 6125\npipeline.jars: >> local:///flink-job/switchdin-topologies-1.0-SNAPSHOT.jar\n >> kubernetes.jobmanager.cpu: >> 1.0\npipeline.classpath: >> file:///flink-jars/flink-connector-rabbitmq_2.12-1.14.4.jar\n >> kubernetes.pod-template-file.jobmanager: >> /tmp/podTemplate_17029501154997462433.yaml\n" >> >> >> >> This email and any attachments are proprietary and confidential and are >> intended solely for the use of the individual to whom it is addressed. Any >> views or opinions expressed are solely those of the author and do not >> necessarily reflect or represent those of SwitchDin Pty Ltd. If you have >> received this email in error, please let us know immediately by reply email >> and delete it from your system. You may not use, disseminate, distribute or >> copy this message nor disclose its contents to anyone. >> SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 >> Australia >> > -- This email and any attachments are proprietary and confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily reflect or represent those of SwitchDin Pty Ltd. If you have received this email in error, please let us know immediately by reply email and delete it from your system. You may not use, disseminate, distribute or copy this message nor disclose its contents to anyone. SwitchDin Pty Ltd (ABN 29 154893857) PO Box 1165, Newcastle NSW 2300 Australia
