thanks Yang, I got to put it to work in the way that you said. https://github.com/felipegutierrez/explore-flink
Best, Felipe -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com On Thu, Sep 24, 2020 at 6:59 AM Yang Wang <danrtsey...@gmail.com> wrote: > > Hi Felipe, > > Currently, if you want to deploy a standalone job/application Flink cluster > on K8s via yamls. > You should have your own image with user jar baked located at > /opt/flink/usrlib. It could not be > specified via config option. Usually, you could add new layer on the official > docker image to > build in the user jar. > > A more graceful solution is init container[1], which could download the user > jar from remote > storage or just copy from local directory /opt/flink/opt to /opt/flink/usrlib. > > [1]. > https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-initialization/ > > Best, > Yang > > > Felipe Gutierrez <felipe.o.gutier...@gmail.com> 于2020年9月18日周五 下午5:52写道： >> >> Hi community, >> >> I am trying to deploy the default WordCount stream application on >> minikube using the official documentation at [1]. I am using minikube >> v1.13.0 on Ubuntu 18.04 and Kubernetes v1.19.0 on Docker 19.03.8. I >> could sucessfully start 1 job manager and 3 task managers using the >> yaml files flink-configuration-configmap.yaml, >> jobmanager-service.yaml, jobmanager-rest-service.yaml, >> jobmanager-session-deployment.yaml, and >> taskmanager-session-deployment.yaml (all available on the Apendix of >> this link [1]). >> >> Then I would like to start the word-count stream job available on the >> flink jar image [2], which I believe is available since it is built >> inside the default flink jar distribution. What I understood that I >> have to do is to create the objects based on the files >> jobmanager-job.yaml and taskmanager-job-deployment.yaml (also >> available on the link [1]). And, I think that I have to replace this >> line below on the object jobmanager-job.yaml >> (spec.template.spec.containers.name[jobmanager]): >> >> args: ["standalone-job", "--job-classname", >> "org.apache.flink.streaming.examples.wordcount.WordCount"] >> >> Is this correct? I am not sure if this is my entire error. I am >> getting the message "Could not find the provided job class >> (org.apache.flink.streaming.examples.wordcount.WordCount) in the user >> lib directory (/opt/flink/usrlib)". As far as I know >> "/opt/flink/usrlib" is the default directory. I am not sure if I have >> to change the property: path /host/path/to/job/artifacts. This is my >> log message of the pod error. >> Do you guys have any idea of what I am missing in my configuration? >> >> Thanks, Felipe >> >> $ kubectl get pods >> NAME READY STATUS RESTARTS AGE >> flink-jobmanager-ftgg9 0/1 CrashLoopBackOff 3 83s >> $ kubectl logs flink-jobmanager-ftgg9 >> Starting Job Manager >> sed: couldn't open temporary file /opt/flink/conf/sedA699Jt: Read-only >> file system >> sed: couldn't open temporary file /opt/flink/conf/sedvZhs0w: Read-only >> file system >> /docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create >> /opt/flink/conf/flink-conf.yaml: Permission denied >> /docker-entrypoint.sh: 91: /docker-entrypoint.sh: cannot create >> /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system >> Starting standalonejob as a console application on host >> flink-jobmanager-ftgg9. >> 2020-09-18 09:15:58,801 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -------------------------------------------------------------------------------- >> 2020-09-18 09:15:58,804 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Preconfiguration: >> 2020-09-18 09:15:58,804 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> >> >> JM_RESOURCE_PARAMS extraction logs: >> jvm_params: -Xmx1073741824 -Xms1073741824 -XX:MaxMetaspaceSize=268435456 >> logs: INFO [] - Loading configuration property: >> jobmanager.rpc.address, flink-jobmanager >> INFO [] - Loading configuration property: taskmanager.numberOfTaskSlots, 4 >> INFO [] - Loading configuration property: blob.server.port, 6124 >> INFO [] - Loading configuration property: jobmanager.rpc.port, 6123 >> INFO [] - Loading configuration property: taskmanager.rpc.port, 6122 >> INFO [] - Loading configuration property: queryable-state.proxy.ports, 6125 >> INFO [] - Loading configuration property: jobmanager.memory.process.size, >> 1600m >> INFO [] - Loading configuration property: >> taskmanager.memory.process.size, 1728m >> INFO [] - Loading configuration property: parallelism.default, 2 >> INFO [] - The derived from fraction jvm overhead memory (160.000mb >> (167772162 bytes)) is less than its min value 192.000mb (201326592 >> bytes), min value will be used instead >> INFO [] - Final Master Memory configuration: >> INFO [] - Total Process Memory: 1.563gb (1677721600 bytes) >> INFO [] - Total Flink Memory: 1.125gb (1207959552 bytes) >> INFO [] - JVM Heap: 1024.000mb (1073741824 bytes) >> INFO [] - Off-heap: 128.000mb (134217728 bytes) >> INFO [] - JVM Metaspace: 256.000mb (268435456 bytes) >> INFO [] - JVM Overhead: 192.000mb (201326592 bytes) >> >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -------------------------------------------------------------------------------- >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Starting StandaloneApplicationClusterEntryPoint (Version: 1.11.0, >> Scala: 2.11, Rev:d04872d, Date:2020-06-29T16:13:14+02:00) >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - OS >> current user: flink >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Current Hadoop/Kerberos user: <no hadoop dependency found> >> 2020-09-18 09:15:58,805 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.262-b10 >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Maximum heap size: 989 MiBytes >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> JAVA_HOME: /usr/local/openjdk-8 >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - No >> Hadoop Dependency available >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - JVM >> Options: >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Xmx1073741824 >> 2020-09-18 09:15:58,806 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Xms1073741824 >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -XX:MaxMetaspaceSize=268435456 >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Dlog.file=/opt/flink/log/flink--standalonejob-0-flink-jobmanager-ftgg9.log >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Program Arguments: >> 2020-09-18 09:15:58,807 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> --configDir >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> /opt/flink/conf >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> --job-classname >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> org.apache.flink.streaming.examples.wordcount.WordCount >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Classpath: >> /opt/flink/lib/flink-csv-1.11.0.jar:/opt/flink/lib/flink-json-1.11.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table-blink_2.11-1.11.0.jar:/opt/flink/lib/flink-table_2.11-1.11.0.jar:/opt/flink/lib/log4j-1.2-api-2.12.1.jar:/opt/flink/lib/log4j-api-2.12.1.jar:/opt/flink/lib/log4j-core-2.12.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.12.1.jar:/opt/flink/lib/flink-dist_2.11-1.11.0.jar::: >> 2020-09-18 09:15:58,808 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> -------------------------------------------------------------------------------- >> 2020-09-18 09:15:58,809 INFO >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Registered UNIX signal handlers for [TERM, HUP, INT] >> 2020-09-18 09:15:58,849 ERROR >> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >> Could not create application program. >> org.apache.flink.util.FlinkException: Could not find the provided job >> class (org.apache.flink.streaming.examples.wordcount.WordCount) in the >> user lib directory (/opt/flink/usrlib). >> at >> org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getJobClassNameOrScanClassPath(ClassPathPackagedProgramRetriever.java:140) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.getPackagedProgram(ClassPathPackagedProgramRetriever.java:123) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:110) >> ~[flink-dist_2.11-1.11.0.jar:1.11.0] >> at >> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:78) >> [flink-dist_2.11-1.11.0.jar:1.11.0] >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/kubernetes.html >> [2] >> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java >> -- >> -- Felipe Gutierrez >> -- skype: felipe.o.gutierrez >> -- https://felipeogutierrez.blogspot.com
