Hi, I'm not aware of any known issues with Hadoop and Flink on Docker.
I also tried what you are doing locally, and it seems to work: flink-jobmanager | 2021-04-15 18:37:48,300 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting StandaloneSessionClusterEntrypoint. flink-jobmanager | 2021-04-15 18:37:48,338 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install default filesystem. flink-jobmanager | 2021-04-15 18:37:48,375 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install security context. flink-jobmanager | 2021-04-15 18:37:48,404 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to flink (auth:SIMPLE) flink-jobmanager | 2021-04-15 18:37:48,408 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-811306162058602256.conf. flink-jobmanager | 2021-04-15 18:37:48,415 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Initializing cluster services. Here's my code: https://gist.github.com/rmetzger/0cf4ba081d685d26478525bf69c7bd39 Hope this helps! On Wed, Apr 14, 2021 at 5:37 PM Flavio Pompermaier <pomperma...@okkam.it> wrote: > Hi everybody, > I'm trying to set up reading from HDFS using docker-compose and Flink > 1.11.3. > If I pass 'env.hadoop.conf.dir' and 'env.yarn.conf.dir' > using FLINK_PROPERTIES (under environment section of the docker-compose > service) I see in the logs the following line: > > "Could not find Hadoop configuration via any of the supported method" > > If I'm not wrong, this means that the HADOOP_CONF_DIR is actually not > generated by the run scripts. > Indeed, If I add HADOOP_CONF_DIR and YARN_CONF_DIR (always under > environment section of the docker-compose service) I don't see that line. > > Is this the expected behavior? > > Below the relevant docker-compose service I use (I've removed the content > of HADOOP_CLASSPATH content because is too long and I didn't report the > taskmanager that is similar): > > flink-jobmanager: > container_name: flink-jobmanager > build: > context: . > dockerfile: Dockerfile.flink > args: > FLINK_VERSION: 1.11.3-scala_2.12-java11 > image: 'flink-test:1.11.3-scala_2.12-java11' > ports: > - "8091:8081" > - "8092:8082" > command: jobmanager > environment: > - | > FLINK_PROPERTIES= > jobmanager.rpc.address: flink-jobmanager > rest.port: 8081 > historyserver.web.port: 8082 > web.upload.dir: /opt/flink > env.hadoop.conf.dir: /opt/hadoop/conf > env.yarn.conf.dir: /opt/hadoop/conf > - | > HADOOP_CLASSPATH=... > - HADOOP_CONF_DIR=/opt/hadoop/conf > - YARN_CONF_DIR=/opt/hadoop/conf > volumes: > - 'flink_shared_folder:/tmp/test' > - 'flink_uploads:/opt/flink/flink-web-upload' > - 'flink_hadoop_conf:/opt/hadoop/conf' > - 'flink_hadoop_libs:/opt/hadoop-3.2.1/share' > > > Thanks in advance for any support, > Flavio >
