Ok - it works in docker when I pick up a file from inside docker system -
e.g - for the following when both the input and output files are
inside docker:
inputFile=/opt/flink/conf/flink-conf.yaml --output=/tmp/counts!
It does not work when using a the file from the host system volume mapped
to docker as in - any pointer?
volumes:
input_data:
name: input_volume
services:
job-cluster:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink-job}
ports:
- "8081:8081"
volumes:
- input_data:~/input/pom.xml:ro
command: job-cluster --job-classname ${FLINK_JOB}
-Djobmanager.rpc.address=job-cluster
-Dparallelism.default=${DEFAULT_PARALLELISM:-1} ${SAVEPOINT_OPTIONS}
${FLINK_JOB_ARGUMENTS}
taskmanager:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink-job}
volumes:
- input_data:/home/asaha/input/pom.xml:ro
command: task-manager -Djobmanager.rpc.address=job-cluster
scale: ${DEFAULT_PARALLELISM:-1}
On Tue, Jul 7, 2020 at 4:27 PM Robert Bradshaw <[email protected]> wrote:
> Does it work when you write to a distributed filesystem? (One issue
> with Docker is that the manager and each of their workers have their
> own local filesystem.)
>
> On Tue, Jul 7, 2020 at 2:17 PM Avijit Saha <[email protected]> wrote:
> >
> > While trying to run the Beam WordCount example on Flink runner using
> Job Manager and Taskmanager docker images using the input pom.xml file, I
> can see the task manager is creating the following files but no output
> inside the counts-0000 file as expected - Any pointer what might be wrong
> here?
> >
> > Thanks
> >
> > bash-4.4$ ls -lrt /tmp
> > total 4
> > drwxr-xr-x 1 flink flink 2 Jul 7 21:15 hsperfdata_flink
> > -rw-r--r-- 1 flink flink 1179 Jul 7 21:15
> jaas-3210126497662256938.conf
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> flink-io-5922e9cc-91ee-4ae3-bee3-90006db36aaa
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> blobStore-1f336cf6-8885-4aff-bd36-ddf38f11c0a0
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> blobStore-13b09c7f-e371-430f-90aa-802aec50735e
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> flink-dist-cache-6330c1d8-3ab8-4baa-812b-485d6d850c1d
> > -rw-r--r-- 1 flink flink 0 Jul 7 21:15
> counts-00000-of-00001
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> flink-netty-shuffle-69803f1d-c641-41b0-a090-7dc542ed98f4
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15 localState
>
