potiuk commented on a change in pull request #9378:
URL: https://github.com/apache/airflow/pull/9378#discussion_r442266574
##########
File path: Dockerfile.ci
##########
@@ -315,6 +315,9 @@ WORKDIR ${AIRFLOW_SOURCES}
ENV PATH="${HOME}:${PATH}"
+# Needed to stop Gunicorn from crashing when /tmp is now mounted from host
+ENV GUNICORN_CMD_ARGS="--worker-tmp-dir /opt/airflow/tmp"
Review comment:
@ashb - the root cause was the removal of Java from the CI image.
@mik-laj spend quite some time after it was removed to make it works and
that involved mapping /tmp from host to both Breeze image and to Java run via
the docker image and forwarded docker socket.
This is all because we have automated system tests in Google Provder.
Previously we just run java in Breeze when we needed it and we used java to
prepare packages to be sent to DataFlow (it needs to package java code and some
jars and submit them to DataFlow)
Previously when java was embedded in the image it "just worked". But when
you removed it from the image (which was a good thing BTW.) we needed to find
another solution. We used the approach that I piloted with the gcloud CLI -
where we have script that runs external tool via Docker run in the docker
engine via the forwarded socket. The problem with that was that we had to pass
some of the temporary files we created in Python unit tests to the java command
line (that was also similar problem for other code ours where we uploaded
temporary files to GCS via gcloud command line).
The easiest way was to map ${AIRFLOW_SOURCES}/tmp to both - the CI airflow
container and to all the tools we run via the forwarded socket. This way the
/tmo (we also did it for /opt/airflow) is shared between the CI container and
the gcloud, java, terraform .and all the other tools which we now do not have
to embed into the CI image - they are all pulled on demand via docker.
Since we switched to run those tools "docker via
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
