Ah, oops and my original email had a typo: > Some python dependencies are not included in the flink distribution tarballs: cloudpickle, py4j and pyflink are in opt/python.
Should read: > Some python dependencies ARE included in the flink distribution tarballs: cloudpickle, py4j and pyflink are in opt/python. On Thu, Jan 26, 2023 at 10:10 AM Andrew Otto <o...@wikimedia.org> wrote: > Let me ask a related question: > > We are building our own base Flink docker image. We will be deploying > both JVM and python apps via flink-kubernetes-operator. > > Is there any reason not to install Flink in this image via `pip install > apache-flink` and use it for JVM apps? > > -Andrew Otto > Wikimedia Foundation > > > > On Tue, Jan 24, 2023 at 4:26 PM Andrew Otto <o...@wikimedia.org> wrote: > >> Hello, >> >> I'm having quite a bit of trouble running pyflink from the default flink >> distribution tarballs. I'd expect the python examples to work as long as >> python is installed, and we've got the distribution. Some python >> dependencies are not included in the flink distribution tarballs: >> cloudpickle, py4j and pyflink are in opt/python. Others are not, e.g. >> protobuf. >> >> Now that I'm looking, I see that the pyflink installation instructions >> <https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/python/installation/> >> are >> to install via pip. >> >> I'm doing this in Docker for use with the flink-kubernetes-operator. In >> the Using Flink Python on Docker >> <https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/standalone/docker/#using-flink-python-on-docker> >> instructions, >> there is a pip3 install apache-flink step. I find this strange, since I'd >> expect the 'FROM flink:1.15.2' part to be sufficient. >> >> By pip installing apache-flink, this docker image will have the flink >> distro installed at /opt/flink and FLINK_HOME set to /opt/flink >> <https://github.com/apache/flink-docker/blob/master/1.16/scala_2.12-java11-ubuntu/Dockerfile>. >> BUT ALSO flink lib jars will be installed at e.g. >> /usr/local/lib/python3.7/dist-packages/pyflink/lib! >> So, by following those instructions, flink is effectively installed twice >> into the docker image. >> >> Am I correct or am I missing something? >> >> Is using pyflink from the flink distribution tarball (without pip) not a >> supported way to use pyflink? >> >> Thanks! >> -Andrew Otto >> Wikimedia Foundation >> >>
