Re: Cannot run pyflink example using Flink CLI

[Biao Geng]

You are right.
It contains the python package `pyflink` and some dependencies like py4j
and cloudpickle but does not contain all relevant dependencies(e.g.
`google.protobuf` as the error log shows, which I also reproduce in my own
machine).

Best,
Biao Geng

Levan Huyen  于2022年10月20日周四 19:53写道：

> Thanks Biao.
>
> May I ask one more question: does the binary package on Apache site (e.g:
> https://archive.apache.org/dist/flink/flink-1.15.2) contain the python
> package `pyflink` and its dependencies? I guess the answer is no.
>
> Thanks and regards,
> Levan Huyen
>
> On Thu, 20 Oct 2022 at 18:13, Biao Geng  wrote:
>
>> Hi Levan,
>> Great to hear that your issue is resolved!
>> For the follow-up question, I am not quite familiar with AWS EMR's
>> configuration for flink but due to the error you attached, it looks like
>> that pyflink may not ship some 'Google' dependencies in the Flink binary
>> zip file and as a result, it will try to find it in your python
>> environment. cc @hxbks...@gmail.com
>> For now, to manage the complex python dependencies, the typical usage of
>> pyflink in multiple node clusters for production is to create your venv and
>> use it in your `flink run` command or in the python code. You can refer to
>> this doc
>> 
>> for details.
>>
>> Best,
>> Biao Geng
>>
>> Levan Huyen  于2022年10月20日周四 14:11写道：
>>
>>> Hi Biao,
>>>
>>> Thanks for your help. That solved my issue. It turned out that in setup1
>>> (in EMR), I got apache-flink installed, but the package (and its
>>> dependencies) are not in the directory `/usr/lib/python3.7/site-packages`
>>> (corresponding to the python binary in `/usr/bin/python3`). For some
>>> reason, the packages are in the current user's location (`~/.local/...)
>>> which Flink did not look at.
>>>
>>> BTW, is there any way to use the pyflink shipped with the Flink binary
>>> zip file that I downloaded from Apache's site? On EMR, such package is
>>> included, and I feel it's awkward to have to install another version using
>>> `pip install`. It will also be confusing about where to add the
>>> dependencies jars.
>>>
>>> Thanks and regards,
>>> Levan Huyen
>>>
>>>
>>> On Thu, 20 Oct 2022 at 02:25, Biao Geng  wrote:
>>>
 Hi Levan,

 For your setup1 & 2, it looks like the python environment is not ready.
 Have you tried python -m pip install apache-flink for the first 2
 setups?
 For your setup3, as you are trying to use `flink run ...` command, it
 will try to connect to a launched flink cluster but I guess you did not
 launch the flink cluster. You can do `start-cluster.sh` first to launch a
 standalone flink cluster and then try the `flink run ...` command.
 For your setup4, the reason why it works well is that it will use the
 default mini cluster to run the pyflink job. So even you haven't started a
 standalone cluster, it can work as well.

 Best,
 Biao Geng

 Levan Huyen  于2022年10月19日周三 17:07写道：

> Hi,
>
> I'm new to PyFlink, and I couldn't run a basic example that shipped
> with Flink.
> This is the command I tried:
>
> ./bin/flink run -py examples/python/datastream/word_count.py
>
> Here below are the results I got with different setups:
>
> 1. On AWS EMR 6.8.0 (Flink 1.15.1):
> *Error: No module named 'google'*I tried with the Flink shipped with
> EMR, or the binary v1.15.1/v1.15.2 downloaded from Flink site. I got that
> same error message in all cases.
>
> Traceback (most recent call last):
>
>   File "/usr/lib64/python3.7/runpy.py", line 193, in
> _run_module_as_main
>
> "__main__", mod_spec)
>
>   File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
>
> exec(code, run_globals)
>
>   File
> "/tmp/pyflink/622f19cc-6616-45ba-b7cb-20a1462c4c3f/a29eb7a4-fff1-4d98-9b38-56b25f97b70a/word_count.py",
> line 134, in 
>
> word_count(known_args.input, known_args.output)
>
>   File
> "/tmp/pyflink/622f19cc-6616-45ba-b7cb-20a1462c4c3f/a29eb7a4-fff1-4d98-9b38-56b25f97b70a/word_count.py",
> line 89, in word_count
>
> ds = ds.flat_map(split) \
>
>   File
> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
> line 333, in flat_map
>
>   File
> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
> line 557, in process
>
>   File
> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/fn_execution/flink_fn_execution_pb2.py",
> line 23, in 
>
> ModuleNotFoundError: No module named 'google'
>
> org.apache.flink.client.program.ProgramAbortException:
> java.lang.RuntimeException: Python process exits with code: 1
>
> 2. On my Mac, without a virtual environment (so `*-pyclientexec
> python3*` is 

Re: Cannot run pyflink example using Flink CLI

[Levan Huyen]

Thanks Biao.

May I ask one more question: does the binary package on Apache site (e.g:
https://archive.apache.org/dist/flink/flink-1.15.2) contain the python
package `pyflink` and its dependencies? I guess the answer is no.

Thanks and regards,
Levan Huyen

On Thu, 20 Oct 2022 at 18:13, Biao Geng  wrote:

> Hi Levan,
> Great to hear that your issue is resolved!
> For the follow-up question, I am not quite familiar with AWS EMR's
> configuration for flink but due to the error you attached, it looks like
> that pyflink may not ship some 'Google' dependencies in the Flink binary
> zip file and as a result, it will try to find it in your python
> environment. cc @hxbks...@gmail.com
> For now, to manage the complex python dependencies, the typical usage of
> pyflink in multiple node clusters for production is to create your venv and
> use it in your `flink run` command or in the python code. You can refer to
> this doc
> 
> for details.
>
> Best,
> Biao Geng
>
> Levan Huyen  于2022年10月20日周四 14:11写道：
>
>> Hi Biao,
>>
>> Thanks for your help. That solved my issue. It turned out that in setup1
>> (in EMR), I got apache-flink installed, but the package (and its
>> dependencies) are not in the directory `/usr/lib/python3.7/site-packages`
>> (corresponding to the python binary in `/usr/bin/python3`). For some
>> reason, the packages are in the current user's location (`~/.local/...)
>> which Flink did not look at.
>>
>> BTW, is there any way to use the pyflink shipped with the Flink binary
>> zip file that I downloaded from Apache's site? On EMR, such package is
>> included, and I feel it's awkward to have to install another version using
>> `pip install`. It will also be confusing about where to add the
>> dependencies jars.
>>
>> Thanks and regards,
>> Levan Huyen
>>
>>
>> On Thu, 20 Oct 2022 at 02:25, Biao Geng  wrote:
>>
>>> Hi Levan,
>>>
>>> For your setup1 & 2, it looks like the python environment is not ready.
>>> Have you tried python -m pip install apache-flink for the first 2
>>> setups?
>>> For your setup3, as you are trying to use `flink run ...` command, it
>>> will try to connect to a launched flink cluster but I guess you did not
>>> launch the flink cluster. You can do `start-cluster.sh` first to launch a
>>> standalone flink cluster and then try the `flink run ...` command.
>>> For your setup4, the reason why it works well is that it will use the
>>> default mini cluster to run the pyflink job. So even you haven't started a
>>> standalone cluster, it can work as well.
>>>
>>> Best,
>>> Biao Geng
>>>
>>> Levan Huyen  于2022年10月19日周三 17:07写道：
>>>
 Hi,

 I'm new to PyFlink, and I couldn't run a basic example that shipped
 with Flink.
 This is the command I tried:

 ./bin/flink run -py examples/python/datastream/word_count.py

 Here below are the results I got with different setups:

 1. On AWS EMR 6.8.0 (Flink 1.15.1):
 *Error: No module named 'google'*I tried with the Flink shipped with
 EMR, or the binary v1.15.1/v1.15.2 downloaded from Flink site. I got that
 same error message in all cases.

 Traceback (most recent call last):

   File "/usr/lib64/python3.7/runpy.py", line 193, in
 _run_module_as_main

 "__main__", mod_spec)

   File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code

 exec(code, run_globals)

   File
 "/tmp/pyflink/622f19cc-6616-45ba-b7cb-20a1462c4c3f/a29eb7a4-fff1-4d98-9b38-56b25f97b70a/word_count.py",
 line 134, in 

 word_count(known_args.input, known_args.output)

   File
 "/tmp/pyflink/622f19cc-6616-45ba-b7cb-20a1462c4c3f/a29eb7a4-fff1-4d98-9b38-56b25f97b70a/word_count.py",
 line 89, in word_count

 ds = ds.flat_map(split) \

   File
 "/usr/lib/flink/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
 line 333, in flat_map

   File
 "/usr/lib/flink/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
 line 557, in process

   File
 "/usr/lib/flink/opt/python/pyflink.zip/pyflink/fn_execution/flink_fn_execution_pb2.py",
 line 23, in 

 ModuleNotFoundError: No module named 'google'

 org.apache.flink.client.program.ProgramAbortException:
 java.lang.RuntimeException: Python process exits with code: 1

 2. On my Mac, without a virtual environment (so `*-pyclientexec
 python3*` is included in the run command): got the same error as with
 EMR, but there's a stdout line from `*print()*` in the Python script

   File
 "/Users/lvh/dev/tmp/flink-1.15.2/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
 line 557, in process

   File "", line 259, in load_module

   File
 

Re: Flink Native K8S RBAC

[Yang Wang]

I have created a ticket[1] to fill the missing part in the native K8s
documentation.

[1]. https://issues.apache.org/jira/browse/FLINK-29705

Best,
Yang

Gyula Fóra  于2022年10月20日周四 13:37写道：

> Hi!
>
> As a reference you can look at how the Flink Kubernetes Operator manages
> RBAC settings:
>
>
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/
>
> https://github.com/apache/flink-kubernetes-operator/blob/main/helm/flink-kubernetes-operator/templates/rbac.yaml
>
> Cheers,
> Gyula
>
> On Wed, Oct 19, 2022 at 9:46 PM Calvin D Souza via user <
> user@flink.apache.org> wrote:
>
>> Hi,
>>
>> I am using custom service account for flink native k8s. These are the
>> rules for the clusterrole I’m using:
>>
>> rules:
>> - apiGroups: [""]
>> resources: ["pods"]
>> verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
>> - apiGroups: [""]
>> resources: ["configmaps"]
>> verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
>> - apiGroups: [""]
>> resources: ["services"]
>> verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
>> - apiGroups: ["apps"]
>> resources: ["deployments"]
>> verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
>> - apiGroups: [""]
>> resources: ["pods/log"]
>> verbs: ["get", "list", "watch"]
>> - apiGroups: ["extensions"]
>> resources: ["deployments"]
>> verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
>>
>>
>> Are there any that I am missing or that are not needed?
>>
>> Thanks,
>> Calvin
>>
>

Re: Tumble Window 会带来反压问题吗？

[yidan zhao]

加大内存segment大小？

macia kk  于2022年10月20日周四 15:31写道：
>
> https://img-beg-sg-1252771144.cos.ap-singapore.myqcloud.com/20221020144100.png
> 看这个图，窗口结束的时候，会产生反压，导致前边的 busy 直接是0，不干活了
>
> https://img-beg-sg-1252771144.cos.ap-singapore.myqcloud.com/20221020152835.png
> 这个是前边在正常消费处理的时候
>
>
>
>
> macia kk  于2022年10月20日周四 14:24写道：
>
> > Hi  yidan
> >
> > 我的的意思是，假设上游 1-10 分钟在处理数据，然后第11分钟就把大批量数据发给 sink，然后上游继续进行 10-20的处理，但是这时候
> > sink 由于数据量大产生了阻塞，造成反压反馈给上游，上游就变慢了。但实际上如果没有反压机制。10-20 的时候，sink
> > 其实可以慢慢写完的。唯一的区别是他发送了一个反压信号，导致上游处理变慢。不知道理解的对不对。
> >
> >
> > 为了要10分钟发送，是因为上游太多数据， 所以我先提前用窗口个聚合一下，目前一秒将近有 800MB 的流量
> >
> >
> >
> > Shammon FY  于2022年10月20日周四 11:48写道：
> >
> >> 如果必须要10分钟，但是key比较分散，感觉这种情况可以增加资源加大一下并发试试，减少每个task发出的数据量
> >>
> >> On Thu, Oct 20, 2022 at 9:49 AM yidan zhao  wrote:
> >>
> >> > 这个描述前后矛盾，写出速度跟不上导致反压，那控制写出速度不是问题更大。不过你不需要考虑这些，因为你控制不了写出速度，只能控制写出时机。
> >> >
> >> > 写出时机是由window的结束时间和watermark决定的，所以如果真要解决，需要控制分窗不要固定整点10分钟。
> >> >
> >> > macia kk  于2022年10月20日周四 00:57写道：
> >> > >
> >> > > 聚合10分钟再输出，到10分钟的时候由于积攒了很多数据，写出速度跟不上，导致反压，然后上游消费就处理变慢了。
> >> > >
> >> > > 如果控制一下写出的速度，让他慢慢写会不会好一些
> >> >
> >>
> >

Re: [DISCUSS] Planning Flink 1.17

[Martijn Visser]

Hi Qingsheng,

I'm definitely interested in participating as a release manager again.

Best regards,

Martijn

On Thu, Oct 20, 2022 at 9:47 AM Qingsheng Ren  wrote:

> Hi everyone,
>
> As we are approaching the official release of Flink 1.16, it’s a good time
> to kick off some discussions and march toward 1.17.
>
> - Release managers
>
> Leonard Xu and I would like to volunteer as release managers for 1.17, and
> it would be great to have someone else working together on this release.
> Please let us know if you have any interest!
>
> - Timeline
>
> Having 1.16 will be released in the next few days and the 4 months release
> cycle after that, we propose to set the feature freezing date on *January
> 17th, 2023* (aligned with our version number 1.17 :-)), so that everyone
> could enjoy the holiday season and Chinese new year.
>
> - What we’ll be focusing
>
> Similar to our previous releases, we’d like to keep an eye on the
> timeline, CI stability, release testing, and any communication and
> coordination across teams and developers. One thing we’d like to mention in
> particular is compatibility, which is a frequent complaint from our
> ecosystem developers and users. We encourage all committers to do an extra
> manual check to see if any public interface is touched before merging a PR.
> We could discuss details in another thread later and update the
> contributing guidelines to list which should be treated as public APIs.
> Please feel free to raise any discussions if you have anything else to
> emphasize specifically.
>
> - Collecting features
>
> We'll create a wiki page under this directory[1] for collecting new
> features targeted in 1.17 as we always did before to give everyone an
> overview and track the progress. Please don’t hesitate to share your ideas
> on the page. In the meantime, we’d like to kindly invite our committers to
> think about and plan what we could deliver to developers and users in this
> release.
>
> Looking forward to working with you all in the coming 1.17 release!
>
> Best regards,
> Qingsheng Ren and Leonard Xu
> Ververica (Alibaba)
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Release+Management+and+Feature+Plan
>

[DISCUSS] Planning Flink 1.17

[Qingsheng Ren]

Hi everyone,

As we are approaching the official release of Flink 1.16, it’s a good time
to kick off some discussions and march toward 1.17.

- Release managers

Leonard Xu and I would like to volunteer as release managers for 1.17, and
it would be great to have someone else working together on this release.
Please let us know if you have any interest!

- Timeline

Having 1.16 will be released in the next few days and the 4 months release
cycle after that, we propose to set the feature freezing date on *January
17th, 2023* (aligned with our version number 1.17 :-)), so that everyone
could enjoy the holiday season and Chinese new year.

- What we’ll be focusing

Similar to our previous releases, we’d like to keep an eye on the timeline,
CI stability, release testing, and any communication and coordination
across teams and developers. One thing we’d like to mention in particular
is compatibility, which is a frequent complaint from our ecosystem
developers and users. We encourage all committers to do an extra manual
check to see if any public interface is touched before merging a PR. We
could discuss details in another thread later and update the contributing
guidelines to list which should be treated as public APIs. Please feel free
to raise any discussions if you have anything else to emphasize
specifically.

- Collecting features

We'll create a wiki page under this directory[1] for collecting new
features targeted in 1.17 as we always did before to give everyone an
overview and track the progress. Please don’t hesitate to share your ideas
on the page. In the meantime, we’d like to kindly invite our committers to
think about and plan what we could deliver to developers and users in this
release.

Looking forward to working with you all in the coming 1.17 release!

Best regards,
Qingsheng Ren and Leonard Xu
Ververica (Alibaba)

[1]
https://cwiki.apache.org/confluence/display/FLINK/Release+Management+and+Feature+Plan

Re: Cannot run pyflink example using Flink CLI

[Biao Geng]

Hi Levan,
Great to hear that your issue is resolved!
For the follow-up question, I am not quite familiar with AWS EMR's
configuration for flink but due to the error you attached, it looks like
that pyflink may not ship some 'Google' dependencies in the Flink binary
zip file and as a result, it will try to find it in your python
environment. cc @hxbks...@gmail.com
For now, to manage the complex python dependencies, the typical usage of
pyflink in multiple node clusters for production is to create your venv and
use it in your `flink run` command or in the python code. You can refer to
this doc

for details.

Best,
Biao Geng

Levan Huyen  于2022年10月20日周四 14:11写道：

> Hi Biao,
>
> Thanks for your help. That solved my issue. It turned out that in setup1
> (in EMR), I got apache-flink installed, but the package (and its
> dependencies) are not in the directory `/usr/lib/python3.7/site-packages`
> (corresponding to the python binary in `/usr/bin/python3`). For some
> reason, the packages are in the current user's location (`~/.local/...)
> which Flink did not look at.
>
> BTW, is there any way to use the pyflink shipped with the Flink binary zip
> file that I downloaded from Apache's site? On EMR, such package is
> included, and I feel it's awkward to have to install another version using
> `pip install`. It will also be confusing about where to add the
> dependencies jars.
>
> Thanks and regards,
> Levan Huyen
>
>
> On Thu, 20 Oct 2022 at 02:25, Biao Geng  wrote:
>
>> Hi Levan,
>>
>> For your setup1 & 2, it looks like the python environment is not ready.
>> Have you tried python -m pip install apache-flink for the first 2 setups?
>> For your setup3, as you are trying to use `flink run ...` command, it
>> will try to connect to a launched flink cluster but I guess you did not
>> launch the flink cluster. You can do `start-cluster.sh` first to launch a
>> standalone flink cluster and then try the `flink run ...` command.
>> For your setup4, the reason why it works well is that it will use the
>> default mini cluster to run the pyflink job. So even you haven't started a
>> standalone cluster, it can work as well.
>>
>> Best,
>> Biao Geng
>>
>> Levan Huyen  于2022年10月19日周三 17:07写道：
>>
>>> Hi,
>>>
>>> I'm new to PyFlink, and I couldn't run a basic example that shipped with
>>> Flink.
>>> This is the command I tried:
>>>
>>> ./bin/flink run -py examples/python/datastream/word_count.py
>>>
>>> Here below are the results I got with different setups:
>>>
>>> 1. On AWS EMR 6.8.0 (Flink 1.15.1):
>>> *Error: No module named 'google'*I tried with the Flink shipped with
>>> EMR, or the binary v1.15.1/v1.15.2 downloaded from Flink site. I got that
>>> same error message in all cases.
>>>
>>> Traceback (most recent call last):
>>>
>>>   File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
>>>
>>> "__main__", mod_spec)
>>>
>>>   File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
>>>
>>> exec(code, run_globals)
>>>
>>>   File
>>> "/tmp/pyflink/622f19cc-6616-45ba-b7cb-20a1462c4c3f/a29eb7a4-fff1-4d98-9b38-56b25f97b70a/word_count.py",
>>> line 134, in 
>>>
>>> word_count(known_args.input, known_args.output)
>>>
>>>   File
>>> "/tmp/pyflink/622f19cc-6616-45ba-b7cb-20a1462c4c3f/a29eb7a4-fff1-4d98-9b38-56b25f97b70a/word_count.py",
>>> line 89, in word_count
>>>
>>> ds = ds.flat_map(split) \
>>>
>>>   File
>>> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
>>> line 333, in flat_map
>>>
>>>   File
>>> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
>>> line 557, in process
>>>
>>>   File
>>> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/fn_execution/flink_fn_execution_pb2.py",
>>> line 23, in 
>>>
>>> ModuleNotFoundError: No module named 'google'
>>>
>>> org.apache.flink.client.program.ProgramAbortException:
>>> java.lang.RuntimeException: Python process exits with code: 1
>>>
>>> 2. On my Mac, without a virtual environment (so `*-pyclientexec python3*`
>>> is included in the run command): got the same error as with EMR, but
>>> there's a stdout line from `*print()*` in the Python script
>>>
>>>   File
>>> "/Users/lvh/dev/tmp/flink-1.15.2/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
>>> line 557, in process
>>>
>>>   File "", line 259, in load_module
>>>
>>>   File
>>> "/Users/lvh/dev/tmp/flink-1.15.2/opt/python/pyflink.zip/pyflink/fn_execution/flink_fn_execution_pb2.py",
>>> line 23, in 
>>>
>>> ModuleNotFoundError: No module named 'google'
>>>
>>> Executing word_count example with default input data set.
>>>
>>> Use --input to specify file input.
>>>
>>> org.apache.flink.client.program.ProgramAbortException:
>>> java.lang.RuntimeException: Python process exits with code: 1
>>>
>>> 3. On my Mac, with a virtual environment and Python package `
>>> *apache-flink`* installed: Flink tried to connect to 

Re: Tumble Window 会带来反压问题吗？

[macia kk]

Hi  yidan

我的的意思是，假设上游 1-10 分钟在处理数据，然后第11分钟就把大批量数据发给 sink，然后上游继续进行 10-20的处理，但是这时候 sink
由于数据量大产生了阻塞，造成反压反馈给上游，上游就变慢了。但实际上如果没有反压机制。10-20 的时候，sink
其实可以慢慢写完的。唯一的区别是他发送了一个反压信号，导致上游处理变慢。不知道理解的对不对。


为了要10分钟发送，是因为上游太多数据， 所以我先提前用窗口个聚合一下，目前一秒将近有 800MB 的流量



Shammon FY  于2022年10月20日周四 11:48写道：

> 如果必须要10分钟，但是key比较分散，感觉这种情况可以增加资源加大一下并发试试，减少每个task发出的数据量
>
> On Thu, Oct 20, 2022 at 9:49 AM yidan zhao  wrote:
>
> > 这个描述前后矛盾，写出速度跟不上导致反压，那控制写出速度不是问题更大。不过你不需要考虑这些，因为你控制不了写出速度，只能控制写出时机。
> >
> > 写出时机是由window的结束时间和watermark决定的，所以如果真要解决，需要控制分窗不要固定整点10分钟。
> >
> > macia kk  于2022年10月20日周四 00:57写道：
> > >
> > > 聚合10分钟再输出，到10分钟的时候由于积攒了很多数据，写出速度跟不上，导致反压，然后上游消费就处理变慢了。
> > >
> > > 如果控制一下写出的速度，让他慢慢写会不会好一些
> >
>

Re: Cannot run pyflink example using Flink CLI

[Levan Huyen]

Hi Biao,

Thanks for your help. That solved my issue. It turned out that in setup1
(in EMR), I got apache-flink installed, but the package (and its
dependencies) are not in the directory `/usr/lib/python3.7/site-packages`
(corresponding to the python binary in `/usr/bin/python3`). For some
reason, the packages are in the current user's location (`~/.local/...)
which Flink did not look at.

BTW, is there any way to use the pyflink shipped with the Flink binary zip
file that I downloaded from Apache's site? On EMR, such package is
included, and I feel it's awkward to have to install another version using
`pip install`. It will also be confusing about where to add the
dependencies jars.

Thanks and regards,
Levan Huyen


On Thu, 20 Oct 2022 at 02:25, Biao Geng  wrote:

> Hi Levan,
>
> For your setup1 & 2, it looks like the python environment is not ready.
> Have you tried python -m pip install apache-flink for the first 2 setups?
> For your setup3, as you are trying to use `flink run ...` command, it will
> try to connect to a launched flink cluster but I guess you did not launch
> the flink cluster. You can do `start-cluster.sh` first to launch a
> standalone flink cluster and then try the `flink run ...` command.
> For your setup4, the reason why it works well is that it will use the
> default mini cluster to run the pyflink job. So even you haven't started a
> standalone cluster, it can work as well.
>
> Best,
> Biao Geng
>
> Levan Huyen  于2022年10月19日周三 17:07写道：
>
>> Hi,
>>
>> I'm new to PyFlink, and I couldn't run a basic example that shipped with
>> Flink.
>> This is the command I tried:
>>
>> ./bin/flink run -py examples/python/datastream/word_count.py
>>
>> Here below are the results I got with different setups:
>>
>> 1. On AWS EMR 6.8.0 (Flink 1.15.1):
>> *Error: No module named 'google'*I tried with the Flink shipped with
>> EMR, or the binary v1.15.1/v1.15.2 downloaded from Flink site. I got that
>> same error message in all cases.
>>
>> Traceback (most recent call last):
>>
>>   File "/usr/lib64/python3.7/runpy.py", line 193, in _run_module_as_main
>>
>> "__main__", mod_spec)
>>
>>   File "/usr/lib64/python3.7/runpy.py", line 85, in _run_code
>>
>> exec(code, run_globals)
>>
>>   File
>> "/tmp/pyflink/622f19cc-6616-45ba-b7cb-20a1462c4c3f/a29eb7a4-fff1-4d98-9b38-56b25f97b70a/word_count.py",
>> line 134, in 
>>
>> word_count(known_args.input, known_args.output)
>>
>>   File
>> "/tmp/pyflink/622f19cc-6616-45ba-b7cb-20a1462c4c3f/a29eb7a4-fff1-4d98-9b38-56b25f97b70a/word_count.py",
>> line 89, in word_count
>>
>> ds = ds.flat_map(split) \
>>
>>   File
>> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
>> line 333, in flat_map
>>
>>   File
>> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
>> line 557, in process
>>
>>   File
>> "/usr/lib/flink/opt/python/pyflink.zip/pyflink/fn_execution/flink_fn_execution_pb2.py",
>> line 23, in 
>>
>> ModuleNotFoundError: No module named 'google'
>>
>> org.apache.flink.client.program.ProgramAbortException:
>> java.lang.RuntimeException: Python process exits with code: 1
>>
>> 2. On my Mac, without a virtual environment (so `*-pyclientexec python3*`
>> is included in the run command): got the same error as with EMR, but
>> there's a stdout line from `*print()*` in the Python script
>>
>>   File
>> "/Users/lvh/dev/tmp/flink-1.15.2/opt/python/pyflink.zip/pyflink/datastream/data_stream.py",
>> line 557, in process
>>
>>   File "", line 259, in load_module
>>
>>   File
>> "/Users/lvh/dev/tmp/flink-1.15.2/opt/python/pyflink.zip/pyflink/fn_execution/flink_fn_execution_pb2.py",
>> line 23, in 
>>
>> ModuleNotFoundError: No module named 'google'
>>
>> Executing word_count example with default input data set.
>>
>> Use --input to specify file input.
>>
>> org.apache.flink.client.program.ProgramAbortException:
>> java.lang.RuntimeException: Python process exits with code: 1
>>
>> 3. On my Mac, with a virtual environment and Python package `
>> *apache-flink`* installed: Flink tried to connect to localhost:8081 (I
>> don't know why), and failed with 'connection refused'.
>>
>> Caused by: org.apache.flink.util.concurrent.FutureUtils$RetryException:
>> Could not complete the operation. Number of retries has been exhausted.
>>
>> at
>> org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:395)
>>
>> ... 21 more
>>
>> Caused by: java.util.concurrent.CompletionException:
>> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException:
>> Connection refused: localhost/127.0.0.1:8081
>>
>>  4. If I run that same example job using Python: `*python word_count.py*`
>> then it runs well.
>>
>> I tried with both v1.15.2 and 1.15.1, Python 3.7 & 3.8, and got the same
>> result.
>>
>> Could someone please help?
>>
>> Thanks.
>>
>