date:20230130

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Yun Tang

Thanks Yuanfei for driving the frocksdb release!

Best
Yun Tang

From: Yuan Mei 
Sent: Tuesday, January 31, 2023 15:09
To: Jing Ge 
Cc: Yanfei Lei ; d...@flink.apache.org 
; user ; user-zh@flink.apache.org 

Subject: Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Thanks Yanfei for driving the release!

Best
Yuan

On Mon, Jan 30, 2023 at 8:46 PM Jing Ge via user 
mailto:u...@flink.apache.org>> wrote:
Hi Yanfei,

Thanks for your effort. Looking forward to checking it.

Best regards,
Jing

On Mon, Jan 30, 2023 at 1:42 PM Yanfei Lei 
mailto:fredia...@gmail.com>> wrote:
It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.

Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
jar, and the FRocksDB in Flink 1.17 would be updated to
6.20.3-ververica-2.0.

Release highlights:
- [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
- [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
- Avoid expensive ToString() call when not in debug[3].
- [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].

Maven artifacts for FRocksDB can be found at:
https://mvnrepository.com/artifact/com.ververica/frocksdbjni

We would like to thank all efforts from the Apache Flink community
that made this release possible!

[1] https://issues.apache.org/jira/browse/FLINK-30457
[2] https://issues.apache.org/jira/browse/FLINK-30321
[3] https://github.com/ververica/frocksdb/pull/55
[4] https://issues.apache.org/jira/browse/FLINK-24932

Best regards,
Yanfei
Ververica(Alibaba)

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Yun Tang

Thanks Yuanfei for driving the frocksdb release!

Best
Yun Tang

From: Yuan Mei 
Sent: Tuesday, January 31, 2023 15:09
To: Jing Ge 
Cc: Yanfei Lei ; d...@flink.apache.org 
; user ; user...@flink.apache.org 

Subject: Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Thanks Yanfei for driving the release!

Best
Yuan

On Mon, Jan 30, 2023 at 8:46 PM Jing Ge via user 
mailto:user@flink.apache.org>> wrote:
Hi Yanfei,

Thanks for your effort. Looking forward to checking it.

Best regards,
Jing

On Mon, Jan 30, 2023 at 1:42 PM Yanfei Lei 
mailto:fredia...@gmail.com>> wrote:
It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.

Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
jar, and the FRocksDB in Flink 1.17 would be updated to
6.20.3-ververica-2.0.

Release highlights:
- [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
- [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
- Avoid expensive ToString() call when not in debug[3].
- [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].

Maven artifacts for FRocksDB can be found at:
https://mvnrepository.com/artifact/com.ververica/frocksdbjni

We would like to thank all efforts from the Apache Flink community
that made this release possible!

[1] https://issues.apache.org/jira/browse/FLINK-30457
[2] https://issues.apache.org/jira/browse/FLINK-30321
[3] https://github.com/ververica/frocksdb/pull/55
[4] https://issues.apache.org/jira/browse/FLINK-24932

Best regards,
Yanfei
Ververica(Alibaba)

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

2023-01-30 Thread Yang Wang

I assume you are using the standalone mode. Right?

For the native K8s mode, the leader address should be
*akka.tcp://flink@JM_POD_IP:6123/user/rpc/dispatcher_1
*when HA enabled.


Best,
Yang

Anton Ippolitov via user  于2023年1月31日周二 00:21写道：

> This is actually what I'm already doing, I'm only setting high-availability:
> kubernetes myself. The other values are either defaults or set by the
> Operator:
> - jobmanager.rpc.port: 6123 is the default value (docs
> 
> )
> -  high-availability.jobmanager.port: 6123 is set by the Operator here
> 
>
> - jobmanager.rpc.address: SERVICE-NAME-HERE.NAMESPACE-HERE is set by the
> Operator here
> 
>  (the
> actual code which gets executed is here
> 
> )
>
>  Looking at what the Lyft Operator is doing here
> ,
>  I thought
> this would be a common issue but since you've never seen this error before,
> not sure what to do 樂
>
> On Fri, Jan 27, 2023 at 10:59 PM Gyula Fóra  wrote:
>
>> We never encountered this problem before but also we don't configure
>> those settings.
>> Can you simply try:
>>
>> high-availability: kubernetes
>>
>> And remove the other configs? I think that can only cause problems and
>> should not achieve anything :)
>>
>> Gyula
>>
>> On Fri, Jan 27, 2023 at 6:44 PM Anton Ippolitov via user <
>> user@flink.apache.org> wrote:
>>
>>> Hi everyone,
>>>
>>> I've been experimenting with Kubernetes HA and the Kubernetes Operator
>>> and ran into the following issue which is happening regularly on
>>> TaskManagers with Flink 1.16:
>>>
>>> Error while retrieving the leader gateway. Retrying to connect to 
>>> akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE:6123/user/rpc/dispatcher_1.
>>> org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not 
>>> complete the operation. Number of retries has been exhausted.
>>>
>>> (The whole stacktrace is quite long, I put it in a Github Gist here
>>> .
>>> Note that I put placeholder values for the Kubernetes Service name and the
>>> Namespace name)
>>>
>>> The job configuration has the following values which should be relevant:
>>> high-availability: kubernetes
>>> high-availability.jobmanager.port: 6123
>>> jobmanager.rpc.address: SERVICE-NAME-HERE.NAMESPACE-HERE
>>> jobmanager.rpc.port: 6123
>>>
>>> Looking a bit more into the logs, I can see that the Akka Actor System
>>> is started with an external address pointing to the Kubernetes Service
>>> defined by jobmanager.rpc.address:
>>> Trying to start actor system, external
>>> address SERVICE-NAME-HERE.NAMESPACE-HERE:6123, bind address 0.0.0.0:6123
>>> .
>>> Actor system started at akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE
>>> :6123
>>>
>>> (I believe the external address for the Akka Actor System is set to
>>> jobmanager.rpc.address from this place
>>> 
>>> in the code but I might be wrong)
>>>
>>> I can also see these logs for the Dispatcher RPC endpoint:
>>> Starting RPC endpoint for
>>> org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
>>> akka://flink/user/rpc/dispatcher_1 .
>>> Successfully wrote leader information
>>> LeaderInformation{leaderSessionID='8fd2bda3-1775-4b51-bf63-8da385247a18',
>>> leaderAddress=akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE:6123/user/rpc/dispatcher_1}
>>> for leader dispatcher into the config map JOB-NAME-HERE-cluster-config-map.
>>>
>>> I confirmed that the HA ConfigMap contains an address which also uses
>>> the Kubernetes Service defined by jobmanager.rpc.address:
>>> $ kubectl get cm JOB-NAME-HERE-cluster-config-map -o json | jq -r
>>> '.data["org.apache.flink.k8s.leader.dispatcher"]'
>>>
>>> ce33b6d4-a55f-475c-9b6e-b21d25c8e6b5,akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE
>>> :6123/user/rpc/dispatcher_1
>>>
>>> When looking at the code of the Operator and Flink itself, I can see
>>> that jobmanager.rpc.address is set automatically by the
>>>

Flink SQL 如何优化以及处理反压

2023-01-30 Thread lxk

Flink版本：1.16.0
目前在使用Flink SQL进行多流关联，并写入Clickhouse中
具体代码如下：
select \
header.id as id, \
LAST_VALUE(header.order_status), \
LAST_VALUE(header.customer_id), \
LAST_VALUE(header.shop_id), \
LAST_VALUE(header.parent_order_id), \
LAST_VALUE(header.order_at), \
LAST_VALUE(header.pay_at), \
LAST_VALUE(header.channel_id), \
LAST_VALUE(header.root_order_id), \
LAST_VALUE(header.last_updated_at), \
item.id as item_id, \
LAST_VALUE(item.order_id) as order_id, \
LAST_VALUE(item.row_num), \
LAST_VALUE(item.goods_id), \
LAST_VALUE(item.s_sku_code), \
LAST_VALUE(item.qty), \
LAST_VALUE(item.p_paid_sub_amt), \
LAST_VALUE(item.p_sp_sub_amt), \
LAST_VALUE(item.bom_type), \
LAST_VALUE(item.last_updated_at) as item_last_updated_at, \
LAST_VALUE(item.display_qty), \
LAST_VALUE(delivery.del_type), \
LAST_VALUE(delivery.time_slot_type), \
LAST_VALUE(delivery.time_slot_date), \
LAST_VALUE(delivery.time_slot_time_from), \
LAST_VALUE(delivery.time_slot_time_to), \
LAST_VALUE(delivery.sku_delivery_type), \
LAST_VALUE(delivery.last_updated_at) as del_last_updated_at, \
LAST_VALUE(promotion.id) as promo_id, \
LAST_VALUE(promotion.order_item_id), \
LAST_VALUE(promotion.p_promo_amt), \
LAST_VALUE(promotion.promotion_category), \
LAST_VALUE(promotion.promo_type), \
LAST_VALUE(promotion.promo_sub_type), \
LAST_VALUE(promotion.last_updated_at) as promo_last_updated_at, \
LAST_VALUE(promotion.promotion_cost) \
from \
  item \
  join \
  header  \
  on item.order_id = header.id \
  left join \
  delivery \
  on item.order_id = delivery.order_id \
  left join \
  promotion \
  on item.id =promotion.order_item_id \
  group by header.id,item.id
在Flink WEB UI 上发现程序反压很严重，而且时不时挂掉：
https://pic.imgdb.cn/item/63d8bebbface21e9ef3c92fe.jpg
参考了京东的一篇文章https://flink-learning.org.cn/article/detail/1e86b8b38faaeefd5ed7f70858aa40bc
 ，对相关参数做了调整,但是发现有些功能在Flink 1.16中已经做了相关优化了，同时加了这些参数之后对程序没有起到任何优化的作用。


conf.setString("table.exec.mini-batch.enabled", "true");
conf.setString("table.exec.mini-batch.allow-latency", "15 s");
conf.setString("table.exec.mini-batch.size", "5000");
conf.setString("table.exec.state.ttl", "86400 s");
conf.setString("table.exec.disabled-operators", "NestedLoopJoin");
conf.setString("table.optimizer.join.broadcast-threshold", "-1");
conf.setString("table.optimizer.multiple-input-enabled", "true");
conf.setString("table.exec.shuffle-mode", "POINTWISE_EDGES_PIPELINED");
conf.setString("taskmanager.network.sort-shuffle.min-parallelism", "8");
想请教下，针对Flink SQL如何处理反压，同时有什么其他的优化手段？

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Yuan Mei

Thanks Yanfei for driving the release!

Best
Yuan

On Mon, Jan 30, 2023 at 8:46 PM Jing Ge via user 
wrote:

> Hi Yanfei,
>
> Thanks for your effort. Looking forward to checking it.
>
> Best regards,
> Jing
>
> On Mon, Jan 30, 2023 at 1:42 PM Yanfei Lei  wrote:
>
>> It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.
>>
>> Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
>> MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
>> jar, and the FRocksDB in Flink 1.17 would be updated to
>> 6.20.3-ververica-2.0.
>>
>> Release highlights:
>> - [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
>> - [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
>> - Avoid expensive ToString() call when not in debug[3].
>> - [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].
>>
>> Maven artifacts for FRocksDB can be found at:
>> https://mvnrepository.com/artifact/com.ververica/frocksdbjni
>>
>> We would like to thank all efforts from the Apache Flink community
>> that made this release possible!
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-30457
>> [2] https://issues.apache.org/jira/browse/FLINK-30321
>> [3] https://github.com/ververica/frocksdb/pull/55
>> [4] https://issues.apache.org/jira/browse/FLINK-24932
>>
>> Best regards,
>> Yanfei
>> Ververica(Alibaba)
>>
>

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Yuan Mei

Thanks Yanfei for driving the release!

Best
Yuan

On Mon, Jan 30, 2023 at 8:46 PM Jing Ge via user 
wrote:

> Hi Yanfei,
>
> Thanks for your effort. Looking forward to checking it.
>
> Best regards,
> Jing
>
> On Mon, Jan 30, 2023 at 1:42 PM Yanfei Lei  wrote:
>
>> It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.
>>
>> Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
>> MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
>> jar, and the FRocksDB in Flink 1.17 would be updated to
>> 6.20.3-ververica-2.0.
>>
>> Release highlights:
>> - [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
>> - [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
>> - Avoid expensive ToString() call when not in debug[3].
>> - [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].
>>
>> Maven artifacts for FRocksDB can be found at:
>> https://mvnrepository.com/artifact/com.ververica/frocksdbjni
>>
>> We would like to thank all efforts from the Apache Flink community
>> that made this release possible!
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-30457
>> [2] https://issues.apache.org/jira/browse/FLINK-30321
>> [3] https://github.com/ververica/frocksdb/pull/55
>> [4] https://issues.apache.org/jira/browse/FLINK-24932
>>
>> Best regards,
>> Yanfei
>> Ververica(Alibaba)
>>
>

Re: 退订

2023-01-30 Thread weijie guo

Hello,

退订请发邮件到user-zh-unsubscr...@flink.apache.org

Best regards,

Weijie


唐凯  于2023年1月19日周四 15:54写道：

> 退订
>
>
>
>
> 唐凯
> mrdon...@foxmail.com
>
>
>
>

Re: 任务本地运行正常，提交到集群报错 - 图片挂掉，文字贴一下报错信息，非常抱歉打扰

2023-01-30 Thread weijie guo

ping 127.0.0.1:33271 可以ping通吗

Best regards,

Weijie


yidan zhao  于2023年1月12日周四 17:48写道：

> 看报错 Could not connect to BlobServer at address
> localhost/127.0.0.1:33271，你本地的配置是不是不对。提交到什么模式部署的集群，配置是否配对了。
>
> WD.Z  于2023年1月10日周二 10:56写道：
> >
> >
> 任务在webui点击submit时报错，看起来是从JM提交到TM时报错，服务器防火墙已关闭，资源足够，还没有安装hadoop，但以standalone模式启动，看了下文档是不需要hadoop?
> 报错中的Caused by列表如下：
> >
> >
> > 2023-01-10 09:46:14,627 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor
> [] - Job e343bc906ea6889d34d9472d40d4f8ff is submitted.
> > 2023-01-10 09:46:14,627 INFO
> org.apache.flink.client.deployment.application.executors.EmbeddedExecutor
> [] - Submitting Job with JobId=e343bc906ea6889d34d9472d40d4f8ff.
> > 2023-01-10 09:46:14,629 WARN
> org.apache.flink.client.deployment.application.DetachedApplicationRunner []
> - Could not execute application:
> > org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error: org.apache.flink.util.FlinkException: Failed to
> execute job 'Flink Streaming Job'.
> >
> >
> > Caused by: java.lang.RuntimeException:
> org.apache.flink.util.FlinkException: Failed to execute job 'Flink
> Streaming Job'.
> >
> >
> > Caused by: org.apache.flink.util.FlinkException: Failed to execute job
> 'Flink Streaming Job'.
> >
> >
> > Caused by: org.apache.flink.util.FlinkException: Failed to execute job
> 'Flink Streaming Job'.
> >
> >
> > Caused by: org.apache.flink.util.FlinkException: Could not upload job
> files.
> >
> >
> > Caused by: java.io.IOException: Could not connect to BlobServer at
> address localhost/127.0.0.1:33271
> >
> >
> > Caused by: java.net.ConnectException: 拒绝连接 (Connection refused)
>

Re: Using pyflink from flink distribution

2023-01-30 Thread Dian Fu

>> What is the reason for including
opt/python/{pyflink.zip,cloudpickle.zip,py4j.zip} in the base distribution
then?  Oh, a guess: to make it easier for TaskManagers to run
pyflink without having pyflink installed themselves?  Somehow I'd guess
this wouldn't work tho; I'd assume TaskManagers would also need some python
transitive dependencies, e.g. google protobuf.

It has some historical reasons. In the first version (1.9.x) which has not
provided Python UDF support, it's not necessary to install PyFlink in the
nodes of TaskManagers. Since 1.10 which supports Python UDF, users have to
install PyFlink in the nodes of TaskManager as there are many transitive
dependencies, e.g. Apache Beam、protobuf、pandas, etc. However, we have not
removed these packages as they are still useful for client node which is
responsible for compiling jobs(it's not necessary to install PyFlink in the
client node).

>> Since we're building our own Docker image, I'm going the other way
around: just install pyflink, and symlink /opt/flink ->
/usr/lib/python3.7/dist-packages/pyflink.  So far so good, but I'm worried
that something will be fishy when trying to run JVM apps via pyflink.

Good idea! It contains all the things necessary needed to run JVM apps in
the PyFlink package and so I think you could just try this way.

Regards,
Dian

On Mon, Jan 30, 2023 at 9:58 PM Andrew Otto  wrote:

> Thanks Dian!
>
> > >> Is using pyflink from the flink distribution tarball (without pip)
> not a supported way to use pyflink?
> > You are right.
>
> What is the reason for including
> opt/python/{pyflink.zip,cloudpickle.zip,py4j.zip} in the base
> distribution then?  Oh, a guess: to make it easier for TaskManagers to run
> pyflink without having pyflink installed themselves?  Somehow I'd guess
> this wouldn't work tho; I'd assume TaskManagers would also need some python
> transitive dependencies, e.g. google protobuf.
>
> > you could remove the JAR packages located under
> /usr/local/lib/python3.7/dist-packages/pyflink/lib manually after `pip
> install apache-flink`
>
> Since we're building our own Docker image, I'm going the other way around:
> just install pyflink, and symlink /opt/flink ->
> /usr/lib/python3.7/dist-packages/pyflink.  So far so good, but I'm worried
> that something will be fishy when trying to run JVM apps via pyflink.
>
> -Ao
>
>
>
> On Sun, Jan 29, 2023 at 1:43 AM Dian Fu  wrote:
>
>> Hi Andrew,
>>
>> >> By pip installing apache-flink, this docker image will have the flink
>> distro installed at /opt/flink and FLINK_HOME set to /opt/flink
>> .
>> BUT ALSO flink lib jars will be installed at e.g.
>> /usr/local/lib/python3.7/dist-packages/pyflink/lib!
>> So, by following those instructions, flink is effectively installed twice
>> into the docker image.
>>
>> Yes, your understanding is correct. The base image `flink:1.15.2`
>> doesn't include PyFlink and so you need to build a custom image if you want
>> to use PyFlink. Regarding to the jar packages which are installed twice,
>> you could remove the JAR packages located under
>> /usr/local/lib/python3.7/dist-packages/pyflink/lib manually after `pip
>> install apache-flink`. It will use the JAR packages located under
>> $FLINK_HOME/lib.
>>
>> >> Is using pyflink from the flink distribution tarball (without pip) not
>> a supported way to use pyflink?
>> You are right.
>>
>> Regards,
>> Dian
>>
>>
>> On Thu, Jan 26, 2023 at 11:12 PM Andrew Otto  wrote:
>>
>>> Ah, oops and my original email had a typo:
>>> > Some python dependencies are not included in the flink distribution
>>> tarballs: cloudpickle, py4j and pyflink are in opt/python.
>>>
>>> Should read:
>>> > Some python dependencies ARE included in the flink distribution
>>> tarballs: cloudpickle, py4j and pyflink are in opt/python.
>>>
>>> On Thu, Jan 26, 2023 at 10:10 AM Andrew Otto  wrote:
>>>
 Let me ask a related question:

 We are building our own base Flink docker image.  We will be deploying
 both JVM and python apps via flink-kubernetes-operator.

 Is there any reason not to install Flink in this image via `pip install
 apache-flink` and use it for JVM apps?

 -Andrew Otto
  Wikimedia Foundation



 On Tue, Jan 24, 2023 at 4:26 PM Andrew Otto  wrote:

> Hello,
>
> I'm having quite a bit of trouble running pyflink from the default
> flink distribution tarballs.  I'd expect the python examples to work as
> long as python is installed, and we've got the distribution.  Some python
> dependencies are not included in the flink distribution tarballs:
> cloudpickle, py4j and pyflink are in opt/python.  Others are not, e.g.
> protobuf.
>
> Now that I'm looking, I see that the pyflink installation instructions
> 
>  are
> to install via

Processing watermarks in a broadcast connected stream

2023-01-30 Thread Sajjad Rizvi

Hi,

I am trying to process watermarks in a BroadcastConnectedStream. However, I am 
not able to find any direct way to handle watermark events, similar to what we 
have in processWatermark1 in a  KeyedCoProcessOperator. Following are further 
details.

In the context of the example given in “A Practical Guide to Broadcast State in 
Apache Flink”, I have 
a user actions stream and a pattern stream. The pattern stream is broadcast and 
connected with the user actions stream. The result is a 
BroadcastConnectedStream. I want to handle user action events and pattern evens 
in this stream. In addition, I want to use a processWatermark function to 
perform an action in response to watermark events.

The problem is that a BroadcastConnectedStream has only process() function, no 
transform(), that takes a (Keyed)BroadcastProcessFunction. A 
BroadcastProcessFunction allows only to process elements, doesn’t provide the 
interface to process watermarks. In contrast, a ConnectedStream (without 
broadcast) provides a transform function, which takes in an operator that 
provides a way to process watermarks.

Is there a way to process watermarks in a BroadcastConnectedStream?

Thanks,
Sajjad

This email and any attachments are for the expressed and sole use of the 
intended recipient(s) and contain information that may be confidential and/or 
legally privileged. Any disclosure, copying, distribution or use of this 
communication by someone other than the intended recipient is strictly 
prohibited. If you are not the intended recipient please delete this email 
immediately. Any information and services described herein are provided by 
Arctic Wolf Networks, Inc.

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

2023-01-30 Thread Anton Ippolitov via user

This is actually what I'm already doing, I'm only setting high-availability:
kubernetes myself. The other values are either defaults or set by the
Operator:
- jobmanager.rpc.port: 6123 is the default value (docs

)
-  high-availability.jobmanager.port: 6123 is set by the Operator here


- jobmanager.rpc.address: SERVICE-NAME-HERE.NAMESPACE-HERE is set by the
Operator here

(the
actual code which gets executed is here

)

 Looking at what the Lyft Operator is doing here
,
I thought
this would be a common issue but since you've never seen this error before,
not sure what to do 樂

On Fri, Jan 27, 2023 at 10:59 PM Gyula Fóra  wrote:

> We never encountered this problem before but also we don't configure those
> settings.
> Can you simply try:
>
> high-availability: kubernetes
>
> And remove the other configs? I think that can only cause problems and
> should not achieve anything :)
>
> Gyula
>
> On Fri, Jan 27, 2023 at 6:44 PM Anton Ippolitov via user <
> user@flink.apache.org> wrote:
>
>> Hi everyone,
>>
>> I've been experimenting with Kubernetes HA and the Kubernetes Operator
>> and ran into the following issue which is happening regularly on
>> TaskManagers with Flink 1.16:
>>
>> Error while retrieving the leader gateway. Retrying to connect to 
>> akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE:6123/user/rpc/dispatcher_1.
>> org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not 
>> complete the operation. Number of retries has been exhausted.
>>
>> (The whole stacktrace is quite long, I put it in a Github Gist here
>> .
>> Note that I put placeholder values for the Kubernetes Service name and the
>> Namespace name)
>>
>> The job configuration has the following values which should be relevant:
>> high-availability: kubernetes
>> high-availability.jobmanager.port: 6123
>> jobmanager.rpc.address: SERVICE-NAME-HERE.NAMESPACE-HERE
>> jobmanager.rpc.port: 6123
>>
>> Looking a bit more into the logs, I can see that the Akka Actor System is
>> started with an external address pointing to the Kubernetes Service defined
>> by jobmanager.rpc.address:
>> Trying to start actor system, external
>> address SERVICE-NAME-HERE.NAMESPACE-HERE:6123, bind address 0.0.0.0:6123.
>> Actor system started at akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE
>> :6123
>>
>> (I believe the external address for the Akka Actor System is set to
>> jobmanager.rpc.address from this place
>> 
>> in the code but I might be wrong)
>>
>> I can also see these logs for the Dispatcher RPC endpoint:
>> Starting RPC endpoint for
>> org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
>> akka://flink/user/rpc/dispatcher_1 .
>> Successfully wrote leader information
>> LeaderInformation{leaderSessionID='8fd2bda3-1775-4b51-bf63-8da385247a18',
>> leaderAddress=akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE:6123/user/rpc/dispatcher_1}
>> for leader dispatcher into the config map JOB-NAME-HERE-cluster-config-map.
>>
>> I confirmed that the HA ConfigMap contains an address which also uses the
>> Kubernetes Service defined by jobmanager.rpc.address:
>> $ kubectl get cm JOB-NAME-HERE-cluster-config-map -o json | jq -r
>> '.data["org.apache.flink.k8s.leader.dispatcher"]'
>>
>> ce33b6d4-a55f-475c-9b6e-b21d25c8e6b5,akka.tcp://flink@SERVICE-NAME-HERE.NAMESPACE-HERE
>> :6123/user/rpc/dispatcher_1
>>
>> When looking at the code of the Operator and Flink itself, I can see
>> that jobmanager.rpc.address is set automatically by the
>> InternalServiceDecorator
>> 
>>  and
>> it points to the Kubernetes Service.
>> However, the comment
>>

Re: beam + flink + k8

2023-01-30 Thread Jan Lukavský


Hi,

can you please share the command-line and complete output of the script? 
Are you using minikube? Can you share list of your running pods?


 Jan

On 1/30/23 14:25, P Singh wrote:

Hi Team,

I am trying to run beam job on top of flink on my local machine 
(kubernetes).


 I have flink 1.14 and beam 2.43 images both running but when i submit 
the job it's not reaching to the flink cluster and getting failed with 
below error.


ERROR:apache_beam.utils.subprocess_server:Starting job service with 
['java', '-jar', 
'/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar', 
'--flink-master', 'http://localhost:8081', '--artifacts-dir', 
'/var/folders/n3/dqblsr792yj4kfs7xlfmdj54gr/T/beam-tempvphhje07/artifacts6kjt60ch', 
'--job-port', '57882', '--artifact-port', '0', '--expansion-port', '0']

ERROR:apache_beam.utils.subprocess_server:Error bringing up service
Traceback (most recent call last):
  File 
"/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py", 
line 88, in start

    raise RuntimeError(
RuntimeError: Service failed to start up with error 1

Any help would be appreciated.

Re: Using pyflink from flink distribution

2023-01-30 Thread Andrew Otto

Thanks Dian!

> >> Is using pyflink from the flink distribution tarball (without pip) not
a supported way to use pyflink?
> You are right.

What is the reason for including
opt/python/{pyflink.zip,cloudpickle.zip,py4j.zip} in the base distribution
then?  Oh, a guess: to make it easier for TaskManagers to run
pyflink without having pyflink installed themselves?  Somehow I'd guess
this wouldn't work tho; I'd assume TaskManagers would also need some python
transitive dependencies, e.g. google protobuf.

> you could remove the JAR packages located under
/usr/local/lib/python3.7/dist-packages/pyflink/lib manually after `pip
install apache-flink`

Since we're building our own Docker image, I'm going the other way around:
just install pyflink, and symlink /opt/flink ->
/usr/lib/python3.7/dist-packages/pyflink.  So far so good, but I'm worried
that something will be fishy when trying to run JVM apps via pyflink.

-Ao

On Sun, Jan 29, 2023 at 1:43 AM Dian Fu  wrote:

> Hi Andrew,
>
> >> By pip installing apache-flink, this docker image will have the flink
> distro installed at /opt/flink and FLINK_HOME set to /opt/flink
> .
> BUT ALSO flink lib jars will be installed at e.g.
> /usr/local/lib/python3.7/dist-packages/pyflink/lib!
> So, by following those instructions, flink is effectively installed twice
> into the docker image.
>
> Yes, your understanding is correct. The base image `flink:1.15.2` doesn't
> include PyFlink and so you need to build a custom image if you want to use
> PyFlink. Regarding to the jar packages which are installed twice, you could
> remove the JAR packages located under
> /usr/local/lib/python3.7/dist-packages/pyflink/lib manually after `pip
> install apache-flink`. It will use the JAR packages located under
> $FLINK_HOME/lib.
>
> >> Is using pyflink from the flink distribution tarball (without pip) not
> a supported way to use pyflink?
> You are right.
>
> Regards,
> Dian
>
>
> On Thu, Jan 26, 2023 at 11:12 PM Andrew Otto  wrote:
>
>> Ah, oops and my original email had a typo:
>> > Some python dependencies are not included in the flink distribution
>> tarballs: cloudpickle, py4j and pyflink are in opt/python.
>>
>> Should read:
>> > Some python dependencies ARE included in the flink distribution
>> tarballs: cloudpickle, py4j and pyflink are in opt/python.
>>
>> On Thu, Jan 26, 2023 at 10:10 AM Andrew Otto  wrote:
>>
>>> Let me ask a related question:
>>>
>>> We are building our own base Flink docker image.  We will be deploying
>>> both JVM and python apps via flink-kubernetes-operator.
>>>
>>> Is there any reason not to install Flink in this image via `pip install
>>> apache-flink` and use it for JVM apps?
>>>
>>> -Andrew Otto
>>>  Wikimedia Foundation
>>>
>>>
>>>
>>> On Tue, Jan 24, 2023 at 4:26 PM Andrew Otto  wrote:
>>>
 Hello,

 I'm having quite a bit of trouble running pyflink from the default
 flink distribution tarballs.  I'd expect the python examples to work as
 long as python is installed, and we've got the distribution.  Some python
 dependencies are not included in the flink distribution tarballs:
 cloudpickle, py4j and pyflink are in opt/python.  Others are not, e.g.
 protobuf.

 Now that I'm looking, I see that the pyflink installation instructions

  are
 to install via pip.

 I'm doing this in Docker for use with the flink-kubernetes-operator.
 In the Using Flink Python on Docker

  instructions,
 there is a pip3 install apache-flink step.  I find this strange, since I'd
 expect the 'FROM flink:1.15.2'  part to be sufficient.

 By pip installing apache-flink, this docker image will have the flink
 distro installed at /opt/flink and FLINK_HOME set to /opt/flink
 .
 BUT ALSO flink lib jars will be installed at e.g.
 /usr/local/lib/python3.7/dist-packages/pyflink/lib!
 So, by following those instructions, flink is effectively installed
 twice into the docker image.

 Am I correct or am I missing something?

 Is using pyflink from the flink distribution tarball (without pip) not
 a supported way to use pyflink?

 Thanks!
 -Andrew Otto
  Wikimedia Foundation

Re: Which flink version is compatible with beam

2023-01-30 Thread Andrew Otto

Hi, I'm not sure about beam, but Flink is not officially compatible with
python3.10.

https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/python/installation/

> Python version (3.6, 3.7 or 3.8) is required for PyFlink.


On Fri, Jan 27, 2023 at 11:50 PM P Singh 
wrote:

> Hi,
>
> It’s not working with flink 1.14 and beam 2.44 or 2.43 with python 3.10.
>
> Please suggest.
>
> Get Outlook for iOS 
> --
> *From:* Yaroslav Tkachenko 
> *Sent:* Friday, January 27, 2023 10:53:49 PM
> *To:* P Singh 
> *Cc:* user@flink.apache.org 
> *Subject:* Re: Which flink version is compatible with beam
>
> Hi! According to this
> https://beam.apache.org/documentation/runners/flink/#flink-version-compatibility,
> 1.14 is the latest supported version.
>
> On Fri, Jan 27, 2023 at 9:19 AM P Singh 
> wrote:
>
> Hi Team,
>
> I am trying to run apache beam pipeline on flink cluster. I have set up
> kubernetes locally with flink1.16and apache/beam_python3.10_sdk:2.44.0.
> When I submit the job using like
>
> python file.py
>
> Job is just hang not able to see on flink UI or logs.
>
> Can you please suggest compatible versions?
>
>
> Looking forward to hearing from you.
>
>

beam + flink + k8

2023-01-30 Thread P Singh

Hi Team,

I am trying to run beam job on top of flink on my local machine
(kubernetes).

 I have flink 1.14 and beam 2.43 images both running but when i submit the
job it's not reaching to the flink cluster and getting failed with below
error.

ERROR:apache_beam.utils.subprocess_server:Starting job service with
['java', '-jar',
'/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
'--flink-master', 'http://localhost:8081', '--artifacts-dir',
'/var/folders/n3/dqblsr792yj4kfs7xlfmdj54gr/T/beam-tempvphhje07/artifacts6kjt60ch',
'--job-port', '57882', '--artifact-port', '0', '--expansion-port', '0']
ERROR:apache_beam.utils.subprocess_server:Error bringing up service
Traceback (most recent call last):
  File
"/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
line 88, in start
raise RuntimeError(
RuntimeError: Service failed to start up with error 1

Any help would be appreciated.

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Jing Ge

Hi Yanfei,

Thanks for your effort. Looking forward to checking it.

Best regards,
Jing

On Mon, Jan 30, 2023 at 1:42 PM Yanfei Lei  wrote:

> It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.
>
> Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
> MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
> jar, and the FRocksDB in Flink 1.17 would be updated to
> 6.20.3-ververica-2.0.
>
> Release highlights:
> - [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
> - [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
> - Avoid expensive ToString() call when not in debug[3].
> - [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].
>
> Maven artifacts for FRocksDB can be found at:
> https://mvnrepository.com/artifact/com.ververica/frocksdbjni
>
> We would like to thank all efforts from the Apache Flink community
> that made this release possible!
>
> [1] https://issues.apache.org/jira/browse/FLINK-30457
> [2] https://issues.apache.org/jira/browse/FLINK-30321
> [3] https://github.com/ververica/frocksdb/pull/55
> [4] https://issues.apache.org/jira/browse/FLINK-24932
>
> Best regards,
> Yanfei
> Ververica(Alibaba)
>

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Jing Ge via user

Hi Yanfei,

Thanks for your effort. Looking forward to checking it.

Best regards,
Jing

On Mon, Jan 30, 2023 at 1:42 PM Yanfei Lei  wrote:

> It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.
>
> Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
> MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
> jar, and the FRocksDB in Flink 1.17 would be updated to
> 6.20.3-ververica-2.0.
>
> Release highlights:
> - [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
> - [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
> - Avoid expensive ToString() call when not in debug[3].
> - [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].
>
> Maven artifacts for FRocksDB can be found at:
> https://mvnrepository.com/artifact/com.ververica/frocksdbjni
>
> We would like to thank all efforts from the Apache Flink community
> that made this release possible!
>
> [1] https://issues.apache.org/jira/browse/FLINK-30457
> [2] https://issues.apache.org/jira/browse/FLINK-30321
> [3] https://github.com/ververica/frocksdb/pull/55
> [4] https://issues.apache.org/jira/browse/FLINK-24932
>
> Best regards,
> Yanfei
> Ververica(Alibaba)
>

[ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Yanfei Lei

It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.

Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
jar, and the FRocksDB in Flink 1.17 would be updated to
6.20.3-ververica-2.0.

Release highlights:
- [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
- [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
- Avoid expensive ToString() call when not in debug[3].
- [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].

Maven artifacts for FRocksDB can be found at:
https://mvnrepository.com/artifact/com.ververica/frocksdbjni

We would like to thank all efforts from the Apache Flink community
that made this release possible!

[1] https://issues.apache.org/jira/browse/FLINK-30457
[2] https://issues.apache.org/jira/browse/FLINK-30321
[3] https://github.com/ververica/frocksdb/pull/55
[4] https://issues.apache.org/jira/browse/FLINK-24932

Best regards,
Yanfei
Ververica(Alibaba)

[ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

2023-01-30 Thread Yanfei Lei

It is very happy to announce the release of FRocksDB 6.20.3-ververica-2.0.

Compiled files for Linux x86, Linux arm, Linux ppc64le, MacOS x86,
MacOS arm, and Windows are included in FRocksDB 6.20.3-ververica-2.0
jar, and the FRocksDB in Flink 1.17 would be updated to
6.20.3-ververica-2.0.

Release highlights:
- [FLINK-30457] Add periodic_compaction_seconds option to RocksJava[1].
- [FLINK-30321] Upgrade ZLIB of FRocksDB to 1.2.13[2].
- Avoid expensive ToString() call when not in debug[3].
- [FLINK-24932] Support build FRocksDB Java on Apple silicon[4].

Maven artifacts for FRocksDB can be found at:
https://mvnrepository.com/artifact/com.ververica/frocksdbjni

We would like to thank all efforts from the Apache Flink community
that made this release possible!

[1] https://issues.apache.org/jira/browse/FLINK-30457
[2] https://issues.apache.org/jira/browse/FLINK-30321
[3] https://github.com/ververica/frocksdb/pull/55
[4] https://issues.apache.org/jira/browse/FLINK-24932

Best regards,
Yanfei
Ververica(Alibaba)

Re: Which flink version is compatible with beam

2023-01-30 Thread bigdatadeveloper8

I have added flink:1.14 with Apache/beam:2.38.0When I submit the job python 
basic.py get warning:Root: make sure locally built python SDK docker image has 
python 3.8 interpreter.
After this warning nothing is happening no logs or trace back.
Please help.


Sent from my Galaxy

 Original message From: P Singh  
Date: 28/01/2023  10:20  (GMT+05:30) To: Yaroslav Tkachenko 
 Cc: user@flink.apache.org Subject: Re: Which flink 
version is compatible with beam 



Hi,



It’s not working with flink 1.14 and beam 2.44 or 2.43 with python 3.10. 



Please suggest. 





Get Outlook for iOS


From: Yaroslav Tkachenko 

Sent: Friday, January 27, 2023 10:53:49 PM

To: P Singh 

Cc: user@flink.apache.org 

Subject: Re: Which flink version is compatible with beam
 


Hi! According to this 
https://beam.apache.org/documentation/runners/flink/#flink-version-compatibility,
 1.14 is the latest supported version.



On Fri, Jan 27, 2023 at 9:19 AM P Singh  wrote:







Hi Team,



I am trying to run apache beam pipeline on flink cluster. I have set up 
kubernetes locally with flink1.16and apache/beam_python3.10_sdk:2.44.0. When I 
submit the job using like 



python file.py 



Job is just hang not able to see on flink UI or logs. 



Can you please suggest compatible versions?






Looking forward to hearing from you.

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

Flink SQL 如何优化以及处理反压

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Re: 退订

Re: 任务本地运行正常，提交到集群报错 - 图片挂掉，文字贴一下报错信息，非常抱歉打扰

Re: Using pyflink from flink distribution

Processing watermarks in a broadcast connected stream

Re: "Error while retrieving the leader gateway" when using Kubernetes HA

Re: beam + flink + k8

Re: Using pyflink from flink distribution

Re: Which flink version is compatible with beam

beam + flink + k8

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Re: [ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

[ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

[ANNOUNCE] FRocksDB 6.20.3-ververica-2.0 released

Re: Which flink version is compatible with beam

20 matches

Site Navigation

Mail list logo

Footer information