Re: Flink1.2对 key 进行分区，和 hash 分区有什么区别?

首先，本身就是对key做的hash哈。只不过不是直接分配到并行的subtask，而是先分到maxParallelism，然后再分到subtask。加了一层主要是方便状态scala。 刘文 于2021年4月6日周二 上午9:33写道： > > > Flink1.2对 key 进行分区，和 hash 分区有什么区别? > 如: 分区数值 = key 的 hash值 % 并行度？ > > 为什么不直接使用 hash 进行分区? > > KeyGroupStreamPartitioner.java > > @Override > public int

Re: Application cluster - Job execution and cluster creation timeouts

Hi Tamir, Maybe I did not make myself clear. Here the "deployer" means our internal Flink application deployer(actually it is ververica platform), not the *ApplicationDeployer* interface in Flink. It helps with managing the lifecycle of every Flink application. And it has the same native K8s

Flink: Exception from container-launch exitCode=2

*The question is cross-posted on Stack Overflow https://stackoverflow.com/questions/66968180/flink-exception-from-container-launch-exitcode-2 . Viewing the question on Stack Overflow is preferred as I

Re: Dynamic configuration via broadcast state

I researched a bit more and another suggested solution is to build a custom source function that somehow waits for each operator to load it's configuration which is infact set in the open method of the source itself. I'm not sure if that's a good idea as that just exposes entire job configuration

Re: flink 创建hbase出错

今天看到spi相关的资料，明白了一些，maven把类打进jar包，但是里面的类不是全部加载进内存，而且有使用到才加载，而像sinktable或者soucetable这些调用一般都是通过多态调用的，导致实现类不会加载，导致找不到。而如果放在lib目录，flink启动的时候会手动加载里面的类。 --Original-- From: "lucas.wu"

questions regarding stateful functions

Upon reading about stateful functions, it seems as though first, a data stream has to flow to an event ingress. Then, the stateful functions will perform computations via whatever functionality it provides. Finally, the results of said computations will flow to the event egress which will be yet

Dynamic configuration via broadcast state

I have to make my flink job dynamically configurable and I'm thinking about using broadcast state. My current static job configuration file consists of configuration of entire set of operators which I load into a case class and then I explicitly pass the relevant configuration of each operator as

Re: Flink - Pod Identity

Great, glad to hear it Swagat! Did you end up using Flink 1.6 or were you able to upgrade to Flink 1.12? Could you also link the ticket back here if you've already made it/ make sure it is not a duplicate of FLINK-18676 ? Best, Austin On Tue,

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

Hi Yang, Thanks for the reply, those information is very helpful. Best, Fuyao From: Yang Wang Date: Tuesday, April 6, 2021 at 01:11 To: Fuyao Li Cc: user Subject: Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment Hi Fuyao, Sorry for the late reply.

Re: Flink Taskmanager failure recovery and large state

Hey Yaroslav, GCS is a somewhat popular filesystem that should work fine with Flink. It seems that the initial scale of a bucket is 5000 read requests per second (https://cloud.google.com/storage/docs/request-rate), your job should be at roughly the same rate (depending on how fast your job

Re: Source Operators Stuck in the requestBufferBuilderBlocking

Hi Sihan, Unfortunately, we are unable to reproduce the issue so far. Could you please describe in more detail the job graph, in particular what are the downstream operators and whether there is any chaining? Do I understand correctly, that Flink returned back to normal at around 8:00; worked

Re: [DISCUSS] Feature freeze date for 1.13

Hi Guowei, Who should I speak to regarding this? I am at the final stages of the PR I believe (Shengkai is kindly helping me make things work) and I would like to push this into 1.13. On Fri, Apr 2, 2021 at 5:43 AM Guowei Ma wrote: > Hi, Yuval > > Thanks for your contribution. I am not a SQL

Re: Flink - Pod Identity

I was able to solve the issue by providing a custom version of the presto jar. I will create a ticket and raise a pull request so that others can benefit from it. I will share the details here shortly. Thanks everyone for your help and support. Especially Austin, he stands out due to his interest

Re: How to know if task-local recovery kicked in for some nodes?

Hi Sonam, We have a similar setup. What I have observed is, when the task manager pod gets killed and restarts again ( i.e. the entire task manager process restarts ) then local recovery doesn't happen. Task manager restore process actually downloads the latest completed checkpoint from the

Task manager local state data after crash / recovery

Hey all, We are running a stateful stream processing job on k8s using per-job standalone deployment entrypoint. Flink version: 1.12.1 *Problem*: We have observed that whenever a task manager is either gracefully shut down or killed ( due to OOM, k8s worker node drain out etc ) it doesn't clean

CheckpointedFunction#snapshotState????????????????

class A extends KeyedProcessFunction

Re: （无主题）

退订 guoyb <861277...@qq.com> 于2021年4月6日周二 下午5:57写道： > To remove your address from the list, send a message to: > > > > ---原始邮件--- > 发件人: "郭华威" 发送时间: 2021年4月6日(周二) 下午5:52 > 收件人: "user-zh@flink.apache.org" 主题: 回复：（无主题） > > > 请问如何退订 > > 在2021年04月06日 16:37，郭华威 写道： > 退订

Re: Application cluster - Job execution and cluster creation timeouts

Hey Yang Thank you for your respond We run the application cluster programmatically. I discussed about it here with an example how to run it from java and not CLI. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Application-cluster-Best-Practice-td42011.html following your

Re: How to know if task-local recovery kicked in for some nodes?

Hi Sonam, The easiest way to see whether local state has been used for recovery is the recovery time. Apart from that you can also look for "Found registered local state for checkpoint {} in subtask ({} - {} - {}" in the logs which is logged on debug. This indicates that the local state is

Re: period batch job lead to OutOfMemoryError: Metaspace problem

> I have tried this method, but the problem still exist. How much memory do you configure for it? > is 21 instances of "org.apache.flink.util.ChildFirstClassLoader" normal Not quite sure about it. AFAIK, each job will have a classloader. Multiple tasks of the same job in the same TM will share

回复： period batch job lead to OutOfMemoryError: Metaspace problem

I have tried this method, but the problem still exist. by heap dump analysis, is 21 instances of "org.apache.flink.util.ChildFirstClassLoader" normal? --原始邮件-- 发件人:

Flink1.12.0版本 FlinkStreaming 如何将偏移量提交到kafka 0.11

flink 1.12版本 kafka版本0.11版本 目前可以消费，但是偏移量无法提交到kafka 我试过相同的代码，kafka版本2.4.1就可以提交偏移量到kafka 目前kafka 0.11版本有问题。无法提交。 有没有大佬帮忙想想办法。如何解决这个版本问题。

回复：（无主题）

To remove your address from the list, send a message to:

回复：（无主题）

请问如何退订 在2021年04月06日 16:37，郭华威 写道： 退订

Re: How to know if task-local recovery kicked in for some nodes?

Hi Sonam, Pulling in Till (cc'ed), I believe he would likely be able to help you here. Cheers, Gordon On Fri, Apr 2, 2021 at 8:18 AM Sonam Mandal wrote: > Hello, > > We are experimenting with task local recovery and I wanted to know whether > there is a way to validate that some tasks of the

Re: Checkpoint timeouts at times of high load

It could very well be that your job gets stuck in a restart loop for some reason. Can you either post the full TaskManager logs here, or try to figure out yourself why the first checkpoint that timed out, timed out? Backpressure or blocked operators are a common cause for this. In your case, it

Re: 如何手动读取Savepoints中的内容？

已解决，阅读完善相关源码后，了解到。需要通过BatchEnvironment，把Savepoint当做输入，然后构造序列化解析器和类型等信息，通过DataSet.collect()进行解析，就可以读取到目标数据了。 -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Why is Hive dependency flink-sql-connector-hive not available on Maven Central?

Thanks for the tip! On Tue, Apr 6, 2021 at 4:25 PM Rui Li wrote: > Hi Yik San, > > Glad to know you've found the jar. Another option to locate the jar is to > just use maven dependency plugin like this: > > *mvn dependency:get >

（无主题）

退订

Re: period batch job lead to OutOfMemoryError: Metaspace problem

I think you can try to increase the JVM metaspace option for TaskManagers through taskmanager.memory.jvm-metaspace.size. [1] [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/memory/mem_trouble/#outofmemoryerror-metaspace Best, Yangze Guo Best, Yangze Guo On Tue, Apr

Re: Why is Hive dependency flink-sql-connector-hive not available on Maven Central?

Hi Yik San, Glad to know you've found the jar. Another option to locate the jar is to just use maven dependency plugin like this: *mvn dependency:get -Dartifact=org.apache.flink:flink-sql-connector-hive-2.3.6_2.12:1.12.2* On Tue, Apr 6, 2021 at 4:10 PM Yik San Chan wrote: > Hi, > > I am able

Flink 1.12.2 sql api use parquet format error

ref:https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/formats/parquet.html env and error: Flink version?? 1.12.2 deployment?? standalone kubernetes session dependency:

如何手动读取Savepoints中的内容？

目前有个需求想要实现Flink SQL的保存点，但是由于在SQL中operator uid是随机生成的，一旦修改SQL会导致无法读取到Savepoints中的状态信息。 想到一种方式是正常执行Savepoint操作，然后启动的时候手动读取Savepoint中的内容，获取Kafka每个分区的消费offset，再替换到SQL代码中。 目前通过在源码中打印相关日志可以发现，保存点触发时，消费的记录正常读取到9，但是手动读取的时候，却找不到这个而信息。 下面是我手动读取的代码，以下内容是参考Savepoint单元测试中的读取案例： String savepointPath

period batch job lead to OutOfMemoryError: Metaspace problem

batch job?? read data from s3 by sql??then by some operators and write data to clickhouse and kafka. after some times, task-manager quit with OutOfMemoryError: Metaspace. env?? flink version??1.12.2 task-manager slot count: 5 deployment?? standalone kubernetes session dependencies??

回复： Flink 1.12.2 sql api 使用parquet格式报错

加下测试看看，你这很明显没找到包 ---原始邮件--- 发件人: "user-zh-return-12015-861277329=qq.com"https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/formats/parquet.html 添加的依赖， 是需要把nbsp;flink-parquet_2.11 文件手动添加到 lib 目录才可以的吗？ --nbsp;原始邮件nbsp;-- 发件人:

（无主题）

退订

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

Hi Fuyao, Sorry for the late reply. It is not very hard to develop your own deployer. Actually, I have 3 days for developing the PoC version of flink-native-k8s-operator. So if you want to have a fully functional K8s operator, maybe two weeks is enough. But if you want to put it into production,

Re: Why is Hive dependency flink-sql-connector-hive not available on Maven Central?

Hi, I am able to find the jar from Maven central. See updates in the StackOverflow post. Thank you! Best, Yik San On Tue, Apr 6, 2021 at 4:05 PM Tzu-Li (Gordon) Tai wrote: > Hi, > > I'm pulling in Rui Li (cc'ed) who might be able to help you here as he > actively maintains the hive

?????? Flink 1.12.2 sql api ????parquet????????

??https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/connectors/formats/parquet.html flink-parquet_2.11 ?? lib ---- ??:

Re: Why is Hive dependency flink-sql-connector-hive not available on Maven Central?

Hi, I'm pulling in Rui Li (cc'ed) who might be able to help you here as he actively maintains the hive connectors. Cheers, Gordon On Fri, Apr 2, 2021 at 11:36 AM Yik San Chan wrote: > The question is cross-posted in StackOverflow >

回复：（无主题）

退订 | | hello | | yangx...@163.com | Signature is customized by Netease Mail Master 在2021年04月06日 11:46，郭华威 写道： 退订

Re: Application cluster - Job execution and cluster creation timeouts

Hi Tamir, Thanks for trying the native K8s integration. 1. We do not have a timeout for creating the Flink application cluster. The reason is that the job submission happens on the JobManager side. So the Flink client does not need to wait for the JobManager running and then exit. I think even

Re: Zigzag shape in TM JVM used memory

Hi, this should be posted on the user mailing list not the dev. Apart from that, this looks like normal/standard behaviour of JVM, and has very little to do with Flink. Garbage Collector (GC) is kicking in when memory usage is approaching some threshold: