Re: 问题求助（Pyflink）

你好， 可以看下source task所在task manager 的日志里看看 consumer 有没有成功获取到kafka partition相关meta信息和认证相关是否成功的信息。 瞿叶奇 <389243...@qq.com> 于2021年1月30日周六 下午3:14写道： > 老师，你好，消费是没有任何问题，可以正常消费。 > > > > > -- 原始邮件 -- > *发件人:* "user-zh" ; > *发送时间:* 2021年1月30日(星期六) 下午3:08 > *收件人:* "user-zh"; >

????????????????Pyflink??

?? ---- ??: "user-zh"

Re:问题求助（Pyflink）

先看下kafka能否通过命令行消费数据. 命令行检查确保能消费,再使用Flink. 在 2021-01-30 14:25:57，"瞿叶奇" <389243...@qq.com> 写道： 老师，您好，我想通过Flink消费kafka写本地csv文件，目前遇到的问题是Flink、Kafka都是kerberos认证的集群，而且我是用的是pyflink，现在程序能执行，不报错，但是不消费数据，csv文件没有结果，但是文件日期修改日期一直在更新。怀疑对接kafka 存在问题，希望老师能够给解决疑惑。 1）Kafka生产数据： 2）pyflink 程序

??????????Pyflink??

Flinkkafka??csv??Flink??Kafkakerberos??pyflink??csv??kafka ?? 1??Kafka?? 2??pyflink

Unknown call expression: avg(amount) when use distinct() in Flink Thanks~!

I posted this question here. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Unknown-call-expression-avg-amount-when-use-distinct-in-Flink-Thanks-td40905.html but my later replies to it is NOT delivered,it's strange. Flink1.12 scala2.12 Could anyone help me? Thanks~!

Unknown call expression: avg(amount) when use distinct() in Flink Thanks~!

I posted this question here. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Unknown-call-expression-avg-amount-when-use-distinct-in-Flink-Thanks-td40905.html but my replies to it is now delivered,it's strange. Flink1.12 scala2.12 Could anyone help me? Thanks~!

Re: Flink 1.11 job hit error "Job leader lost leadership" or "ResourceManager leader changed to new address null"

Thank you~ Xintong Song On Sat, Jan 30, 2021 at 8:27 AM Xintong Song wrote: > There's indeed a ZK version upgrading during 1.9 and 1.11, but I'm not > aware of any similar issue reported since the upgrading. > I would suggest the following: > - Turn on the DEBUG log see if there's any

Re: Flink 1.11 job hit error "Job leader lost leadership" or "ResourceManager leader changed to new address null"

There's indeed a ZK version upgrading during 1.9 and 1.11, but I'm not aware of any similar issue reported since the upgrading. I would suggest the following: - Turn on the DEBUG log see if there's any valuable details - Maybe try asking in the Apache Zookeeper community, see if this is a known

Re: Deduplicating record amplification

Great, thank you for the confirmation! On Thu, Jan 28, 2021 at 11:25 PM Arvid Heise wrote: > Hi Rex, > > there cannot be any late event in processing time by definition (maybe on > a quantum computer?), so you should be fine. The timestamp of records in > processing time is monotonously

LEAD/LAG functions

Another (hopefully newbie) question. Trying to use LEAD/LAG over window functions. I get the following error. The exact same query works properly using FIRST_VALUE instead of LEAD. Thanks in advance... - Patrick Flink SQL> describe l1_min ;

Re: Flink SQL OVER window

Thanks, Seth. That did the trick! On Fri, Jan 29, 2021 at 10:51 AM Seth Wiesman wrote: > You need to use TUMBLE_ROWTIME to extract a time attribute from a window, > TUMBLE_END is just a timestamp. > > >

Re: Flink SQL OVER window

You need to use TUMBLE_ROWTIME to extract a time attribute from a window, TUMBLE_END is just a timestamp. https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html#selecting-group-window-start-and-end-timestamps Seth On Fri, Jan 29, 2021 at 9:14 AM Patrick Angeles

Re: Flink SQL OVER window

Forgot to mention, I am using Flink 1.12. On Fri, Jan 29, 2021 at 10:11 AM Patrick Angeles wrote: > Fairly new to Flink here so this might be a newbie mistake, but here's the > problem. I created the following table and view: > > >> CREATE TABLE test ( >> >> event_time TIMESTAMP(3), >>

Flink SQL OVER window

Fairly new to Flink here so this might be a newbie mistake, but here's the problem. I created the following table and view: > CREATE TABLE test ( > > event_time TIMESTAMP(3), > > symbol STRING, > > price DOUBLE, > > WATERMARK FOR event_time AS event_time -

Re: flink checkpoints adjustment strategy

Do you have advice on how to determine why a checkpoint failed? 1. Timeout (that's easy to discover as the UI logs them). 2. Other errors are not so easy to find. How can I find other errors? Are they in the UI, or good old-fashioned logging? On Fri, Jan 29, 2021 at 3:11 AM Congxian Qiu wrote:

Re: Resource changed on src filesystem after upgrade

Hi Xintong Song, > - Does this error happen for every of your dataset jobs? For a problematic > job, does it happen for every container? > - What is the `jobs.jar`? Is it under `lib/`, `opt` of your client side > filesystem, or specified as `yarn.ship-files`, `yarn.ship-archives` or >

FOR SYSTEM_TIME AS OF 维表关联 报错

各位大佬 在flink sql客户端执行维度关联报错 sql语句：insert into sink_a select a.user_id, b.user_name from source_a as a left join source_b FOR SYSTEM_TIME AS OF a.proc_time b on a.user_id = b.user_id; 报错信息 [ERROR] Could not execute SQL statement. Reason:

Re: flink checkpoints adjustment strategy

Hi Marco You need to figure out why the checkpoint timed out(you can see the consumed time of each period for one checkpoint in UI), if it indeed needs such long time to complete the checkpoint, then you need to configure a longer timeout. If there are some checkpoint errors, we need

Re:提交job的命令，./bin/flink run-application -t yarn-application ... 和 ./bin/flink run -m yarn-cluster ...

-t是flink1.12出来的，通过-t指定提交模式后，yarn相关参数要通过 -D来指定，例如 -D yarn.application.name 在 2021-01-29 12:52:41，"lp" <973182...@qq.com> 写道： >如题，在 ./flink --help中看到提交job的命令有两个相似的，貌似都会将jobManager发送yarn >node上去之行，但不明白他们区别，官网也未找到他们的区别，请帮忙解释下他们之间的区别？ > > > >-- >Sent from:

Re: [ANNOUNCE] Apache Flink 1.10.3 released

Thanks Xintong for driving this release! Regards, Dian > 在 2021年1月29日，下午5:24，Till Rohrmann 写道： > > Thanks Xintong for being our release manager. Well done! > > Cheers, > Till > > On Fri, Jan 29, 2021 at 9:50 AM Yang Wang > wrote: > Thanks Xintong for driving

Re: [ANNOUNCE] Apache Flink 1.10.3 released

Thanks Xintong for driving this release! Regards, Dian > 在 2021年1月29日，下午5:24，Till Rohrmann 写道： > > Thanks Xintong for being our release manager. Well done! > > Cheers, > Till > > On Fri, Jan 29, 2021 at 9:50 AM Yang Wang > wrote: > Thanks Xintong for driving

Re: 为什么window的reduceFunction不支持RichFunction呢

你好， 请问想用reduce 里面需要存储状态，可以用 aggregate() 吗？ 看了一下代码，窗口的话，是把数据存储在窗口里面的状态。reduce 会生成 reduce 对应的状态存储放入窗口。任务窗口函数是不是自定义的，里面的状态也不能自定义。有聚合的情况下，只是把聚合函数作用在了窗口的状态上，里面的状态是要保存聚合结果的状态。 可以任务 窗口加聚合是一个算子。 likai 1137591...@qq.com > 在 2021年1月29日，下午12:49，赵一旦 写道： > > windowFunc

?????? ????????????

kafka ??3??, ??, flink??3 ---- ??: "user-zh"

Re: 未生成水位线

看一下 WaterMarkAssigner节点 是否有 数据流入 -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Flink on Kubernetes, Task/Job Manager Recycles

I think you need to enable the HA(high availability) for your Flink cluster[1]. Currently, we have the ZooKeeperHAService and KubernetesHAService. In the HA mode, all the meta data(e.g. job graph path, checkpoint counter, checkpoint path) will be stored on ZooKeeper or Kubernetes ConfigMap. And

Re: [ANNOUNCE] Apache Flink 1.10.3 released

Thanks Xintong for being our release manager. Well done! Cheers, Till On Fri, Jan 29, 2021 at 9:50 AM Yang Wang wrote: > Thanks Xintong for driving this release. > > Best, > Yang > > Yu Li 于2021年1月29日周五 下午3:52写道： > >> Thanks Xintong for being our release manager and everyone else who made >>

Re: [ANNOUNCE] Apache Flink 1.10.3 released

Thanks Xintong for being our release manager. Well done! Cheers, Till On Fri, Jan 29, 2021 at 9:50 AM Yang Wang wrote: > Thanks Xintong for driving this release. > > Best, > Yang > > Yu Li 于2021年1月29日周五 下午3:52写道： > >> Thanks Xintong for being our release manager and everyone else who made >>

Re: Configuring ephemeral storage limits when using Native Kubernetes

Hi Emilien, Thanks for trying the native Flink integration. Unfortunately, we still do not have the ability to set the ephemeral storage limit. I think it could be supported via pod template[1]. I am still working on this ticket and already have a draft PR[2]. I believe it could be supported in

Re:关于Flink作业的负载监控 task-load指标

Hi， 在 `MailboxProcessor#runMailboxLoop` 中分别计算 default mailbox actions(处理业务数据) 和 event mailbox actions(checkpoint 同步阶段、timer等) 的时间占比，假设分别为t1，t2，都介于[0,1]之间。 那么理论上 t1 + t2 + idle的占比 = 1；这样可以根据 t1， t2 的值来判断单个线程的 CPU 是否跑满了。 Best, Hailong 在 2021-01-29 12:25:56，"1305332" <1305...@163.com>

Re: Timers not firing until stream end

1) Outside of small-scale tests the periodic emission of watermarks should ensure that the latency is not unbounded. Your test just runs so quickly that this never triggers. As for the triggering element being emitted first: The reason this happen is because watermarks are not really attached

Re: 提交job的命令，./bin/flink run-application -t yarn-application ... 和 ./bin/flink run -m yarn-cluster ...

-m yarn-cluster和-t yarn-per-job都是可以用来提交per-job任务到Yarn集群的 只是背后实现的CLI不一样而已，前者FlinkYarnSessionCLI是以前的方式 后者是在1.10引入的一个更加通用的方式，可以和K8s、Standalone等保持一致 另外，还有一个差异是，-m yarn-cluster是可以支持-yq -ynm等这些CLI参数的 -t yarn-per-job只能通过-D的方式来设置 Best, Yang lp <973182...@qq.com> 于2021年1月29日周五 下午3:00写道： >

????????????

EnvironmentSettings settings = EnvironmentSettings.newInstance().inStreamingMode().useBlinkPlanner().build(); StreamExecutionEnvironment executionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment(); executionEnvironment.setParallelism(3); Map

Re: [ANNOUNCE] Apache Flink 1.10.3 released

Thanks Xintong for driving this release. Best, Yang Yu Li 于2021年1月29日周五 下午3:52写道： > Thanks Xintong for being our release manager and everyone else who made > the release possible! > > Best Regards, > Yu > > > On Fri, 29 Jan 2021 at 15:05, Xintong Song wrote: > >> The Apache Flink community is

Re: [ANNOUNCE] Apache Flink 1.10.3 released

Thanks Xintong for driving this release. Best, Yang Yu Li 于2021年1月29日周五 下午3:52写道： > Thanks Xintong for being our release manager and everyone else who made > the release possible! > > Best Regards, > Yu > > > On Fri, 29 Jan 2021 at 15:05, Xintong Song wrote: > >> The Apache Flink community is