Re: flink 1.12.0 k8s session部署异常

这个问题的根本原因是云上LoadBalancer一直在给Flink创建的service发送RST包导致了 这个JIRA[1]可以了解更多信息 临时绕过去的方案就是在log4j2配置里面把org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint这个类的log级别调到ERROR [1]. https://issues.apache.org/jira/browse/FLINK-18129 Best, Yang 18756225...@163.com <18756225...@163.com> 于2021年3月24日周三

Re: EOFException on attempt to scale up job with RocksDB state backend

Hi Yun, Finally I was able to try to rescale with block blobs configured - rescaled from 6 to 8 w/o problem. So loos like indeed there is problem with page blob. Thank you for help, Alexey From: Alexey Trenikhun Sent: Thursday, March 18, 2021 11:31 PM To: Yun

Re: Re: About Memory Spilling to Disk in Flink

Hi, Roc Thanks for your detailed explanation. I could not find any "stream" operator that uses `ExternalSorterBuilder` by "find usage" of the IDEA. Best, Guowei On Wed, Mar 24, 2021 at 3:27 PM Roc Marshal wrote: > Hi, Guowei Ma. > As far as I know, flink writes some in-memory data to disk

Re: [EXTERNAL] Re: PyFlink DataStream Example Kafka/Kinesis?

Hi Shuiqiang, Thanks for letting me know. Feel free to send any beginner level contributions for this effort my way  . Best, kevin From: Shuiqiang Chen Date: Wednesday, March 24, 2021 at 10:31 PM To: "Bohinski, Kevin" Cc: user Subject: [EXTERNAL] Re: PyFlink DataStream Example

Re: PyFlink DataStream Example Kafka/Kinesis?

Sure, no problem. You can refer to the implementation of Kafka connector, they are very much alike. Xinbin Huang 于2021年3月25日周四 上午10:55写道： > Hi Shuiqiang, > > Thanks for the quick response on creating the ticket for Kinesis > Connector. Do you mind giving me the chance to try to implement the >

Re: 退订

你需要发邮件到 Best, Kezhu Wang On March 25, 2021 at 10:15:56, drewfranklin (drewfrank...@163.com) wrote: 退订

Re: With the checkpoint interval of the same size, the Flink 1.12 version of the job checkpoint time-consuming increase and production failure, the Flink1.9 job is running normally

Hi，Congxian ,thanks for your replay. job run on Flink1.9 （checkpoint interval 3min） job run on Flink1.12 （checkpoint interval 10min）

Re: PyFlink DataStream Example Kafka/Kinesis?

I have just created the jira https://issues.apache.org/jira/browse/FLINK-21966 and will finish it soon. Best, Shuiqiang Xinbin Huang 于2021年3月25日周四 上午10:43写道： > Hi Shuiqiang, > > I am interested in the same feature. Do we have a ticket to track this > right now? > > Best > Bin > > On Wed, Mar

Re: With the checkpoint interval of the same size, the Flink 1.12 version of the job checkpoint time-consuming increase and production failure, the Flink1.9 job is running normally

Hi，Congxian ,thanks for your replay. job run on Flink1.9 （checkpoint interval 3min） job run on Flink1.12 （checkpoint interval 10min）

Re: PyFlink DataStream Example Kafka/Kinesis?

Hi Kevin, Kinesis connector is not supported yet in Python DataStream API. We will add it in the future. Best, Shuiqiang Bohinski, Kevin 于2021年3月25日周四 上午5:03写道： > Is there a kinesis example? > > > > *From: *"Bohinski, Kevin" > *Date: *Wednesday, March 24, 2021 at 4:40 PM > *To: *"Bohinski,

退订

退订

State size increasing exponentially in Flink v1.9

Hey, Hope you all are doing well! I am using flink v1.9 with RocksDBStateBackend, but over time the state size is increasing exponentially. I am using MapState in my project & seeing memory spike, after looking at heap dump I see duplicates in it. I also have logic added to remove expired

Re: PyFlink DataStream Example Kafka/Kinesis?

Is there a kinesis example? From: "Bohinski, Kevin" Date: Wednesday, March 24, 2021 at 4:40 PM To: "Bohinski, Kevin" Subject: Re: PyFlink DataStream Example Kafka/Kinesis? Nevermind, found this for anyone else looking:

Re: PyFlink DataStream Example Kafka/Kinesis?

Nevermind, found this for anyone else looking: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-python-test/python/datastream/data_stream_job.py From: "Bohinski, Kevin" Date: Wednesday, March 24, 2021 at 4:38 PM To: user Subject: PyFlink DataStream Example

PyFlink DataStream Example Kafka/Kinesis?

Hi, Is there an example kafka/kinesis source or sink for the PyFlink DataStream API? Best, kevin

Re: Evenly distribute task slots across task-manager

Hi Matthias, Thanks for your reply. In my case, yes the upstream operator for the operator which is not distributed evenly among task managers is a flink Kafka connector with a rebalance(shuffling). Regards, Vignesh On Tue, 23 Mar, 2021, 6:48 pm Matthias Pohl, wrote: > There was a similar

Re: DataDog and Flink

yep, not a single EP that does all the dump but something like this works ( dirty but who cares :)) .. The vertex metrics are the most numerous any way ```curl -s http:///jobs/[job_id] | jq -r '.vertices' | jq '.[].id' | xargs -I {} curl http://xx/jobs/[job_id]/vertices/{}/metrics | jq

Re: Kubernetes HA - attempting to restore from wrong (non-existing) savepoint

Hi Yang, HA data was cleared, but it would be re-created when Kubernetes will restart failed (due to code 2) job. So upgrade will happen on life job. I guess upgrade procedure, should recheck or monitor kubernetes job to ensure that it is completed Thanks, Alexey

Re: OOM issues with Python Objects

Hi Dian, I have unit tests for which both sets of code (Row subclass vs. custom Python class) passes. The OOM occurs when reading a large amount of data from a kafka topic. At the moment I don't have a simple example to reproduce the issue, I'll let you know. On Tue, Mar 23, 2021 at 2:17 AM

Native kubernetes execution and History server

Hi, I would like to use native kubernetes execution [1] for one batch job and let scheduling on kubernetes. Flink version: 1.12.2. Kubernetes job: apiVersion: batch/v1beta1 kind: CronJob metadata: name: scheduled-job spec: schedule: "*/1 * * * *" jobTemplate: spec: template:

Re: DataDog and Flink

Yes, I will do that. Regarding the metrics dump through REST, it does provide for the TM specific but refuses to do it for all jobs and vertices/operators etc .Moreover I am not sure I have access to the vertices ( vertex_id ) readily from the UI. curl http://[jm]/taskmanagers/[tm_id] curl

Re: flink sql jmh failure

Hi, Yik San I use a library wroten by myself and trying to verify the performance. Yik San Chan 于2021年3月24日周三 下午9:07写道： > Hi Jie, > > I am curious what library do you use to get the ClickHouseTableBuilder > > On Wed, Mar 24, 2021 at 8:41 PM jie mei wrote: > >> Hi, Community >> >> I run a jmh

Re: flink sql jmh failure

Hi Jie, I am curious what library do you use to get the ClickHouseTableBuilder On Wed, Mar 24, 2021 at 8:41 PM jie mei wrote: > Hi, Community > > I run a jmh benchmark task get blew error, which use flink sql consuming > data from data-gen connector(10_000_000) and write data to clickhouse.

Re: With the checkpoint interval of the same size, the Flink 1.12 version of the job checkpoint time-consuming increase and production failure, the Flink1.9 job is running normally

Hi From the description, the time used to complete the checkpoint in 1.12 is longer. could you share more detail about the time consumption when running job on 1.9 and 1.12? Best, Congxian Haihang Jing 于2021年3月23日周二 下午7:22写道： > 【Appearance】For jobs with the same configuration (checkpoint

Re: Pyflink tutorial output

Ah, there they are. Thanks! On Tue, Mar 23, 2021 at 10:26 PM Dian Fu wrote: > How did you check the output when submitting to the kubernetes session > cluster? I ask this because the output should be written to the local > directory “/tmp/output” on the TaskManagers where the jobs are running

Re: flink sql count distonct 优化

Hi,guomuhua 开启本地聚合，是不需要自己打散进行二次聚合的哈，建议看看官方的文档介绍。 Best, Robin guomuhua wrote > 在SQL中，如果开启了 local-global 参数：set > table.optimizer.agg-phase-strategy=TWO_PHASE; > 或者开启了Partial-Final 参数：set table.optimizer.distinct-agg.split.enabled=true; > set >

yarn模式下应用的core/hdfs/yarn site不生效

hi all 如果使用yarn-cluster模式提交认为，并没有将应用下的hdfs-site coresite yarnsite上传依赖，而是使用yarn集群的默认配置。如果yarn集群的默认配置和应用的配置存在差异化，目前只能通过-yt解决。 为什么不上传这些依赖，只上传了flink-conf.yaml？

Re: Fault Tolerance with RocksDBStateBackend

Ok, thank you, Guowei Ma -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: FileSystemTableSink支持自定义分隔符写入

通过设置properties（csv.field-delimiter）可以指定分隔符。不过只能指定单字符。多字符分割不支持。 -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Do data streams partitioned by same keyBy but from different operators converge to the same sub-task?

Let me make the example more concrete. Say O1 gets as input a data stream T1 which it splits into two using some function and produces DataStreams of type T2 and T3, each of which are partitioned by the same key function TK. Now after O2 processes a stream, it could sometimes send the stream to O3

回复: flink 1.12.0 k8s session部署异常

我也遇到这个问题，集群可以提交正常提交任务，但是jobmanager的日志一直有这个, 请问可有办法解决？ 发件人： casel.chen 发送时间： 2021-02-07 16:33 收件人： user-zh@flink.apache.org 主题： flink 1.12.0 k8s session部署异常 在k8s上部署sesson模式的flink集群遇到jobmanager报如下错误，请问这是什么原因造成的？要如何fix? 2021-02-07 08:21:41,873 INFO

Re: Flink 消费kafka ，写ORC文件

谢谢回复 简单实现了一下BucketAssigner，可以实现需求 @Override public String getBucketId(Map element, Context context) { if(context.timestamp() - context.currentProcessingTime() < 0) { return "dt="+context.timestamp(); }

Fault Tolerance with RocksDBStateBackend

Hi everyone, I want to build a flink cluster with 3 machines. What if I choose RocksDBStateBackend with next settings: #== # Fault tolerance and checkpointing

退订

退订中文邮件列表

Re: Do data streams partitioned by same keyBy but from different operators converge to the same sub-task?

For an example of a similar join implemented as a RichCoFlatMap, see [1]. For more background, the Flink docs have a tutorial [2] on how to work with connected streams. [1] https://github.com/apache/flink-training/tree/master/rides-and-fares [2]

Re: DataDog and Flink

Hi Vishal, REST API is the most direct way to get through all metrics as Matthias pointed out. Additionally, you could also add a JMX reporter and log to the machines to check. But in general, I think you are on the right track. You need to reduce the metrics that are sent to DD by configuring

Re: DataDog and Flink

Hi Vishal, what about the TM metrics' REST endpoint [1]. Is this something you could use to get all the metrics for a specific TaskManager? Or are you looking for something else? Best, Matthias [1]

Re: Do data streams partitioned by same keyBy but from different operators converge to the same sub-task?

1. yes - the same key would affect the same state variable 2. you need a join to have the same operator process both streams Matthias On Wed, Mar 24, 2021 at 7:29 AM vishalovercome wrote: > Let me make the example more concrete. Say O1 gets as input a data stream > T1 > which it splits into

Re: Flink Streaming Counter

Hi Vijayendra, what about the example from the docs you already referred to [1]? Best, Matthias [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/metrics.html#counter On Tue, Mar 23, 2021 at 6:48 PM Vijayendra Yadav wrote: > Hi Pohl, > > Thanks for getting back to me so quickly.

FileSystemTableSink支持自定义分隔符写入

Hi，有个疑问： FileSystemTableSink 目前看是有csv和json的普通文本写入。 后续能支持配置 “字段分隔符” 写入文件系统吗？

Re:Re: About Memory Spilling to Disk in Flink

Hi, Guowei Ma. As far as I know, flink writes some in-memory data to disk when memory is running low. I noticed that flink uses ExternalSorterBuilder for batch operations in the org.apache.flink.runtime.operator.sort package, but I'm curious to confirm if this technique is also used in

Re: interval join 如何用 process time

你好，DataStream API 中的 Interval Join 目前还不支持 process time，参考 [1]. 不过如果不要去严格准确的 process time 的话，是否可以在 Join 之前把 process time 用某个字段带出来，当 event time 用？ [1]. https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/operators/joining.html#interval-join -- Sent from:

Re: About Memory Spilling to Disk in Flink

Hi, Roc Could you explain more about your question? Best, Guowei On Wed, Mar 24, 2021 at 2:47 PM Roc Marshal wrote: > Hi, > > Can someone tell me where flink uses memory spilling to write to disk? > Thank you. > > Best, Roc. > > > >

About Memory Spilling to Disk in Flink

Hi, Can someone tell me where flink uses memory spilling to write to disk? Thank you. Best, Roc.

Re: Fault Tolerance with RocksDBStateBackend

Hi, You need some persistent storages(like hdfs) for the checkpoint. It is Flink's fault tolerance prerequisites.[1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/stream/state/checkpointing.html#prerequisites Best, Guowei On Wed, Mar 24, 2021 at 1:21 PM Maminspapin

Re: Kubernetes HA - attempting to restore from wrong (non-existing) savepoint

Hi Alexey, >From your attached logs, I do not think the new start JobManager will recover from the wrong savepoint. Because you could find the following logs to indicate that the HA related ConfigMaps have been cleaned up successfully. 1393 {"ts":"2021-03-20T02:02:18.506Z","message":"Finished