Re: presto s3p checkpoints and local stack

Hi Marco, ideally you solve everything with IAM roles, but you can also use credentials providers such as EnvironmentVariableCredentialsProvider[1]. The key should be s3.aws.credentials.provider: com.amazonaws.auth.EnvironmentVariableCredentialsProvider Remember to put the respective jar into

Re: Flink SQL and checkpoints and savepoints

I went through a few of the recent Flink Forward videos and didn't see solutions to this problem. It sounds like some companies have solutions but they didn't talk about them in enough detail to do something similar. On Thu, Jan 28, 2021 at 11:45 PM Dan Hill wrote: > Is this savepoint recovery

Re: [ANNOUNCE] Apache Flink 1.10.3 released

Thanks Xintong for being our release manager and everyone else who made the release possible! Best Regards, Yu On Fri, 29 Jan 2021 at 15:05, Xintong Song wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.10.3, which is the third bugfix release for

Re: [ANNOUNCE] Apache Flink 1.10.3 released

Thanks Xintong for being our release manager and everyone else who made the release possible! Best Regards, Yu On Fri, 29 Jan 2021 at 15:05, Xintong Song wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.10.3, which is the third bugfix release for

Re: Flink SQL and checkpoints and savepoints

Is this savepoint recovery issue also true with the Flink Table API? I'd assume so. Just doublechecking. On Mon, Jan 18, 2021 at 1:58 AM Timo Walther wrote: > I would check the past Flink Forward conference talks and blog posts. A > couple of companies have developed connectors or modified

Re: Publish heartbeat messages in all Kafka partitions

Hi Alexey, I don't see a way to do it with one message in FlinkKafkaProducer. So you have to multiply the heartbeat yourself. I'd imagine the easiest way would be to query the number of partitions of the output topic (it's static in Kafka) in the heartbeat producer and already produce as all

Re: Deduplicating record amplification

Hi Rex, there cannot be any late event in processing time by definition (maybe on a quantum computer?), so you should be fine. The timestamp of records in processing time is monotonously increasing. Best, Arvid On Fri, Jan 29, 2021 at 1:14 AM Rex Fenley wrote: > Switching to

[ANNOUNCE] Apache Flink 1.10.3 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.10.3, which is the third bugfix release for the Apache Flink 1.10 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

退订

[ANNOUNCE] Apache Flink 1.10.3 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.10.3, which is the third bugfix release for the Apache Flink 1.10 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: 提交job的命令，./bin/flink run-application -t yarn-application ... 和 ./bin/flink run -m yarn-cluster ...

应该说是否：1.11和1.12这里这两种提交方式 是不是一样的，只不过命令有了变化？ 官网中的摘录如下： flink-1.11: Run a single Flink job on YARN Example: ./bin/flink run -m yarn-cluster ./examples/batch/WordCount.jar -- flink-1.12: Per-Job Cluster Mode Example: ./bin/flink run -t

Re: reduce函数的trigger问题

窗口没有结束，所有的数据都还在的 xiaolail...@163.com 于2021年1月29日周五 上午11:27写道： > 您好！最近刚开始学习flink，问一个关于trigger的问题： > > 如下的reduce操作： > env.socketTextStream("localhost", ) > .flatMap(new Splitter()) > .keyBy(value -> value.f0) >

回复：关于Flink作业的负载监控 task-load指标

滴滴文章描述如下： 我们一直希望能精确衡量任务的负载状况，使用反压指标指标只能粗略的判断任务的资源够或者不够。 结合新版的 Mailbox 线程模型，所有互斥操作全部运行在 TaskThread 中，只需统计出线程的占用时间，就可以精确计算任务负载的百分比。 未来可以使用指标进行任务的资源推荐，让任务负载维持在一个比较健康的水平 在2021年01月29日 11:59，1305332<1305...@163.com> 写道： Hi,everyone: 看到滴滴的一篇文章中指出task-load指标，这个线程的占用时间 具体该怎么统计呢？求大神指点

关于Flink作业的负载监控 task-load指标

Hi,everyone: 滴滴的一篇文档中提到： "我们一直希望能精确衡量任务的负载状况，使用反压指标指标只能粗略的判断任务的资源够或者不够。结合新版的 Mailbox 线程模型，所有互斥操作全部运行在 TaskThread 中，只需统计出线程的占用时间，就可以精确计算任务负载的百分比。 未来可以使用指标进行任务的资源推荐，让任务负载维持在一个比较健康的水平” 关于统计出线程的占用时间，这个具体该怎么做呢？

Re: kafka 所有分区无数据的情况下，导致watermark无法前进

所有分区无数据，为什么还期望watermark推进呢？目的是啥。貌似没啥需要计算的呀。 LakeShen 于2021年1月28日周四 下午7:42写道： > 如果是窗口类聚合，可以尝试一下自定义窗口 Trigger > > Best, > LakeShen > > 林影 于2021年1月28日周四 下午5:46写道： > > > Hi, Jessica.J.Wang > > 开源flink看起来没这个功能哈，文档翻了一遍没找到 > > > > Jessica.J.Wang 于2021年1月28日周四 下午5:25写道： > > > > > 你使用的是什么窗口呢，是

提交job的命令，./bin/flink run-application -t yarn-application ... 和 ./bin/flink run -m yarn-cluster ...

如题，在 ./flink --help中看到提交job的命令有两个相似的，貌似都会将jobManager发送yarn node上去之行，但不明白他们区别，官网也未找到他们的区别，请帮忙解释下他们之间的区别？ -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: reduce函数的trigger问题

此处的trigger你可以理解成是计算结果的输出时机，你的reduce计算还是增量的哈。 xiaolail...@163.com 于2021年1月29日周五 上午11:27写道： > 您好！最近刚开始学习flink，问一个关于trigger的问题： > > 如下的reduce操作： > env.socketTextStream("localhost", ) > .flatMap(new Splitter()) > .keyBy(value -> value.f0) >

Re: 为什么window的reduceFunction不支持RichFunction呢

不是，flink是提供了richReduce，但不支持基于window的richReduce。 基于window的reduce只支持简单reduce，具体需要做自定义状态计算的，按照注释要求使用windowFunction和ProcessWindowFunction去做呀。 一直都是这样，1.12也是的哈。 Kezhu Wang 于2021年1月29日周五 上午11:40写道： > reduceFunction 是和 ReducingStateDescriptor 配合的， 并不是 > “window的”。“WindowOperator” 的 function 是

关于Flink作业的负载监控 task-load指标

Hi,everyone: 看到滴滴的一篇文章中指出task-load指标，这个线程的占用时间 具体该怎么统计呢？求大神指点 | | 1305332 | | 1305...@163.com | 签名由网易邮箱大师定制

Re: Flink 1.11 job hit error "Job leader lost leadership" or "ResourceManager leader changed to new address null"

The ZK client side uses 15s connection timeout and 60s session timeout in Flink. There's nothing similar to a heartbeat interval configured, which I assume is up to ZK's internal implementation. These things have not changed in FLink since at least 2017. If both ZK client and server complain

Hi??

Hi dear friend. how are you doing today and how is business moving? i contact to buy from your company kindly send me your latest catalog. also inform me about the 1)Minimum Order Quantity, 2)Delivery time or FOB, 3) payment terms warranty. Please contact us via email: Your early reply is highly

Re: 为什么window的reduceFunction不支持RichFunction呢

reduceFunction 是和 ReducingStateDescriptor 配合的， 并不是 “window的”。“WindowOperator” 的 function 是 “InternalWindowFunction”，这个可以是 “RichFunction”。 Flink 中的 state 除了可以绑定 key，也可以绑定 namespace，只是 namespace 并没有直接暴露给用户。如果需要的话，可以自己写个 WindowOperator，暴露个 WindowFunction。 Interface WindowFunction { // You could

Re: 为什么window的reduceFunction不支持RichFunction呢

Hi, nobleyd, 请问你是在哪个版本发现 reduceFunction 不支持 RichFunction 呢？ 我在1.12 版本试了一下是支持的呀，而且 JavaDoc 里也有 RichReduceFunction 类[1] [1]. https://ci.apache.org/projects/flink/flink-docs-release-1.12/api/java/org/apache/flink/api/common/functions/RichReduceFunction.html -- Sent from:

reduce函数的trigger问题

您好！最近刚开始学习flink，问一个关于trigger的问题： 如下的reduce操作： env.socketTextStream("localhost", ) .flatMap(new Splitter()) .keyBy(value -> value.f0) .window(TumblingEventTimeWindows.of(Time.seconds(15))) .reduce(new ReduceFunction>() {

Re: 对于 Flink 可撤回流做聚合，当上游数据撤回时，最终聚合值变小了，怎么解决这种问题。

要看一下具体的Sql 或者具体的算子 下游的 sink needRetract=false的情况，有些场景可以抑制上游算子的回撤，Retract 可以优化成 Upsert -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: 使用DataStream API + HBaseRowInputFormat读取HBase时表时，Row arity of from (2) does not match this serializers field length (1)

可以参照一下 HBaseTableSource 里面的实现方法 HBaseTableSchema hbaseSchema = new HBaseTableSchema(); hbaseSchema.addColumn(xxx) hbaseSchema.setRowKey(xxx); execEnv.createInput(new HBaseRowInputFormat(conf, tableName, hbaseSchema), getReturnType()) .name(explainSource()); -- Sent

Re:对于 Flink 可撤回流做聚合，当上游数据撤回时，最终聚合值变小了，怎么解决这种问题。

以前做过，自定义sink，更新值小于 存储值的时候不更新 在 2021-01-25 16:00:28，"LakeShen" 写道： >Hi 社区， > >之前遇到一个问题，在 Flink 里面，对于一个数据流（能够撤回的）进行聚合时，当这个数据流有大量撤回时，最终聚合值会变小。这种情况之前有看到说加一个 >mini batch 来减轻这种情况的影响，还有别的方法能够解决这个问题吗? > >Best, >LakeShen

Re: 怎么理解 tolerableCheckpointFailureNumber

Hi, tolerableCheckpointFailureNumber 限制的是最大可容忍的连续失败checkpoint计数 continuousFailureCounter [1]，例如将tolerableCheckpointFailureNumber 设置成3，连续失败3次，continuousFailureCounter 会累计到3，作业就会尝试重启。 如果中间有一个checkpoint成功了，continuousFailureCounter 就会重置为零 [2]。

怎么理解 tolerableCheckpointFailureNumber

tolerableCheckpointFailureNumber 是设置可容忍的checkpoint失败次数，具体怎么理解呢？比如 设置为3 1. 当checkpoint 失败时，该值+1，直到 大于 3，实时作业就发生失败或重启？ 2. 当checkpoint 失败时，是立即进行下个checkpoint？还是根据周期设置自动触发？ 3. 该值是累加值吗

Re: Initializing broadcast state

Hi Nick Following is an example(could not run but just to explain the idea). I use the `KeyedBroadcastProcessFunction` because I saw your code use the keyedstate. private static class StatefulFunctionWithKeyedStateAccessedOnBroadcast extends KeyedBroadcastProcessFunction { private

Configuring ephemeral storage limits when using Native Kubernetes

Hello, I'm trying to run Flink on Kubernetes, and I recently switched from lyft/flinkk8soperator to the Flink Native Kubernetes deployment mode. I have a long running job, that I want to deploy (using application mode), and after a few hours, I noticed the deployment was disappearing. After a

Publish heartbeat messages in all Kafka partitions

Hello, We need to publish heartbeat messages in all topic partitions. Is possible to produce single message and then somehow broadcast it to all partitions from FlinkKafkaProducer? Or only way that message source knows number of existing partitions and sends 1 message to 1 partition? Thanks,

Re: Deduplicating record amplification

Switching to TumblingProcessingTimeWindows seems to have solved that problem. For my own understanding, this won't have any "late" and therefore dropped records right? We cannot blindly drop a record from the aggregate evaluation, it just needs to take all the records it gets in a window and

flink checkpoints adjustment strategy

I am kind of stuck in determining how large a checkpoint interval should be. Is there a guide for that? If a timeout time is 10 minutes, we time out, what is a good strategy for adjusting that? Where is a good starting point for a checkpoint? How shall they be adjusted? We often see checkpoint

Flink on Kubernetes, Task/Job Manager Recycles

Hello - I am running some testing with flink on Kubernetes. Every let's say five to ten days, all the jobs disappear from running jobs. There's nothing under completed jobs, and there's no record of the submitted jar files in the cluster. In some manner or another, it is almost like going into

[Stateful Functions] Problems with Protobuf Versions

Hi, I have a bit of a strange problem: I can't get a Statefun Application to Compile or Run (Depending on the exact Protobuf version) with a Protobuf version newer than 3.3.0. I have had this problem over multiple project setups and multiple versions of Flink Statefun with Java8. Protobuf

Re: Deduplicating record amplification

It looks like it wants me to call assignTimestampsAndWatermarks but I already have a timer on my window which I'd expect everything entering this stream would simply be aggregated during that window .window(TumblingEventTimeWindows.of(Time.seconds(1))) On Thu, Jan 28, 2021 at 12:59 PM Rex Fenley

Re: Deduplicating record amplification

I think I may have been affected by some late night programming. Slightly revised how I'm using my aggregate val userDocsStream = this.tableEnv .toRetractStream(userDocsTable, classOf[Row]) .keyBy(_.f1.getField(0)) val compactedUserDocsStream = userDocsStream

Re: Flink 1.11 job hit error "Job leader lost leadership" or "ResourceManager leader changed to new address null"

After checking the log I found the root cause is zk client timeout on TM: ``` 2021-01-25 14:01:49,600 WARN org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 40020ms for sessionid 0x404f9ca531a5d6f 2021-01-25 14:01:49,610

question on checkpointing

Is it possible that checkpointing times out due to an operator taking too long? Also, does windowing affect the checkpoint barriers?

Re: Timers not firing until stream end

Chesnay, 1) Correct, I'd like the timeout event (generated at eventTime==1000) to appear in its correct time sequence in the output, i.e. before eventTime exceeds 1000. It's great that Flink can deal with out-of-orderness, but I didn't expect it to spontaneously create it (especially with

AW: Stateful Functions - accessing the state aside of normal processing

Hi Gordon, If operating on checkpoints instead of savepoints this might be OK. But since this is not in the current scope I digged into Flink docs and found the "queryable state"

Re: presto s3p checkpoints and local stack

Is it possible to use an environmental credentials provider? On Thu, Jan 28, 2021 at 8:35 AM Arvid Heise wrote: > Hi Marco, > > afaik you don't need HADOOP_HOME or core-site.xml. > > I'm also not sure from where you got your config keys. (I guess from the > Presto page, which probably all work

Very slow recovery from Savepoint

Hello there, I am trying to find the solution for a problem we are having in our Flink setup related to very slow recovery from a Savepoint. I have searched in the mailing list, found a somewhat similar problem, the bottleneck there was the HD usage, but I am not seeing this in our case. Here is a

Connect to schema registry via SSL

Hello, I'm trying to us Flink SQL (on Ververica Platform, so no other options than pure Flink SQL) to read confluent avro messages from Kafka, when the schema registry secured via SSL. Would you know what are the correct properties to setup in the kafka consumer config? The following options

Re: presto s3p checkpoints and local stack

Hi Marco, afaik you don't need HADOOP_HOME or core-site.xml. I'm also not sure from where you got your config keys. (I guess from the Presto page, which probably all work if you remove hive., maybe we should also support that) All keys with prefix s3 or s3p (and fs.s3, fs.s3p) are routed

Re: What causes a buffer pool exception? How can I mitigate it?

> > Regarding the try catch block Sorry I meant the try catch in SensorMessageToSensorTimeSeriesFunction. Also, just to be clear, does disabling restart make it easier for you to > debug? > Yes the log will be quite small then. Currently, it's just repeating the same things a couple of times.

Re: HybridMemorySegment index out of bounds exception

FYI: Yuval and I scheduled a call to investigate this serialization issue remotely on Monday. If you have any idea by looking at the code beforehand, let us know. On 28.01.21 16:57, Yuval Itzchakov wrote: Hi Timo, The code example I posted doesn't really match the code that is causing this

presto s3p checkpoints and local stack

Hi, I got s3a working on localstack. The missing piece of information from Flink documentation seems to be that the system requires a HADOOP_HOME and core-site.xml. Flink documentation states that s3p (presto) should be used for file checkpointing into s3. I am using RocksDB, which I assume also

Re: HybridMemorySegment index out of bounds exception

Hi Timo, The code example I posted doesn't really match the code that is causing this issue. I tried to extend it a bit but couldn't make the reproduction work there. I am no longer using the serialized strings, but registering the custom serializers with the runtime during bootstrap and

Re: HybridMemorySegment index out of bounds exception

This is helpful information. So I guess the problem must be in the flink-table module and not in flink-core. I will try to reserve some time tomorrow to look into the code again. How did you express RawType(Array[String])? Again with fully serialized type string? Could it be related to

Re: HybridMemorySegment index out of bounds exception

Hi Timo, I tried replacing it with an ordinary ARRAY DataType, which doesn't reproduce the issue. If I use a RawType(Array[String]), the problem still manifests, so I assume it's not directly related to a Kryo serialization of the specific underlying type (io.circe.Json), but something in the way

Re: HybridMemorySegment index out of bounds exception

Hi Yuval, we should definitely find the root cause of this issue. It helps if the exception happens frequently to nail down the problem. Have you tried to replace the JSON object with a regular String? If the exception is gone after this change. I believe it must be the serialization and

Re: What causes a buffer pool exception? How can I mitigate it?

Regarding the try catch block, it rethrows the exception. Here is the code: catch (RuntimeException e) { logger.error("Error in timer.", e); throw e; } That would be okay, right? Also, just to be clear, does disabling restart make it easier for you to debug? On Thu, Jan 28, 2021 at 1:17 AM

Question

Hi, Is there any library to use and remember the apache flink snapshot? Thank you -- Regards, Abu Bakar Siddiqur Rahman

Re: key group from xx to yy does not contain zz异常

感谢老师解答，keyBy的执行逻辑看来我理解的太肤浅了。随机数生成逻辑在keyBy前通过map赋值到具体字段，保证后续keyby时稳定，应该就对了。再次感谢老师指点迷津。 -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Timers not firing until stream end

I'm not sure I see the problem in your output. For any given key the timestamps are in order, and the events where devices are offline seem to occur at the right time. Is it just that you'd like the following line to occur earlier in the output?

使用DataStream API + HBaseRowInputFormat读取HBase时表时，Row arity of from (2) does not match this serializers field length (1)

Hi: 您好，我在使用DataStream API 读取HBase表时，使用了HBaseRowInputFormat，并根据HBaseTableSchema了schema，代码如下： val env = StreamExecutionEnvironment.getExecutionEnvironment val hbaseTableSchema = TableSchema.builder() .add(TableColumn.of("id", DataTypes.STRING())) .add(TableColumn.of("f1",

为什么window的reduceFunction不支持RichFunction呢

问题如title，当然当前Flink的API还是可以实现RichReduceFunction的效果的，就是基于windowFunction或者processWindowFuntion。 但是，windowFunc和reduceFunc最大的区别在于windowFunc是窗口触发操作，而reduce是实时的增量操作。 如果说我的窗口计算对于每个record的到来都需要一个极复杂的操作，我更希望在reduce中完成，而不是windowFunc中完成，因为那样会导致整个窗口所有key的数据几乎在同一时刻触发，这回导致压力变高。

Cannot access state from a empty taskmanager - using kubernetes

Hi, We have followed the instructions in the following link ""Enabling Queryable State" with kubernetes: https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/standalone/kubernetes.html#enabling-queryable-state *When the replicas of the task-manager pods is 1 we

Re: key group from xx to yy does not contain zz异常

Hi， 原因是你的key selector引入了随机变量 （也就是下面的方法keyBy），导致其select出来的key不是固定的 public KeySelector keyBy(int parallelism) { return value -> Joiner.on(SeparatorConstant.BAR).join(value.getMetricsId(), ThreadLocalRandom.current().nextInt(parallelism)); } 例如原先的key selector选出的key是

Re:Re: flink-1.12 通过-t指定模式后无法指定yarn参数

好的 感谢！ 在 2021-01-28 15:52:36，"silence" 写道： >flink1.12后所有的yarn相关的参数通过-D进行指定 >例：-D yarn.application.name=xxx 替代以前的-ynm xxx >更多配置参考文档https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html#yarn > > > >-- >Sent from: http://apache-flink.147419.n8.nabble.com/

Problem restirng state

All, We are getting the exception, copied at the end of this post. The exception is thrown when a new flink job is submitted; when Flink tries to restore the previous state. Environment: Flink version: 1.10.1 State persistence: Hadoop 3.3 Zookeeper 3.5.8 Parallelism: 4 The code

Re: kafka 所有分区无数据的情况下，导致watermark无法前进

如果是窗口类聚合，可以尝试一下自定义窗口 Trigger Best, LakeShen 林影 于2021年1月28日周四 下午5:46写道： > Hi, Jessica.J.Wang > 开源flink看起来没这个功能哈，文档翻了一遍没找到 > > Jessica.J.Wang 于2021年1月28日周四 下午5:25写道： > > > 你使用的是什么窗口呢，是 Tumble或者Hop吗，如果没数据 但是想提前输出结果可以用 emit > > > > >

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Applied that parameter and that seems to get me some progress here. I still get the shade overlapping classes warning, but I get the PostgreSQLTableFactory in the merged table.factories.Factory service file. However, now on runtime the application fails to find the debezium source function class

关于flink-shaded-xxx的问题

如题，我想知道flink shade了多个包，比如jackson，guava等。 其目的是（1）flink用到这些，为了避免冲突所以shade。还是（2）flink推荐用户直接使用flink shade好的这些包？ 如上，我想知道是否“推荐”用户直接使用flink shade的这些包。还是我们自己去依赖自己的包，比如我当前就用到了jackson，以及guava（我直接用了最新的30-jre的版本）。

Apache Flink Job Manager High CPU with Couchbase

Hello, We work in a multinational company that produces diesel engines and is working on an IoT platform to analyze engine performance based on sensor data. We are using Flink for deploying analytics stream processing jobs. We recently integrated these jobs with Couchbase (serving as a Cache)

Re: Flink1.12 批处理模式，分词统计时单词个数为1的单词不会被打印

你用的应该是 1.12.0 版本吧。这是一个已知问题 [1]，升级到 1.12.1 有修复。 Thank you~ Xintong Song [1] https://issues.apache.org/jira/browse/FLINK-20764 On Thu, Jan 28, 2021 at 4:55 PM xhyan0427 <15527609...@163.com> wrote: > 代码： > val env = StreamExecutionEnvironment.getExecutionEnvironment > >

key group from xx to yy does not contain zz异常

线上部署flink项目时，启动爆如下错误，在测试环境可正常启动，job依赖的flink版本是1.10，flink 集群版本是1.12-SNAPSHOT，与线上一致。有点摸不着头脑，麻烦各位老师帮分析分析 堆栈信息： java.lang.IllegalArgumentException: key group from 44 to 45 does not contain 4 at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:161) at

回复： 关于端到端的延迟监控

谢谢大家，清楚了 | | 1305332 | | 1305...@163.com | 签名由网易邮箱大师定制 在2021年01月28日 17:56，Jessica.J.Wang 写道： 官方的LatencyMarker 表示的是数据的流通性， 他和数据是在同一个pipeline 中顺序处理的，如果你的算子都是同步的情况 是可以反应出数据的真实处理延迟，生产上是可以使用的，但是 延迟粒度 metrics.latency.granularity 最好调整成 single或者 operator ,防止latency上报太多 压垮服务 但当你的算子是个异步

检查点成功，但从检查点恢复失败。使用了guava的bloomFilter，有人帮忙分析下吗？

如下，我使用到了Guava的BloomFilter，貌似是基于kyro序列化的。检查点成功了，但是基于检查点恢复任务是失败的。 报错堆栈如下，关键错误是什么无法访问public修饰的成员？ java.lang.Exception: Exception while creating StreamOperatorStateContext. at org.apache.flink.streaming.api.operators. StreamTaskStateInitializerImpl.streamOperatorStateContext(

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Hi Sebastián, Could you try to add combine.children="append" attribute to the transformers configuration? You can also see the full shade plugin configuration here [1]. Best, Jark [1]:

Re: 关于端到端的延迟监控

官方的LatencyMarker 表示的是数据的流通性， 他和数据是在同一个pipeline 中顺序处理的，如果你的算子都是同步的情况 是可以反应出数据的真实处理延迟，生产上是可以使用的，但是 延迟粒度 metrics.latency.granularity 最好调整成 single或者 operator ,防止latency上报太多 压垮服务 但当你的算子是个异步 用AsyncWaitOperator实现的话，因为latencyMarker并没有像watermark一样 addToWorkQueue,直接处理上报metrics,所以延迟信息就不准确了

Re: JobManager seems to be leaking temporary jar files

Code-wise it appears that thing have gotten simpler and we can use use a URLClassLoader within PackagedProgram. We probably won't get around a dedicated close() method on the PackagedProgram. I think in FLINK-21164 I think have identified the right places to issue this call within the jar

Re: Conflicts between the JDBC and postgresql-cdc SQL connectors

Hi Jark! Please find the full pom file attached. Best Regards, On Thu, 28 Jan 2021 at 03:21, Jark Wu wrote: > Hi Sebastián, > > I think Dawid is right. > > Could you share the pom file? I also tried to > package flink-connector-postgres-cdc with ServicesResourceTransformer, and > the Factory

Re: What causes a buffer pool exception? How can I mitigate it?

Also could you please provide the jobmanager log? It could also be that the underlying failure is somewhere else. On Thu, Jan 28, 2021 at 10:17 AM Arvid Heise wrote: > Hi Marco, > > In general, sending a compressed log to ML is totally fine. You can > further minimize the log by disabling

Re: kafka 所有分区无数据的情况下，导致watermark无法前进

你使用的是什么窗口呢，是 Tumble或者Hop吗，如果没数据 但是想提前输出结果可以用 emit https://help.aliyun.com/document_detail/98951.html?spm=5176.11065259.1996646101.searchclickresult.513d1037M5VADa -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: What causes a buffer pool exception? How can I mitigate it?

Hi Marco, In general, sending a compressed log to ML is totally fine. You can further minimize the log by disabling restarts. I looked into the logs that you provided. 2021-01-26 04:37:43,280 INFO org.apache.flink.runtime.taskmanager.Task >[] - Attempting to cancel task forward

Flink1.12 批处理模式，分词统计时单词个数为1的单词不会被打印

代码： val env = StreamExecutionEnvironment.getExecutionEnvironment env.setRuntimeMode(RuntimeExecutionMode.BATCH) // 在DataStream API上以批处理方式执行 // 本地测试文件 val inputStream = env.readTextFile(getClass.getResource("/hello.txt").getPath) // 分词统计，问题：批处理模式的时候，sum 为 1 的单词不会被打印

Re: Timers not firing until stream end

Scratch that - your WatermarkStrategy DOES work (when I implement it correctly!). Well, almost: As you can see below (code pushed to repo), the Timer events are still appearing somewhat late in the stream - 4 events late in this case. It may be just good-enough for my purposes, though it will make

Re: Timers not firing until stream end

Chesnay, I cannot reproduce this - I've tried the approaches you suggest, but nothing I've done makes the timers fire at the correct time in the stream - they only fire when the stream has ended. If you have an EventTime example where they fire at the right time in the stream, I'd love to see it.

Re: flink slot communication

Hi, Yes Dawid is correct. Communications between two tasks on the same TaskManager are not going through the network, but via "local" channel (`LocalInputChannel`). It's still serialising and deserializing the data, but there are no network overheads, and local channels have only half of the

Re: 关于端到端的延迟监控

每一条records处理过程中透传开始时间，在sink算子通过metrics上报opentsdb、Prometheus 等时间序列数据库，最后通过grafana等可视化工具展示。 wpp <1215303...@qq.com> 于2021年1月28日周四 下午2:53写道： > 这个延迟，只是给一个参考意义吧，

Re: kafka 所有分区无数据的情况下，导致watermark无法前进

Hi，刘小红： 这个我查阅过，实验过，通过idle source 是无法解决的。 可以查看这个链接：http://apache-flink.147419.n8.nabble.com/Flink-SQL-td4535.html 在我的这个场景里面，上游已经配置了idle source，上游如果突然没有数据了，下游的flink 窗口还是无法关闭输出结果。 这个说明idle source 无法解决这个问题 刘小红 <18500348...@163.com> 于2021年1月28日周四 下午3:01写道： >