Kafka producer failed with InvalidTxnStateException when performing commit transaction

Hi, Currently, I was trying to update our kafka cluster with larger ` transaction.max.timeout.ms`. The original setting is kafka's default value (i.e. 15 minutes) and I tried to set as 3 hours. When I was doing rolling-restart for my brokers, this exception came to me on the next checkpoint

Fwd: Flink program，Full GC (System.gc()）

Hi Xintong, Thanks for your answer！ I also think that is not a big problem because it’ takes less than 0.5 second。I only want to find what was caused. "JVM also does that automatically, as long as there are continuous activities of creating / destroying objects in heap” I also find some

回复: flink KeyedBroadcastProcessFunction 不能触发onTimer

而且我发现查找了下 KeyedProcessFunction 的onTimer引用对应如下: KeyedBroadcastProcessFunction 的onTimer引用对应如下: KeyedProcessFunction 的onTimer方法签名是这样的: public void onTimer(long timestamp, OnTimerContext ctx, Collector out) throws Exception {} KeyedBroadcastProcessFunction 的onTimer方法签名是这样的: public void

Re: Implementing a low level join

Hi Felipe, > I want to implement a join operator which can use different strategies for joining tuples. Not all kinds of join strategies can be applied to streaming jobs. Take sort-merge join as an example, it's impossible to sort an unbounded data. However, you can perform a window join and use

Re:Re: Re: [Discuss] What should the "Data Source" be translated into Chinese

好的，按照大家的意见统一对“Data Source”和“Data Sink”不翻译。 在 2019-08-14 05:19:13，"Yu Li" 写道： >支持不翻译而在专有名词表中增加注释。 > >从现有Flink翻译规范中的“术语表”来看，对于某些专有名词也是不翻译而只解释含义的，比如checkpoint。对于Data Source和Data >Sink也建议采用同样的处理方式。 > >PS. 这个是关于中文翻译的讨论，不太适合发到@dev邮件列表，希望大家能将讨论范围限定在中文邮件列表内，谢谢！ > >Best Regards, >Yu > > >On Tue, 13 Aug

Re: Making broadcast state queryable?

Thank you for the honest response, Yu! There is so much that comes to mind when we look at Flink as a "application framework" (my talk in Flink Forward in Berlin will be about

Re: Making broadcast state queryable?

Hi Oytun, Sorry but TBH such support will probably not be added in the foreseeable future due to lack of committer bandwidth (not only support queryable broadcast state but all about QueryableState module) as pointed out in other threads [1] [2]. However, I think you could open a JIRA for this

Making broadcast state queryable?

Hi there, Can we set a broadcast state as queryable? I've looked around, not much to find about it. I am receiving UnknownKvStateLocation when I try to query with the descriptor/state name I give to the broadcast state. If it doesn't work, what could be the alternative? My mind goes around

Re: Flink 1.8: Using the RocksDB state backend causes "NoSuchMethodError" when trying to stop a pipeline

Hi Tobias First of all, I think you would not need to ADD the flink-statebackend-rocksdb jar package into your docker image's lib folder, as the flink-dist jar package within lib folder already include all classes of flink-statebackend-rocksdb. I think the root cause is that you might assemble

Implementing a low level join

Hi all, I want to implement a join operator which can use different strategies for joining tuples. I saw that with CoProcessFunction I am able to implement low-level joins [1]. However, I do know how to decide between different algorithms to join my tuples. On the other hand, to do a broadcast

Re: Scylla connector

Scylla is protocol compatible with Cassandra, so you can just use the Cassandra connector. Scylla has extended the Go gocql package to make it shard aware, but such an extension does not exist for the Cassandra Java driver. That just means that the driver will sent requests to any shard on a

Re: Flink program，Full GC (System.gc()）

Hi Andrew, The behavior you described doesn't looks like a problem to me. I mean what are the bad consequences for having a full GC (which takes less than 0.5 second) per hour? The full GC is not necessarily triggered by explicitly calling "System.gc()" in Flink. JVM also does that

Re: Apache flink 1.7.2 security issues

The flink job manager UI isn't meant to be accessed from outside a firewall I think. Plus I dont think it was designed with security in mind and honestly it doesn't need to in my opinion. If you need security then address your network setup. And if it is still a problem the just turn off the

Flink 1.8: Using the RocksDB state backend causes "NoSuchMethodError" when trying to stop a pipeline

Hi, I am using Apache Beam 2.14.0 with Flink 1.8.0 and I have included the RocksDb dependency in my projects pom.xml as well as baked it into the Dockerfile like this: FROM flink:1.8.0-scala_2.11 ADD --chown=flink:flink

Re: Apache flink 1.7.2 security issues

Thanks for reporting this issue. It is already discussed on Flink's dev mailing list in this thread: -> https://lists.apache.org/thread.html/10f0f3aefd51444d1198c65f44ffdf2d78ca3359423dbc1c168c9731@%3Cdev.flink.apache.org%3E Please continue the discussion there. Thanks, Fabian Am Di., 13. Aug.

回复: [Discuss] What should the "Data Source" be translated into Chinese

赞同，想了半天没想到合适的中文翻译data sink -- 发件人：Jeff Zhang 日　期：2019年08月13日 18:20:34 收件人： 抄　送：dafei1...@sina.com; dev 主　题：Re: [Discuss] What should the "Data Source" be translated into Chinese 不建议翻译 Simon Su 于2019年8月13日周二 下午6:18写道： >

Re: I'm not able to make a stream-stream Time windows JOIN in Flink SQL

I wrote a demo example for time windowed join which you can pick up [1] [1] https://gist.github.com/docete/8e78ff8b5d0df69f60dda547780101f1 *Best Regards,* *Zhenghua Gao* On Tue, Aug 13, 2019 at 4:13 PM Zhenghua Gao wrote: > You can check the plan after optimize to verify it's a regular join

Re: [Discuss] What should the "Data Source" be translated into Chinese

Hi, 这里有翻译的Convention, 可以作为参考, 个人建议这里不翻译 https://cwiki.apache.org/confluence/display/FLINK/Flink+Translation+Specifications Zili Chen 于2019年8月13日周二 下午6:59写道： > Source 和 Sink 的通译是 源 和 汇，data source 和 data sink 因此分别对应 数据源 和 > 数据汇。出处可参见中国计算机协会的《计算机科学技术名词》，线上检索在这里[1] > > 但是具体到 FLINK 的情况，实际上我建议所有

Apache flink 1.7.2 security issues

Hello, We are using Apache Flink 1.7.2 version. During our security scans following issues are reported by our scan tool. Please let us know your comments on these issues. [1] 150085 Slow HTTP POST vulnerability Severity Potential Vulnerability - Level 3 Group Information Disclosure Threat

Flink program，Full GC (System.gc()）

Flink Version: 1.8.1 deploy：standalone state.backend.fs.memory-threshold=128k A very very simple flink program and without other jar dependended； But trigger full gc every hour by Full GC (System.gc() in jobmanager Jobmanager I only find this where called System.gc()，but not sure when

Flink program，Full GC (System.gc()）

Flink Version: 1.8.1 deploy：standalone state.backend.fs.memory-threshold=128k A very very simple flink program and without other jar dependended； But trigger full gc every hour by Full GC (System.gc() in jobmanager Jobmanager I only find this where called System.gc()，but not sure when

Re: [Discuss] What should the "Data Source" be translated into Chinese

Source 和 Sink 的通译是 源 和 汇，data source 和 data sink 因此分别对应 数据源 和 数据汇。出处可参见中国计算机协会的《计算机科学技术名词》，线上检索在这里[1] 但是具体到 FLINK 的情况，实际上我建议所有 FLINK 专有名词都不要翻译，翻译反而增加理解难度。 Best, tison. [1] http://www.termonline.cn/index.htm hsw 于2019年8月13日周二 下午6:46写道： > 赞同，想了半天没想到合适的中文翻译data sink > > > >

Re: Apache flink 1.7.2 security issues

Hi! Thank you for reporting this! At the moment, the Flink REST endpoint is not secure in the way that you can expose it publicly. After all, you can submit Flink jobs to it which by definition support executing arbitrary code. Given that access to the REST endpoint allows by design arbitrary

Re: [Discuss] What should the "Data Source" be translated into Chinese

我也更倾向于不翻译 Data Source 和 Data Sink Kurt Young 于2019年8月13日周二 下午6:21写道： > "更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可" +1 > > Best, > Kurt > > > On Tue, Aug 13, 2019 at 6:18 PM Simon Su wrote: > > > 更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可 > > > > > > Thanks, > > SImon > > > > > > On

Re: [Discuss] What should the "Data Source" be translated into Chinese

"更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可" +1 Best, Kurt On Tue, Aug 13, 2019 at 6:18 PM Simon Su wrote: > 更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可 > > > Thanks, > SImon > > > On 08/13/2019 18:07， wrote： > How about translate "data sink" into “数据漕” > 漕，读作：cáo。汉字基本字义指通过水道运输粮食：漕运|漕粮。==>

Re: [Discuss] What should the "Data Source" be translated into Chinese

更倾向不去翻译Data Source和Data Sink, 通过用中文对其做解释即可 Thanks, SImon On 08/13/2019 18:07， wrote： How about translate "data sink" into “数据漕” 漕，读作：cáo。汉字基本字义指通过水道运输粮食：漕运|漕粮。==> https://baike.baidu.com/item/%E6%BC%95?forcehttps=1%3Ffr%3Dkg_hanyu - 原始邮件 - 发件人：Kurt Young 收件人：dev , user-zh

回复：Re: [Discuss] What should the "Data Source" be translated into Chinese

How about translate "data sink" into “数据漕” 漕，读作：cáo。汉字基本字义指通过水道运输粮食：漕运|漕粮。==> https://baike.baidu.com/item/%E6%BC%95?forcehttps=1%3Ffr%3Dkg_hanyu - 原始邮件 - 发件人：Kurt Young 收件人：dev , user-zh 主题：Re: [Discuss] What should the "Data Source" be translated into Chinese 日期：2019年08月13日

Re: [Discuss] What should the "Data Source" be translated into Chinese

可以直接保留不用翻译吗？ Best, Jingsong Lee -- From:WangHengwei Send Time:2019年8月13日(星期二) 11:50 To:user-zh Subject:[Discuss] What should the "Data Source" be translated into Chinese Hi all, I'm working on [FLINK-13405] Translate

Re:[Discuss] What should the "Data Source" be translated into Chinese

Hi all, 更倾向于不翻译 Data Source 和 Data Sink Thanks, SImon On 08/13/2019 17:27，WangHengwei wrote： Hi all, I'm working on [FLINK-13405] Translate "Basic API Concepts" page into Chinese. I have a problem. Usually we translate "Data Source" into "数据源" but there is no agreed translation for

[Discuss] What should the "Data Source" be translated into Chinese

Hi all, I'm working on [FLINK-13405] Translate "Basic API Concepts" page into Chinese. I have a problem. Usually we translate "Data Source" into "数据源" but there is no agreed translation for "Data Sink". Since it often appears in documents, I think we'd better to have a unified

Re: Testing with ProcessingTime and FromElementsFunction with TestEnvironment

Hi Michal, you need to implement a source which does not terminate. Take a look at the InifiteSource [1] which does exactly this. That way there won't be a Long.MAX_VALUE being sent when closing the source operator. [1]

Re: [External] Re: From Kafka Stream to Flink

Hi Do you have an expected version of Flink to include the capability to ingest an upsert stream as a dynamic table? We have such need in our current project. What we have done is to emulate such behavior working at low level with states (e.g. update existing value if key exists, create a new

Re: flink on yarn，提交方式是per job的话，如何保证高可用？

JM is restarted by YARN on failure [1]. [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/jobmanager_high_availability.html#yarn-cluster-high-availability *Best Regards,* *Zhenghua Gao* On Tue, Aug 13, 2019 at 4:51 PM 陈帅 wrote: > 请教一下：flink on yarn，提交方式是per job的话，如何保证高可用？ >

flink on yarn，提交方式是per job的话，如何保证高可用？

请教一下：flink on yarn，提交方式是per job的话，如何保证高可用？

Re: [Discuss] What should the "Data Source" be translated into Chinese

cc user-zh mailing list, since there are lots of chinese speaking people. Best, Kurt On Tue, Aug 13, 2019 at 4:02 PM WangHengwei wrote: > Hi all, > > > I'm working on [FLINK-13405] Translate "Basic API Concepts" page into > Chinese. I have a problem. > > Usually we translate "Data

Re: I'm not able to make a stream-stream Time windows JOIN in Flink SQL

You can check the plan after optimize to verify it's a regular join or time-bounded join(Should have a WindowJoin). The most direct way is breakpoint at optimizing phase [1][2]. And you can use your TestData and create an ITCase for debugging [3] [1]

Re: Flink cannot recognized catalog set by registerCatalog.

OK, Thanks Jark Thanks, SImon On 08/13/2019 14:05，Jark Wu wrote： Hi Simon, This is a temporary workaround for 1.9 release. We will fix the behavior in 1.10, see FLINK-13461. Regards, Jark On Tue, 13 Aug 2019 at 13:57, Simon Su wrote: Hi Jark Thanks for your reply. It’s weird that

Re: Flink cannot recognized catalog set by registerCatalog.

Hi Simon, This is a temporary workaround for 1.9 release. We will fix the behavior in 1.10, see FLINK-13461. Regards, Jark On Tue, 13 Aug 2019 at 13:57, Simon Su wrote: > Hi Jark > > Thanks for your reply. > > It’s weird that In this case the tableEnv provide the api called >