static table in flink

Hello, I have questions about static table in flink. Join the stream table with static table. I'm looking at temporal table, while the time based table would grow exponentially over period. Any suggestions? Stream tables checks the contains in static table(updates once everyday with new set of

Re: Flink vs Kafka streams

Thanks Congxian. Yes, its been very hard to manage the cluster and thats why we are trying to evaluate alternate choices. If anyone has found better methods to deploy and scale, it would be great to know so that we can adopt the same as well. Thanks On Fri, Nov 8, 2019 at 1:56 AM Congxian Qiu

Flink DataStream KeyedStream 与 AggregateFunction

请教下: 1、DataStream 如按用户ID KeyBy后，同一个用户ID的数据最终会被分到一个Partition中吗？ 2、假设1成立，这样就会有数据倾斜的问题。该如何解决？ 3、假设1成立，如: DataStream .keyBy(userID) .timeWindow() .aggregate(new

Re: flink on yarn-cluster kerberos authentication for hbase

Hi Can you share more information regarding how you currently setup your Kerberos that only works with Zookeeper? Does your HBase share the same KDC? -- Rong On Fri, Nov 8, 2019 at 12:33 AM venn wrote: > Thanks, I already seen, not work for me > > > > *发件人**:* Jaqie Chan > *发送时间:* Friday,

Re: Unable to retrieve Kafka consumer group offsets

Thanks Till, The link the the thread is here: http://mail-archives.apache.org/mod_mbox/flink-user/201702.mbox/%3ccagr9p8cekzvx3fxqi363o3tbd37nuppe7l_+c7+kx_z_nqy...@mail.gmail.com%3E On Fri, Nov 8, 2019 at 5:18 AM Till Rohrmann wrote: > Hi Harrison, > > which mailing list thread are you

Re: Flink Filters have state?

Thanks Till! This was helpful! Tim On Fri, Nov 8, 2019, 7:16 AM Till Rohrmann wrote: > Hi Tim, > > 1. The filter is stored within the JobGraph which is persisted to a > persistent storage if HA is enabled. Usually, this is either HDFS, S3 or > any other highly available file system. It is

Re: Flink SQL dynamic configuration

Thanks for your helps Till. I appreciate it. On Fri, Nov 8, 2019 at 9:02 PM Till Rohrmann wrote: > Hi Jaqie, > > not sure whether this is easily possible with Flink's SQL API but if you > used the DataStream API directly you could create a connected stream where > you have two inputs. One input

Re: Unable to retrieve Kafka consumer group offsets

Hi Harrison, which mailing list thread are you referring to exactly? I've pulled in Becket who might be able to tell you more about the Kafka connector. Cheers, Till On Thu, Nov 7, 2019 at 11:11 PM Harrison Xu wrote: > I am using Flink 1.9.0 and KafkaConsumer010 (Kafka 0.10.1.1). Attempting >

Re: Flink Filters have state?

Hi Tim, 1. The filter is stored within the JobGraph which is persisted to a persistent storage if HA is enabled. Usually, this is either HDFS, S3 or any other highly available file system. It is just a serialized POJO. If you want to store your filter's state you would need to use Flink's state

Re: Flink SQL dynamic configuration

Hi Jaqie, not sure whether this is easily possible with Flink's SQL API but if you used the DataStream API directly you could create a connected stream where you have two inputs. One input could be the normal message stream and the other input could be the configuration stream. So whenever there

Re: Does Flink merge streams deterministically?

Hi Amran, in the general case, the merging of two streams is not deterministic. Cheers, Till On Thu, Nov 7, 2019 at 3:24 AM amran dean wrote: > > The Flink Kafka consumer's watermark extractor claims:. > > "The watermark extractor will run per Kafka partition, watermarks will be > merged

Re: Till Rohrmann - Can you please share your code for FF - SF - Flink as a lib

Thanks a lot Till !! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Limit max cpu usage per TaskManager

Hi Lu Niu, Yes, you could use `yarn.containers.vcores` to set the vcores of taskmanager. However, it could not guarantee that the application do not affect each other. By default, the yarn cluster are using cgroup share. That means a taskmanager could use more cpu than it allocated. When the

Re: 在 Trigger 里可以知道 Window 中数据的状况吗

Hi 能否用 KeyedProcessFunction 来实现你的需求呢？ Best, Congxian Utopia 于2019年11月5日周二 下午9:15写道： > 我是用自定义的 Trigger 的，但是在 Trigger 中获取不到 Window 中的数据信息。 > > zhisheng 于2019年11月5日周二 下午8:16写道： > > > 可以自定义 trigger，将两者的条件结合起来才fire > > > > Utopia 于2019年11月5日周二 下午2:19写道： > > > > > 不好意思没有描述清楚，我们业务场景是需要使用

Re: Flink savepoint(checkpoint) recovery dev debug

Hi You can ref CheckpointMetadataLoadingTest#testLoadAndValidateSavepoint for more information. Even more, you can ref the line 136-line145 [1], and init a CompletedCheckpointStorageLocation to make it work. [1]

Re: Flink vs Kafka streams

Hi >From your description, seems the big problem is scale in and out, and there maybe a big downtime for trigger savepoint and restore from the savepoint. Previously, we have proposed a feature named stop-with-checkpoint[1] same as the stop-with-savepoint, but triggering a checkpoint instead of

Re: How can I get the backpressure signals inside my function or operator?

humm, that is also another possibility. Thanks for your suggestion! *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Thu, Nov 7, 2019 at 10:41 PM Yuval Itzchakov wrote: > Hi, > > We've been

回复: flink on yarn-cluster kerberos authentication for hbase

Thanks, I already seen, not work for me 发件人: Jaqie Chan 发送时间: Friday, November 8, 2019 4:14 PM 收件人: venn 抄送: user@flink.apache.org 主题: Re: flink on yarn-cluster kerberos authentication for hbase Hello, Does this have helps to you?

Re: flink on yarn-cluster kerberos authentication for hbase

Hello, Does this have helps to you? https://stackoverflow.com/questions/34596165/how-to-do-kerberos-authentication-on-a-flink-standalone-installation Regards 嘉琪 On Fri, Nov 8, 2019 at 4:00 PM venn wrote: > HI Guys: > > Who can share some example for flink on yarn-cluster kerberos

flink on yarn-cluster kerberos authentication for hbase

HI Guys: Who can share some example for flink on yarn-cluster kerberos authentication for hbase I auth as what I do in java program, it look like just authentication zookeeper, cannot access for hbase thanks