read a finite number of messages from Kafka using Kafka connector without extending it?

Hi, We are considering to use Flink SQL for ad hoc data analytics on real-time Kafka data, and want to limit the queries to process data in the past 5-10 minutes. To achieve that, one possible approach is to extend the current Kafka connect to have it only read messages in a given period of time

Re: StandAlone job on k8s fails with "Unknown method truncate" on restore

Not sure, but it seems this https://issues.apache.org/jira/browse/FLINK-10203 may be a connected issue. On Fri, Feb 15, 2019 at 11:57 AM Vishal Santoshi wrote: > That log does not appear. It looks like we have egg and chicken issue. > > 2019-02-15 16:49:15,045 DEBUG

[Meetup] Apache Flink+Beam+others in Seattle. Feb 21.

Hello everyone, There is an upcoming meetup happening in the Google Seattle office, on February 21st, starting at 5:30pm: https://www.meetup.com/seattle-apache-flink/events/258723322/ People will be chatting about Beam, Flink, Hive, and AthenaX . Anyone who is

Re: StandAlone job on k8s fails with "Unknown method truncate" on restore

That log does not appear. It looks like we have egg and chicken issue. 2019-02-15 16:49:15,045 DEBUG org.apache.hadoop.hdfs.DFSClient - Connecting to datanode 10.246.221.10:50010 2019-02-15 16:49:15,045 DEBUG

Re: TaskManager gets confused after the JobManager restarts

Hi Till, I will have to test it with flink 1.7.1 and get back to you. Thanks! Best, Ethan > On Feb 15, 2019, at 4:01 AM, Till Rohrmann wrote: > > Hi Ethan, > > can you observe a similar behaviour with Flink 1.7.1? Flink 1.4.2 is no > longer supported by the community. > > Cheers, > Till >

Re: Flink Job sometimes does not stop as expected on cancelling.

Hi Team, Following up to check if any of you have faced this issue. If so how to resolve this. Regards, Mahantesh Patil. On Mon, Feb 11, 2019 at 3:50 PM Mahantesh Patil wrote: > Hi Team, > > We have Flink jobs running in cluster mode. When I cancel the job and > check for status it still

Re: In-Memory state serialization with kryo fails

Hi Gordon, thx for you time, will try to find other suitable serializer. > On 13 Feb 2019, at 07:25, Tzu-Li (Gordon) Tai wrote: > > Hi, > > I would suggest to avoid Kryo for state serialization, especially if this job > is meant for production usage. > It might get in the way in the future

[flink :: connected-streams :: integration-tests]

Hi mates, I got some troubles with the implementation of integration tests for the job, based on connected streams. It has the following logic: I got two streams, first one is a stream of rules, and another one is a stream of events to apply events on each rule, I’ve implemented a

Re: [flink-cep] - What is the difference between GroupPattern and Pattern?

Hi Jiayi, GroupPattern is used to support patterns like this: A (B C) {3} D. The sub-group (B C) is considered as a GroupPattern. Without it, it's impossible to support patterns like that. You can refer to the GroupITCase for more information. Regards, Dian > 在 2019年2月15日，下午4:10，bupt_ljy

Re: Window elements for certain period for delayed processing

Many Thanks Fabian I will start to investigate ProcessFunction Regards -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Synchronize reading from two Kafka Topics of different size

Hi Olle, what you are describing is indeed a problem in Flink. The solution to the problem would be to synchronize the event time across sources so that a source can throttle down when it realizes that it has advanced too far [1]. At the moment, this feature is in development, but not yet

Re: KafkaTopicPartition internal class treated as generic type serialization

Hi, I'm having the exact same issue with flink 1.4.0 using scala 2.11 . Do you have any suggestion on how to fix this ? I don't see how to register a custom serializer for a class I did not write. Thanks ! > I disabled generic type serialization via > env.getConfig.disableGenericTypes() >

Re: TaskManager gets confused after the JobManager restarts

Hi Ethan, can you observe a similar behaviour with Flink 1.7.1? Flink 1.4.2 is no longer supported by the community. Cheers, Till On Thu, Feb 14, 2019 at 5:06 PM Ethan Li wrote: > The related job manager log is > https://gist.github.com/Ethanlm/86a10e786ad9025ddaa27c113c536da8 > > On Feb 14,

Re: Flink DataStream: A few dates are getting through very slowly

Hi Marke, what is the expected behaviour in your case? Is the throughput to slow or do you expect that no elements should be output? Cheers, Till On Thu, Feb 14, 2019 at 1:28 PM Marke Builder wrote: > Hi, > > I'm using a simply streaming app with processing time and without states. > The app

Re: Production readiness

Hi Aitozi, resuming a job with a higher parallelism than the initially defined max parallelism (in your case 128) is not possible. For this one would need to rewrite the savepoint information (basically rehash the keys) as Andrey said. Cheers, Till On Thu, Feb 14, 2019 at 3:50 AM aitozi wrote:

Re: Using RemoteExecutionEnvironment from IDE

Hi Benjamin, what do you mean with "start a RemoteExecutionEnvironment from the IDE from scratch"? Are you asking whether you can restart the cluster against which the RemoteExecutionEnvironment submits the job? This is not possible. If you want to debug `RemoteInputChannels` locally, you could

Re: Using RemoteExecutionEnvironment from IDE

Hi Andrey, I made some bigger changes in the logic of the RemoteInputChannels, means that this class should be recompiled for every run. Attaching to the IDE is pretty helpful, I already tried that before. But I think that there is just no solution for this and I have rerun the environment

Re: Re: flink使用异常超时

Hi Congxian 我这里改用了G1收集器，而且我在taskmanager上看到fgc的次数也才几次，最多十几次，没有特别严重的gc。并且，也把状态后端放到了rockDB上了。 但是，跑了不到一小时，也是报异常，在taskmanager的日志上看到的日志为： org.apache.flink.util.FlinkException: The assigned slot container_e23_1545597259276_0490_01_06_0 was removed. at

Re: Impact of occasional big pauses in stream processing

Hi Ajay, Rong and Andrey are right. Backpressure can eventually be propagated up until it reaches the sources where it slows down the reading from your external system. But unless your cluster is only temporarily under-provisioned for your use case, the system should be able to catch up with the

Test FileUtilsTest.testDeleteDirectory failed when building Flink

Hi all, Recently we migrate Flink build to a new docker image, after which the build job always fails with test errors on local file system permissions. For example: FileUtilsTest.testDeleteDirectory:129 this should fail with an exception. I notice the following statements in the javadoc of

Re: How to load multiple same-format files with single batch job?

H François, The TableEnvironment.connect() method can only be used if you provide (quite a bit) more code. It requires a TableSourceFactory and handling of all the properties that are defined in the other builder methods. See [1]. I would recommend to either register the BatchTableSource

Re: 订阅Apache Flink 中文邮件列表

要訂閱請寄信到user-zh-subscr...@flink.apache.org 不需要標題內文，收到確認信按照指示回覆即可完成訂閱 fysoft2006 於 2019年2月15日 週五 下午3:22寫道： > 您好，订阅Apache Flink 中文邮件列表，谢谢！

Re: long lived standalone job session cluster in kubernetes

Alright, I'll get back to you once the PRs are open. Thanks a lot for your help :-) Cheers, Till On Thu, Feb 14, 2019 at 5:45 PM Heath Albritton wrote: > My team and I are keen to help out with testing and review as soon as > there is a pill request. > > -H > > On Feb 11, 2019, at 00:26, Till

[flink-cep] - What is the difference between GroupPattern and Pattern?

Hi, all I notice that we have GroupPattern, which is subclass of Pattern in flink-cep module. I’m not very sure about the meaning of GroupPattern and its necessity. I will appreciate if anyone can list some examples for me to help understand this? Thanks! Best Regards, Jiayi Liao