Re: Flink Kafka reads too many bytes .... Very rarely

2018-03-08 Thread Stephan Ewen
t; > > This is actually a big relief, I toss out the garbage data and keep > rolling. > > > > I hope this helps, thank you. > > > > > > > > *From: *Stephan Ewen > *Date: *Thursday, March 1, 2018 at 9:26 AM > *To: *"user@flink.apache.org&qu

Re: Flink Kafka reads too many bytes .... Very rarely

2018-03-07 Thread Philip Doctor
you. From: Stephan Ewen Date: Thursday, March 1, 2018 at 9:26 AM To: "user@flink.apache.org" Cc: Philip Doctor Subject: Re: Flink Kafka reads too many bytes .... Very rarely Can you specify exactly where you have that excess of data? Flink uses basically Kafka's standard consum

Re: Flink Kafka reads too many bytes .... Very rarely

2018-03-01 Thread Stephan Ewen
gt; >> >> >> >> *From: *Philip Doctor >> *Date: *Tuesday, February 27, 2018 at 10:02 PM >> *To: *"Tzu-Li (Gordon) Tai" , Fabian Hueske < >> fhue...@gmail.com> >> *Cc: *"user@flink.apache.org" >> *Subject: *Re: Flink

Re: Flink Kafka reads too many bytes .... Very rarely

2018-03-01 Thread Fabian Hueske
u’ve got an idea for a > work around, I’d be all ears too. > > > > > > *From: *Philip Doctor > *Date: *Tuesday, February 27, 2018 at 10:02 PM > *To: *"Tzu-Li (Gordon) Tai" , Fabian Hueske < > fhue...@gmail.com> > *Cc: *"user@flink.apache.org&quo

Re: Flink Kafka reads too many bytes .... Very rarely

2018-02-27 Thread Philip Doctor
. If you’ve got an idea for a work around, I’d be all ears too. From: Philip Doctor Date: Tuesday, February 27, 2018 at 10:02 PM To: "Tzu-Li (Gordon) Tai" , Fabian Hueske Cc: "user@flink.apache.org" Subject: Re: Flink Kafka reads too many bytes Very rarely Honestly

Re: Flink Kafka reads too many bytes .... Very rarely

2018-02-27 Thread Philip Doctor
. If I can help more, please let me know. Thank you for your replies. -Phil From: "Tzu-Li (Gordon) Tai" Date: Tuesday, February 27, 2018 at 3:12 AM To: Fabian Hueske , Philip Doctor Cc: "user@flink.apache.org" Subject: Re: Flink Kafka reads too many bytes Very rar

Re: Flink Kafka reads too many bytes .... Very rarely

2018-02-27 Thread Tzu-Li (Gordon) Tai
Hi Philip, Yes, I also have the question that Fabian mentioned. Did you start observing this only after upgrading to 1.4.0? Could you let me know what exactly your deserialization schema is doing? I don’t have any clues at the moment, but maybe there are hints there. Also, you mentioned that th

Re: Flink Kafka reads too many bytes .... Very rarely

2018-02-27 Thread Fabian Hueske
Hi, Thanks for reporting the error and sharing the results of your detailed analysis! I don't have enough insight into the Kafka consumer, but noticed that you said you've been running your application for over a year and only noticed the faulty behavior recently. Flink 1.4.0 was released in mid

Flink Kafka reads too many bytes .... Very rarely

2018-02-26 Thread Philip Doctor
Hello, I’m using Flink 1.4.0 with FlinkKafkaConsumer010 and have been for almost a year. Recently, I started getting messages of the wrong length in Flink causing my deserializer to fail. Let me share what I’ve learned: 1. All of my messages are 520 bytes exactly when my producer places th