Re: London Apache Beam meetup 2: call for speakers

2017-11-06 Thread Jean-Baptiste Onofré
Hi, unfortunately, I have to decline the invite as I will be at Strata Singapore in the same time :( I'm very sorry about that. You can count on me for the 3rd edition ! Regards JB On 11/07/2017 01:41 AM, Matthias Baetens wrote: Hi all! Hope you are well. We are back for a second edition

Re: KafkaIO Autocommit

2017-11-06 Thread Raghu Angadi
Can you post full log of one the workers/executors that reads at least one record? You can remove application logs. On Mon, Nov 6, 2017 at 12:45 PM, NerdyNick wrote: > It's all partitions and it's random on how far behind they show. The topic > in question has a low

Re: KafkaIO Autocommit

2017-11-06 Thread Raghu Angadi
On Mon, Nov 6, 2017 at 10:39 AM, NerdyNick wrote: > Hey Raghu, > > Runner is Spark on Yarn cluster. Executor/Thread would be the Thread(Core) > within a Spark Executor. I have nothing I can find in the logs saying that > they aren't being closed cleanly outside of forced

Re: KafkaIO Autocommit

2017-11-06 Thread Raghu Angadi
What is the runner? Can you elaborate a bit more what you mean by 'executor/thread shutting down'? If the KafkaIO reader shutdown cleanly, it would call close the consumer cleanly, triggering auto commit. But if you shutdown a pipeline, it might not cleanly close the consumers. What is the

KafkaIO Autocommit

2017-11-06 Thread NerdyNick
There seems to be a lot of oddities with the auto offset committer and the watermark management as well as kafka offsets in general. One issue I keep having is the auto committer will just not commit any offsets. So the topic will look like it's backing up. From what I've been able to trace on it

Re: Beam Slack Channel Invitation Request

2017-11-06 Thread Jean-Baptiste Onofré
Done, you should have received an invite. Welcome ! Regards JB On 11/06/2017 12:57 PM, Damien Hawes wrote: I too would like an invite to the slack channel. On 5 Nov 2017 05:11, "Mingmin Xu" > wrote: sent, welcome! On Sat, Nov 4, 2017

Re: Beam Slack Channel Invitation Request

2017-11-06 Thread Damien Hawes
I too would like an invite to the slack channel. On 5 Nov 2017 05:11, "Mingmin Xu" wrote: > sent, welcome! > > On Sat, Nov 4, 2017 at 4:42 PM, Tristan Shephard < > tristanasheph...@gmail.com> wrote: > >> Hello, >> >> Can someone please add me to the Beam slack channel? >> >>