Re: Running Nexmark with PubSub issue

Hi Raghu, Thank you, I’ll take a look on your changes. WBR, Alexey > On 13 Apr 2018, at 20:14, Raghu Angadi wrote: > > Hi Alexey, > > I am not sure if I can get to it soon. If you are planning to run nexmark > with Kafka, would you mind porting changes from my branch to a new PR? I will >

Re: Running Nexmark with PubSub issue

Hi Alexey, I am not sure if I can get to it soon. If you are planning to run nexmark with Kafka, would you mind porting changes from my branch to a new PR? I will be happy to review and get it merged. Raghu. On Thu, Apr 12, 2018 at 6:14 AM Alexey Romanenko wrote: > Raghu, Kenneth, > Yes, creat

Re: Running Nexmark with PubSub issue

Raghu, Kenneth, Yes, creating a separate class instead of inner one helped to overcome this issue with serialisation. Seems like this a bug in NexmarkLauncher, so I’ll create a jira for this. Thank you for help with this. Btw, Raghu, are you going to submit a PR from your branch? I think this i

Re: Running Nexmark with PubSub issue

Yea, this is a common issue with serializable anonymous inner classes in general. It would be nice if Beam Java could have an overarching solution to limiting the closure to things actually touched. Kenn On Wed, Apr 11, 2018 at 10:30 AM Raghu Angadi wrote: > I noticed it too while adding KafkaI

Re: Running Nexmark with PubSub issue

I noticed it too while adding KafkaIO support for Nexmark (this was in parallel to another PR for KafkaIO that got merged recently). The anonymous inner class for DoFn is not serializable. I moved it to a static class in my branch, but didn't test it yet : https://github.com/rangadi/beam/commit/b4

Running Nexmark with PubSub issue

Hi all, For the moment, I'm working on BEAM-4048 to add Kafka source/sink support with different modes to Nexmark, like it has for PubSub (SUBSCRIBE_ONLY, PUBLISH_ONLY and COMBINED). It seems that the code will be similar to what we have for Pu