Re: Design rational behind copying via serializing in flink runner

2020-09-07 Thread Maximilian Michels
Hey Teodor, Copying is the default behavior. This is tunable via the pipeline option 'objectReuse', i.e. 'objectReuse=true'. The option is disabled by default because users may not be aware of object reuse and recycle objects in their process functions which will have unexpected side

Re: Design rational behind copying via serializing in flink runner

2020-09-06 Thread Teodor Spæren
Hey Brian! Sorry for the late reply, this one kind of got lost in my mail client. Still trying to figure this mailing list thing out, hehe. I would like to try to see if a simple return there will speed things up. I've never built BEAM by hand though, but is a full build as described in

Re: Design rational behind copying via serializing in flink runner

2020-08-31 Thread Brian Hulette
Hi Teodor, I actually forward your message to dev@ before, but I foolishly removed user@ from the distro so I think you weren't able to see it. Sorry about that! +Lukasz Cwik replied there [1]. I'll copy it here and we can keep discussing on this thread: The idea in Beam has always been to make

Design rational behind copying via serializing in flink runner

2020-08-31 Thread Teodor Spæren
Hey! First time posting to a mailing list, hope I did it correctly :) I'm writing a master thesis at the University of Oslo and right now I'm looking at the performance overhead of using Beam with the Flink runnner versus plain Flink. I've written a simple program, a custom source outputing