Re: Data type serialization and testing

2021-04-30 Thread Dave Maughan
Hi Timo, Thanks for your suggestions. I did notice the chaining option. I'll give them a try. Is there an established pattern for running tests against a local cluster? Or any examples you could point me to? I did notice a FlinkContainer (testcontainers) but it appears to be in a module that is

Re: Data type serialization and testing

2021-04-30 Thread Timo Walther
Hi Dave, maybe it would be better to execute your tests against a local cluster instead of the mini cluster. Also object reuse should be disabled and chaining should be disabled to force serialization. Maybe others have better ideas. Regards, Timo On 30.04.21 10:25, Dave Maughan wrote:

Data type serialization and testing

2021-04-30 Thread Dave Maughan
Hi, I recently encountered a scenario where the data type being passed between operators in my streaming job was modified such that it broke serialization. This was due to a non-Avro top-level data type containing an Avro field. The existing integration test (mini cluster) continued to work and