Re: CaseClassSerializer and/or TraversableSerializer may still not be threadsafe?

2018-04-18 Thread Stefan Richter
Hi, I agree that this looks like a serializer is shared between two threads, one of them being the event processing loop. I am doubting that the problem is with the async fs backend, because there is code in place that will duplicate all serializers for the async snapshot thread and this is

Re: CaseClassSerializer and/or TraversableSerializer may still not be threadsafe?

2018-04-17 Thread Stephan Ewen
Thanks for reporting this, also thanks for checking out that this works with RocksDB and also with synchronous checkpoints. I would assume that this issue lies not in the serializer itself, but in accidental sharing in the FsStateBackend async snapshots. Do you know if the issue still exists in

CaseClassSerializer and/or TraversableSerializer may still not be threadsafe?

2018-04-17 Thread joshlemer
Hello all, I am running Flink 1.4.0 on Amazon EMR, and find that asynchronous snapshots fail when using the Filesystem back-end. Synchronous snapshots succeed, and RocksDB snapshots succeed (both async and sync), but async Filesystem snapshots fail with this error: