Hi Storm devs, We've narrowed down the issue - multiple threads in our bolts are accessing output collectors at the same time. In storm1, this was fine (presumably something in clojure synchronized it, or it was thread-safe), but in storm2, this causes multiple threads to try and write to the same buffer at the same time, causing data corruption and really weird behaviour when the corrupted data was deserialized at the other end. We've fixed this by putting a mutex on all the collectors.
I know there's been some back-and-forth around this in the past, but now, output collectors are definitely not thread-safe in storm2. Simon -----Original Message----- From: Simon Cooper <[email protected]> Sent: 03 April 2020 12:16 To: [email protected] Subject: Crashes when running storm 2.1 topologies on multiple workers Hi Storm devs, We've encountered a serious problem when trying to run a storm 2.1 topology across multiple workers - it looks like the data is being corrupted somewhere between being serialized on the sending worker and being deserialized on the receiving worker. This means it's impossible for us to run storm 2.1 on topologies with more than one worker! The relevant storm bug is STORM-3582. We've seen multiple exceptions in this area, some of which point towards kryo being at fault here, maybe involving custom serializers. However, I've so far been unable to reproduce the issue in test cases. Is anyone able to give us some pointers to try and work out what may be going wrong? This does seem like a very serious issue for the latest storm release (unfortunately we're unable to try storm 2.0 to see when it was introduced due to other issues with that release). Many thanks, Simon Cooper This message, and any files/attachments transmitted together with it, is intended for the use only of the person (or persons) to whom it is addressed. It may contain information which is confidential and/or protected by legal privilege. Accordingly, any dissemination, distribution, copying or use of this message, or any part of it or anything sent together with it, other than by intended recipients, may constitute a breach of civil or criminal law and is hereby prohibited. Unless otherwise stated, any views expressed in this message are those of the person sending it and not the sender's employer. No responsibility, legal or otherwise, of whatever nature, is accepted as to the accuracy of the contents of this message or for the completeness of the message as received. Anyone who is not the intended recipient of this message is advised to make no use of it and is requested to contact Featurespace Limited as soon as possible. Any recipient of this message who has knowledge or suspects that it may have been the subject of unauthorised interception or alteration is also requested to contact Featurespace Limited.
