You are right. I think output collectors are not thread-safe in storm 2.x.
On Tue, Apr 7, 2020 at 7:35 AM Simon Cooper <[email protected]> wrote: > Hi Storm devs, > > We've narrowed down the issue - multiple threads in our bolts are > accessing output collectors at the same time. In storm1, this was fine > (presumably something in clojure synchronized it, or it was thread-safe), > but in storm2, this causes multiple threads to try and write to the same > buffer at the same time, causing data corruption and really weird behaviour > when the corrupted data was deserialized at the other end. We've fixed this > by putting a mutex on all the collectors. > > I know there's been some back-and-forth around this in the past, but now, > output collectors are definitely not thread-safe in storm2. > > Simon > > -----Original Message----- > From: Simon Cooper <[email protected]> > Sent: 03 April 2020 12:16 > To: [email protected] > Subject: Crashes when running storm 2.1 topologies on multiple workers > > Hi Storm devs, > > We've encountered a serious problem when trying to run a storm 2.1 > topology across multiple workers - it looks like the data is being > corrupted somewhere between being serialized on the sending worker and > being deserialized on the receiving worker. This means it's impossible for > us to run storm 2.1 on topologies with more than one worker! > > The relevant storm bug is STORM-3582. We've seen multiple exceptions in > this area, some of which point towards kryo being at fault here, maybe > involving custom serializers. However, I've so far been unable to reproduce > the issue in test cases. > > Is anyone able to give us some pointers to try and work out what may be > going wrong? This does seem like a very serious issue for the latest storm > release (unfortunately we're unable to try storm 2.0 to see when it was > introduced due to other issues with that release). > > Many thanks, > Simon Cooper > > This message, and any files/attachments transmitted together with it, is > intended for the use only of the person (or persons) to whom it is > addressed. It may contain information which is confidential and/or > protected by legal privilege. Accordingly, any dissemination, distribution, > copying or use of this message, or any part of it or anything sent together > with it, other than by intended recipients, may constitute a breach of > civil or criminal law and is hereby prohibited. Unless otherwise stated, > any views expressed in this message are those of the person sending it and > not the sender's employer. No responsibility, legal or otherwise, of > whatever nature, is accepted as to the accuracy of the contents of this > message or for the completeness of the message as received. Anyone who is > not the intended recipient of this message is advised to make no use of it > and is requested to contact Featurespace Limited as soon as possible. Any > recipient of this message who has knowledge or suspects that it may have > been the subject of unauthorised interception or alteration is also > requested to contact Featurespace Limited. >
