On Oct 10, 2014, at 4:11 PM, Harsha <[email protected]> wrote: > it could be couple of issues more of speculation at this point > nimbus writes the topology assignments to zookeeper and a watcher > triggers on supervisors to read the assignments > and subsequently download the topology jar from nimbus. so this watcher > trigger could be faster for the same dc and I am not quite sure how > reliable is this across dc. how is you zookeeper quorum setup is it > across dc or just lives in single dc?.
It is a single zookeeper instance spread across multiple dc’s. The thing is, it’s not that the trigger is slow. All supervisors start downloading at exactly the same time, it’s just that the remote supervisors do the pull at a much slower rate. We’re working with networking at the moment to try to ensure that there isn’t something strange with the firewall having an impact on the transfer, but like I said, other things like scp work just fine. > Having a single zookeeper ensemble across dcs probably not advisable > one pattern I've seen is to have one zookeeper ensemble per dc and > replicate changes happening in one dc in another zookeeper ensemble. I didn’t know it was possible to have two separate instances and replicate. Any pointers on docs to make this work? I’d be happy to give this a try. > And another issue could be if all of the supervisors trying to download > this jar from nimbus might cause contention there . This would make sense but the local dc get the topology very quickly and finishes. When these finish, the speed of the remote dc’s doesn’t change. > -Harsha Thank you! -jeremy > > > On Fri, Oct 10, 2014, at 12:44 PM, Jeremy Hansen wrote: >> Thanks for the response. >> >> On Oct 10, 2014, at 9:10 AM, Harsha <[email protected]> wrote: >> >>> Jeremy, >>> I haven't run a cluster across data centers but I feel like >>> zookeeper might be an issue here since nimbus receives >>> heartbeats from supervisor and workers via zookeeper. Did you >>> looked in zookeeper logs if there is any misbehavior. >>> >>> From your description it seems to me that your nimbus is in one dc and >>> all your supervisors are in other dc is that right? >> >> Correct. We have supervisors in the same dc as the nimbus server and >> some in another data center. The supervisors local to the nimbus server >> get the topology immediately. The remote supervisors take about 15 >> minutes to load. >> >> I will take a look at zookeeper, but it just seems like the transfer >> itself is the problem. Does zookeeper play a role once the topology >> transfer has been initiated? >> >> Thanks >> -jeremy >> >>> >>> "Doing transfers of the same amount of data, such as just scp’ing the >>> topology across the network is just fine". >>> What kind of spout are you using and is it trying to read this data from >>> remote dc?. >>> >>> -Harsha >>> >>> >>> On Fri, Oct 10, 2014, at 01:19 AM, Jeremy Hansen wrote: >>>> >>>> We have a cluster where workers are distributed across multiple data >>>> centers. Supervisor nodes in the remote dc (remote meaning the dc where >>>> nimbus is not running) pull the topology at a very slow rate. It’s not >>>> failing, it’s not retrying, it’s not timing out, it’s just very slow. >>>> This seems to be specific to netty. Doing transfers of the same amount >>>> of data, such as just scp’ing the topology across the network is just >>>> fine. Copies in seconds. Has anyone seen a situation where netty was >>>> limiting throughput from worker nodes with higher latency? We’re seeing >>>> 50 - 60 ms in latency from the remote data center. Local workers load a >>>> 350M topology in about 15 - 20 seconds. Remote workers takes about 10 - >>>> 15 minutes. >>>> >>>> This is apache storm 0.9.2 incubating. >>>> >>>> I can actually just sit there and ‘du’ the topology path to see that it’s >>>> pulling, just very very slowly. >>>> >>>> There’s many options to adjust buffers, but I’ve tried turing them all >>>> with no change in the behavior. It almost feels as if netty is setting >>>> some kind of tcp window size to throttle the throughput. >>>> >>>> Thank You >>>> -jeremy >>
