On Fri, Jan 18, 2013 at 08:07:14PM -0800, Sijie Guo wrote: > in most of case, there are subscriptions, since Hedwig's use case is > reliable pub/sub system. so most of applications subscribed and never > unsubscribe. so the case you described 'If there's no subscribers, we can > just discard the message' is rare in Hedwig. Yes, so my point is that it's quite straight forward to avoid publish latency in this scenario.
> It affects the incoming publish requests at the 'ledger change' time > period. Especially, for a continuous publish stream, how you decide the > point to avoid the hit? > > One possible way is Hub server could pre-create ledger before the point > changing ledger. pre-creation could run in background, not affecting > publish path. but pre-creation increase the possibility to have zombie > ledgers and exhausted ledger id space. And also it would make ledger change > logic pretty complex. so it is not good. You have to deal with zombies at some level anyhow. If you don't have zombie ledgers, you can still have zombie topics. If you fix the zombie topics, you can still have zombie users. In any case, I don't think the zombie ledgers are a huge issue. Lets say, that for each crash, you zombify 100 ledgers, over a year of running a 1000 machine cluster, you may have 100 crashs, thats only 10,000 zombies which really isn't very much in that scale. I don't think it will make code complex either. We just need a preallocator thread that feeds into a blocking queue. The code that wants the ledger can then just take from the queue. -Ivan
