I'm not much of an expert in the behaviour of DNS servers :) so I thought I'd think aloud a little...
I guess we don't really care too much about old broker entries in DNS; if a client can't connect it just tries the next one it finds - so its really just an optimisation; a good DNS server would zap them quickly but even if they stick around for 24 hours its not that big a deal as there's hopefully not gonna be loads of brokers being started/stopped on random IP addresses or anything :) The thing I'm pondering is if folks boot up a new broker on a new box, we want clients to be able to find it pretty soon. In most DNS servers you've worked with before - if you add a new entry (rather than entries expiring) do servers tend to reflect that quickly to clients? I can imagine in a highly federated DNS server network it might take a while to propogate. Just wondered :) 2008/5/19 Sridhar2008 <[EMAIL PROTECTED]>: > >>Agreed. >> >>We've got a pluggable discovery mechanism; both for the broker >to >>advertise itself or for brokers/clients to discover things, so I >guess >>DNS just becomes another provider. >>http://activemq.apache.org/discovery.html > > Exactly ! The pluggable nature of ActiveMQ is great for enhancements like > these. > >>The difference with DNS is you may use your existing DNS >provider or >>might wanna run a custom implementation to get the timeout >stuff etc? > > Again, right on ! The timeout may not be that big a deal depending on the > implementation ... for example, it might be a lazy cleanup process which may > or may not be part of the DNS master server. For the clients (ie producers > or consumers), if a broker in the list does not respond, it will just > timeout to the next (note: 'next' could be as simple as next in the list or > based on some policy). > > > James.Strachan wrote: >> >> Agreed. >> >> We've got a pluggable discovery mechanism; both for the broker to >> advertise itself or for brokers/clients to discover things, so I guess >> DNS just becomes another provider. >> http://activemq.apache.org/discovery.html >> >> The difference with DNS is you may use your existing DNS provider or >> might wanna run a custom implementation to get the timeout stuff etc? >> >> 2008/5/19 Sridhar2008 <[EMAIL PROTECTED]>: >>> >>> >>> Thanks for the link. I quickly glanced through the dnsjava website, and >>> not >>> clear if it requires a specific server for updates to work. >>> >>> Also, most large organizations have their own legacy DNS in place, with >>> some >>> home grown way of updating the DNS records. So we need to accommodate >>> this >>> use case (for example, making 'auto-registration' optional). >>> >>> >>> James.Strachan wrote: >>>> >>>> Ah cool thanks for that! I was wondering if an existing DNS library >>>> might be useful. e.g. >>>> >>>> http://www.dnsjava.org/ >>>> >>>> which has a DNS client and server. >>>> >>>> I guess ZeroConf is kinda similar too; just not requiring a new or >>>> custom DNS server (though at the expense of requiring multicast >>>> support) >>>> >>>> 2008/5/19 Sridhar2008 <[EMAIL PROTECTED]>: >>>>>> Incidentally I'm intrigued by the DNS discovery stuff; it sounds >>>>>> >great >>>>>>:). What did you have in mind? That each broker on startup >would >>>>>>register a DNS entry (with a timeout maybe so they get >removed) and >>>>> >>>>> Yes, auto-registration. Couple of things to note: >>>>> >>>>> - The 'timeout feature' again is DNS master specific, the existing ones >>>>> need >>>>> to be enhanced to support this aging out. The standard timeout you see >>>>> in >>>>> the DNS entry is for a different purpose: refresh between master and >>>>> slave >>>>> DNS servers. >>>>> >>>>> - There is no standard APY for web dns registration, so we will >>>>> define >>>>> an >>>>> APY, with specific implementatioin to interact with the DNS server >>>>> (folks >>>>> can chose to change the implementation for their env). >>>>> >>>>> - Most messaging systems run in trusted env, but some use cases may >>>>> need >>>>> authentication before a broker is allowed to do 'auto-registration'. >>>>> This >>>>> can be addressed later. >>>>> >>>>>>clients would ping DNS when attempting to connect in the >>>>>>FailoverTransport? >>>>> >>>>> Yes, we would specify the cluster name in client config and they would >>>>> connect using 'ClusterTransport' :-) >>>>> >>>>> >>>>> James.Strachan wrote: >>>>>> >>>>>> 2008/5/15 Sridhar2008 <[EMAIL PROTECTED]>: >>>>>>> >>>>>>> >>>>>>> Rob/James, >>>>>>> >>>>>>> Thanks for the feedback. I will address both of your questions in a >>>>>>> combined >>>>>>> way :-) >>>>>>> >>>>>>> Rob> Do you need persistent messaging - or non-persistent only? >>>>>>> James> or just pick one broker per operation/transaction? >>>>>>> >>>>>>> Initially, it is going to be the above. Since the messaging usage >>>>>>> scenarios >>>>>>> are many, I will just assume the use case of interest for now is >>>>>>> durable >>>>>>> transfer using cluster for high availability. In this case I am not >>>>>>> sure >>>>>>> that it buys much to send to multiple brokers simultaneously (see my >>>>>>> next >>>>>>> paragraph below) - adds additional headaches as James sites below >>>>>>> (however, >>>>>>> there may be other use cases where this might be useful), in addition >>>>>>> to >>>>>>> making client-side logic complex. >>>>>> >>>>>> Agreed. >>>>>> >>>>>> For sending to a topic (outside of a transaction which may include >>>>>> other operations) then sending the message to all brokers is a no >>>>>> brainer. I guess so long as acknowledgements only get sent to the >>>>>> broker they came from & for queue sending we only send to one of the >>>>>> brokers it should be fairly straight forward. >>>>>> >>>>>> (So a little bit of hacking to the FanOutTransport should do the trick >>>>>> I >>>>>> think). >>>>>> >>>>>> We could get clever going forward; where rather than randomly picking >>>>>> one of the brokers (or round robbin) we kinda partition destinations >>>>>> across the available brokers? >>>>>> >>>>>> >>>>>> >>>>>>> Some thoughts: Eventual goal is a broker down in the cluster implies >>>>>>> capacity hit and not a service hit. So, I will rather solve this use >>>>>>> case >>>>>>> as >>>>>>> a distributed storage issue (after 'cluster transport' is added) - >>>>>>> investigate either a DHT based solution or perhaps something like >>>>>>> Hadoop >>>>>>> with a JDBC interface. The state is replicated in multiple nodes of >>>>>>> the >>>>>>> cluster so the broker that is down can be ignored (ie no need to >>>>>>> sweat >>>>>>> on >>>>>>> recovering its state). >>>>>> >>>>>> Agreed. >>>>>> >>>>>> FWIW distributing the state is trivial - the FanOutTransport can do >>>>>> this today really. The issue is ensuring consistency (so more to do >>>>>> with locking & consistency than moving the messages around). >>>>>> >>>>>> Incidentally I'm intrigued by the DNS discovery stuff; it sounds great >>>>>> :). What did you have in mind? That each broker on startup would >>>>>> register a DNS entry (with a timeout maybe so they get removed) and >>>>>> clients would ping DNS when attempting to connect in the >>>>>> FailoverTransport? >>>>>> >>>>>> -- >>>>>> James >>>>>> ------- >>>>>> http://macstrac.blogspot.com/ >>>>>> >>>>>> Open Source Integration >>>>>> http://open.iona.com >>>>>> >>>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17323314.html >>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> James >>>> ------- >>>> http://macstrac.blogspot.com/ >>>> >>>> Open Source Integration >>>> http://open.iona.com >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17323601.html >>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> James >> ------- >> http://macstrac.blogspot.com/ >> >> Open Source Integration >> http://open.iona.com >> >> > > -- > View this message in context: > http://www.nabble.com/Cluster-transport-...-tp17221068s2354p17323803.html > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. > > -- James ------- http://macstrac.blogspot.com/ Open Source Integration http://open.iona.com
