Yes, in the reference architectures for AWS you split your hosts across three zones (not two). ML7 even understands "zones" and zone assignments shows up in the admin interface. The latency between zones is fast enough that it still works as a single cluster.
For regular data center deployments, zones can represent racks. -jh- On Oct 29, 2013, at 1:49 PM, Ron Hitchens <[email protected]> wrote: > > I bow down before David's far superior Amazon mojo. > > Yes, AWS topology changes the values of many of the > variables in this formula. You really need expert advice > when building out a system like this on a service like that. > > The one caveat I'd throw in, which I've heard from people > who are AWS experts, is that when configuring a cluster that > crosses zones is to be wary of "split brain syndrome". This > can happen if you configure a cluster with half the nodes in > one zone and half in another, with local disk replication across. > > If the comm link between zones goes down for any reason, > then the cluster would be split in half. The nodes in each > half would think they're the survivors, form a quorum and do > a fail over. You'd then have two independent clusters that > are doing updates independently of each other. > > An unlikely event, but it could be a nightmare to untangle > if it happens, so it's worth planning for. > > On Oct 29, 2013, at 10:16 PM, David Lee <[email protected]> wrote: > >> I would like to make a slight adjustment to this generality. >> That is in Cloud Computing, Amazon AWS/EC2 in particular. >> >> Amazon has "regions" which are composed of 3 or more "zones" ... Each "zone" >> is as nearly a seperate datacenter as physically possible except for >> connectivity. >> That is they are generally built on seperate geographic topology (dont share >> flood plains, earthquake fault lines etc) and are generally fed by issolated >> infrastructure ( power, internet etc). >> Yet ... they are close enough together that the latency between zones is >> faster than the latency to an SSD drive write cycle. Most zones are within >> 2ms of each other. Thats fast. >> While this is not quite as fast as talking to nodes sharing the same rack >> and bridge on a FIOS channel ... its still *damn fast*. >> Creating a cluster across amazon zones within the same region can achieve >> both HA and DR to a large degree. It's not perfect as its *possible* (I >> belive its happened once in the last 10 years) for >> multiple zones in a region to fail simultaneously ... but its vastly more >> resilient than putting all your eggs in the same datacenter yet the latancy >> between zones in the same region is so small that >> creating clusters within a zone are extremely fast. Maybe not *quite* as >> fast as on the same rack but it's a good compromise. >> Especially if you make a hybrid. That is several nodes in each zone >> co-located but cross-replicating ("local disk failover") across zones. >> If you want real DR you should create a replica cluster in a different >> region and do foreign replication to it. >> But even without that ... its pretty amazing. >> >> The result is a very fault resistant HA system that performs very well. A >> good compromise. >> >> Look forward to a coming announcement with will make such an architecture >> easier to build. >> But you dont *have* to wait. MarkLogic 6.0 runs great on Amazon EC2 out of >> the box. >> But we have made things simpler still to create a fault tolerant system that >> is also HA and performing. >> Nothing is perfect, everything is a compromise, but this combination of >> multi zone single region clusters is as near as perfect as I can imagine, or >> anyone else could build on their own. >> >> -David >> >> >> ----------------------------------------------------------------------------- >> David Lee >> Lead Engineer >> MarkLogic Corporation >> [email protected] >> Phone: +1 812-482-5224 >> Cell: +1 812-630-7622 >> www.marklogic.com >> >> >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Ron Hitchens >> Sent: Tuesday, October 29, 2013 3:40 PM >> To: MarkLogic Developer Discussion >> Subject: Re: [MarkLogic Dev General] Reg: E-Node and D-Node configuration >> >> >> You can think of a MarkLogic cluster as a single virtual >> server. A cluster is made up of nodes (E, D or E/D) but the >> cluster should be thought of as an indivisible unit. >> >> D (data) nodes are MarkLogic processes that have forests >> attached. E (evaluator) nodes are those nodes which run >> XQuery/XSLT requests on an appserver. In a cluster, all nodes >> share the same appserver configuration, so any node can be an >> E node. Typically, when configuring dedicated E and D nodes, >> you configure things to send requests to only those nodes that >> you want to act as E's, allowing the others to act only as D's. >> >> Communication between nodes in a cluster is basically this: >> >> For queries (read-only) no locks are needed (read up on MVCC). >> Each search operation is fired in parallel to every D node >> in the cluster (this is the "map" phase). When the last D node >> has responded, the E node can then merge the results (the "reduce"). >> >> So, the lower the latency in communication between nodes, the >> better the overall throughput. You really don't want any slow >> links between nodes in the cluster because it can slow down all >> the E nodes. >> >> For update (write), cluster-wide locks must be obtained for >> documents that are, or might be, updated. All nodes in the cluster >> must acknowledge the lock(s) before the update(s) can proceed. This >> basically means that updates can't happen faster than the slowest >> responding node in the cluster. Oh, and the locks need to be >> released as well, via inter-node communication. >> >> Again, bad for overall performance when communication links >> between nodes slow down, even with super-fast, beefy hardware. >> >> As Mike pointed out, clusters are not database replication. >> You cluster to improve performance by spreading the immediate >> work across multiple CPU and disks co-located together. You >> can add synchronous replication between nodes in a cluster to >> provide for HA failover in the event a node fails. This has a >> latency cost, but makes the cluster more robust. You replicate >> databases asynchronously between clusters to provide for disaster >> recovery if an entire cluster is lost or becomes unreachable. >> >> Hope that helps. >> >> --- >> Ron Hitchens {[email protected]} +44 7879 358212 >> >> On Oct 28, 2013, at 10:03 PM, Arindam3 B <[email protected]> wrote: >> >>> >>> Thanks Mike for the great walkthrough. Just trying to understand more on >>> the xqdp protocol. Can you throw some light on how it operates between >>> enodes n dnodes? >>> >>> Thanks & Regards >>> Arindam >>> >>> -----Michael Blakeley <[email protected]> wrote: ----- >>> >>> ======================= >>> To: MarkLogic Developer Discussion <[email protected]> >>> From: Michael Blakeley <[email protected]> >>> Date: 10/28/2013 10:31PM >>> Subject: Re: [MarkLogic Dev General] Reg: E-Node and D-Node configuration >>> ======================= >>> Hosts within a cluster should have low-latency communications: gigabit >>> ethernet or better. Ideally they should all be on the same switch and/or >>> VLAN, with no router hops between hosts. If you try to set up a cluster >>> across a WAN link you are likely to see poor performance and poor >>> reliability. You might be trying to handle high availability (HA) and >>> disaster recovery (DR) with a single cluster: that would be a mistake. >>> >>> For high availability, use a single cluster with low-latency >>> communications. Configure forest replication and host failover to provide >>> the desired degree of protection against host failures. The docs at >>> http://docs.marklogic.com/guide/cluster/failover talk about this as >>> "local-disk failover". >>> >>> For disaster recovery - scenarios where an entire data center goes offline >>> - use database replication to a different cluster. This can use >>> higher-latency communications, such as a WAN link. The docs at >>> http://docs.marklogic.com/guide/database-replication describe this. The DR >>> replica cluster can also implement local-disk failover to provide its own >>> HA. >>> >>> -- Mike >>> >>> On 28 Oct 2013, at 06:41 , Arindam3 B <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> I had a query regarding the E-Node and D-Node setup in Marklogic. >>>> >>>> In a distributed environment, if I plan to keep the Enodes and DNodes >>>> separately in different physical locations over the LAN or WAN (across >>>> geographies), what is the potential risk? >>>> How does failover work in that scenario? >>>> I have read that ENodes and DNodes communicate through XQDP protocol, so >>>> in this case will there be performance issues? >>>> >>>> Does Marklogic recommend having ENode and DNode cluster in the same >>>> physical box? >>>> If so, then across the network if we have a set of E-D-Nodes, how is the >>>> network latency reduced while synching the data during replication? >>>> >>>> If you can provide me with some information about XQDP protocol it would >>>> be great!! >>>> >>>> Thanks & Regards >>>> Arindam Bose >>>> =====-----=====-----===== >>>> Notice: The information contained in this e-mail >>>> message and/or attachments to it may contain >>>> confidential or privileged information. If you are >>>> not the intended recipient, any dissemination, use, >>>> review, distribution, printing or copying of the >>>> information contained in this e-mail message >>>> and/or attachments to it are strictly prohibited. If >>>> you have received this communication in error, >>>> please notify us by reply e-mail or telephone and >>>> immediately and permanently delete the message >>>> and any attachments. Thank you >>>> >>>> >>>> _______________________________________________ >>>> General mailing list >>>> [email protected] >>>> http://developer.marklogic.com/mailman/listinfo/general >>> >>> _______________________________________________ >>> General mailing list >>> [email protected] >>> http://developer.marklogic.com/mailman/listinfo/general >>> >>> _______________________________________________ >>> General mailing list >>> [email protected] >>> http://developer.marklogic.com/mailman/listinfo/general >> >> _______________________________________________ >> General mailing list >> [email protected] >> http://developer.marklogic.com/mailman/listinfo/general >> _______________________________________________ >> General mailing list >> [email protected] >> http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
