Re: [Pacemaker] Question: How many nodes can join a cluster?
- Original Message - From: Andreas Vogelsang a.vogels...@uni-muenster.de To: pacemaker@oss.clusterlabs.org Sent: Monday, October 18, 2010 9:46:12 AM Subject: [Pacemaker] Question: How many nodes can join a cluster? Hello, I’m creating a presentation about a virtual Linux-HA Cluster. I just asked me how many nodes pacemaker can handle. Mr. Schwartzkopff wrote in his Book that Linux-HA version 2 can handle up to 16 Nodes. Is this also true for pacemaker? AFAIR Pacemaker can theoretically handle up to 255, but the practical limits are determined by the messaging limits of the cluster communications layer and the size of the CIB. Heartbeat should mostly be fine up to 16 nodes, and I believe Corosync and Pacemaker are currently being regularly tested for up to 32 nodes. Maybe Steve and Andrew can chime in more details. Cheers, Florian ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Question: How many nodes can join a cluster?
On 18 October 2010 10:52, Florian Haas florian.h...@linbit.com wrote: - Original Message - From: Andreas Vogelsang a.vogels...@uni-muenster.de To: pacemaker@oss.clusterlabs.org Sent: Monday, October 18, 2010 9:46:12 AM Subject: [Pacemaker] Question: How many nodes can join a cluster? Hello, I’m creating a presentation about a virtual Linux-HA Cluster. I just asked me how many nodes pacemaker can handle. Mr. Schwartzkopff wrote in his Book that Linux-HA version 2 can handle up to 16 Nodes. Is this also true for pacemaker? I have been asked the same question and I said to them, let's say it is 126, what is the use of having 126 nodes in the cluster? Can someone imagine himself going through the logs to find why the resource-XXX failed while there are 200 resources?!! The only use of having 126 nodes is if you want to have HPC, but HPC is total different story than high available clusters. Even in N+N setup I would go with more than 4 or 6 nodes. My 2 cents, Pavlos ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Question: How many nodes can join a cluster?
On 18 October 2010 11:13, Dan Frincu dfri...@streamwide.ro wrote: Pavlos Parissis wrote: On 18 October 2010 10:52, Florian Haas florian.h...@linbit.com wrote: - Original Message - From: Andreas Vogelsang a.vogels...@uni-muenster.de To: pacemaker@oss.clusterlabs.org Sent: Monday, October 18, 2010 9:46:12 AM Subject: [Pacemaker] Question: How many nodes can join a cluster? Hello, I’m creating a presentation about a virtual Linux-HA Cluster. I just asked me how many nodes pacemaker can handle. Mr. Schwartzkopff wrote in his Book that Linux-HA version 2 can handle up to 16 Nodes. Is this also true for pacemaker? I have been asked the same question and I said to them, let's say it is 126, what is the use of having 126 nodes in the cluster? Can someone imagine himself going through the logs to find why the resource-XXX failed while there are 200 resources?!! The only use of having 126 nodes is if you want to have HPC, but HPC is total different story than high available clusters. Even in N+N setup I would go with more than 4 or 6 nodes. My 2 cents, Pavlos Actually, the syslog_facility in corosync.conf allows you to specify either a log file for each node in the cluster (locally), or setting up a remote syslog server. Either way, identifying the node by hostname or some other identifier should point out what is going on where. Granted, it's a large amount of data to process, therefore (such is the case with any large deployment) SNMP is a much better alternative for tracking issues, or (if you have _126_ times the same resource) adding some notification options to the RA might be a choice, such as SNMP trap, or even email. BTW, I'm also interested in this, I remember reading something about 64 nodes, but I'd appreciate an official response. Have you ever done troubleshooting on a 4 node cluster at 01:00 night? believe me it is not fun. I don't say there are no use cases which require a lot of nodes, but I have my doubts if there are a lot of use cases for High Available Clusters. Adding without a second thought nodes and services increase complexity, which is one of the main root cause of major problems. Cheers, Pavlos ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Question: How many nodes can join a cluster?
On Monday, October 18, 2010, Pavlos Parissis wrote: On 18 October 2010 10:52, Florian Haas florian.h...@linbit.com wrote: - Original Message - From: Andreas Vogelsang a.vogels...@uni-muenster.de To: pacemaker@oss.clusterlabs.org Sent: Monday, October 18, 2010 9:46:12 AM Subject: [Pacemaker] Question: How many nodes can join a cluster? Hello, I’m creating a presentation about a virtual Linux-HA Cluster. I just asked me how many nodes pacemaker can handle. Mr. Schwartzkopff wrote in his Book that Linux-HA version 2 can handle up to 16 Nodes. Is this also true for pacemaker? I have been asked the same question and I said to them, let's say it is 126, what is the use of having 126 nodes in the cluster? Can someone imagine himself going through the logs to find why the resource-XXX failed while there are 200 resources?!! The only use of having 126 nodes is if you want to have HPC, but HPC is total different story than high available clusters. No, not entirely. Pacemaker managed Lustre systems are quite common. And although 126 nodes is a rather high number, it is still possible for large sites. It also makes sense to manage Lustre in a global configuration, although usually for Lustre a subset of two pairs forms an OSS or MDS Lustre fail-over system. The reason is that Lustre requires an ordered shutdown sequence (MDT first). While I already wrote scripts to that with the traditional heartbeat pair setup, it is really far more complex than to do it with pacemaker. So our scripts generate a set of constraints that only pairs can run MDS/OSS resources, but still everything is in global pacemaker setup. We also have syslog-ng rules and a patched logd (patches sent to this list, need to update them again) to filter out all pacemaker debug logs, so that we can easily see messages from the lustre RA in syslogs. Cheers, Bernd -- Bernd Schubert DataDirect Networks ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Question: How many nodes can join a cluster?
I want to point out why a virtual cluster is better than a physical one. In my eyes it’s because of the power usage, cooling and you need less space. I think it’s easier to cool down one server rack than a room full of tower. In one rack you put, let’s say ten ESX server. On every ESX server you let run three nodes. So you’ve got 30 nodes. Is this a realistic idea? Unfortunately I am not an expert in creating resources. So I don’t know if you need 30 nodes for any resource. Von: Pavlos Parissis [mailto:pavlos.paris...@gmail.com]mailto:[mailto:pavlos.paris...@gmail.com] Gesendet: Montag, 18. Oktober 2010 10:58 An: The Pacemaker cluster resource manager Betreff: Re: [Pacemaker] Question: How many nodes can join a cluster? On 18 October 2010 10:52, Florian Haas florian.h...@linbit.commailto:florian.h...@linbit.com wrote: - Original Message - From: Andreas Vogelsang a.vogels...@uni-muenster.demailto:a.vogels...@uni-muenster.de To: pacemaker@oss.clusterlabs.orgmailto:pacemaker@oss.clusterlabs.org Sent: Monday, October 18, 2010 9:46:12 AM Subject: [Pacemaker] Question: How many nodes can join a cluster? Hello, I’m creating a presentation about a virtual Linux-HA Cluster. I just asked me how many nodes pacemaker can handle. Mr. Schwartzkopff wrote in his Book that Linux-HA version 2 can handle up to 16 Nodes. Is this also true for pacemaker? I have been asked the same question and I said to them, let's say it is 126, what is the use of having 126 nodes in the cluster? Can someone imagine himself going through the logs to find why the resource-XXX failed while there are 200 resources?!! The only use of having 126 nodes is if you want to have HPC, but HPC is total different story than high available clusters. Even in N+N setup I would go with more than 4 or 6 nodes. My 2 cents, Pavlos ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Question: How many nodes can join a cluster?
El día Mon, 18 Oct 2010 11:23:41 +0200, Pavlos Parissis pavlos.paris...@gmail.com escribía: Have you ever done troubleshooting on a 4 node cluster at 01:00 night? believe me it is not fun. Troubleshooting *anything* at 1am is no fun at all :-) I don't say there are no use cases which require a lot of nodes, but I have my doubts if there are a lot of use cases for High Available Clusters. Adding without a second thought nodes and services increase complexity, which is one of the main root cause of major problems. We have a cluster with 5 nodes that handles ~50 virtual machines with Xen. If we were to support more VMs (for example, because we were selling VPS services), we would need more nodes. I'm not sure if it's typical, but it's a good case scenario for potentially many nodes in a cluster. -- Roberto Suarez Soto Allenta Consulting r...@allenta.com www.allenta.com +34 881 922 600 Este correo electrónico contiene información estrictamente confidencial y es de uso exclusivo del destinatario, quedando prohibida a cualquier otra persona su revelación, copia, distribución, o el ejercicio de cualquier acción relativa a su contenido. Si ha recibido este mensaje por error, por favor conteste a su remitente mediante correo electrónico y proceda a borrarlo de su sistema. Rogamos nos comunique inmediatamente sobre cualquier inconveniente que pueda tener usted en relación al envío de este tipo de correo electrónico. Sus datos personales serán tratados de forma confidencial y no serán cedidos a terceros ajenos a ALLENTA CONSULTING, S.L. En cualquier caso, podrá ejercer los derecho de oposición, acceso, rectificación y cancelación de acuerdo con lo establecido en la Ley Orgánica 15/99, de 13 de diciembre, de Protección de Datos de Carácter Personal dirigiéndose a ALLENTA CONSULTING, S.L. en C/Enrique Mariñas 36, 2º piso, oficina 8, 15009 – A Coruña o en la dirección de electrónico i...@allenta.com ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] Question: How many nodes can join a cluster?
On Mon, Oct 18, 2010 at 9:46 AM, Vogelsang, Andreas a.vogels...@uni-muenster.de wrote: Hello, I’m creating a presentation about a virtual Linux-HA Cluster. I just asked me how many nodes pacemaker can handle. Mr. Schwartzkopff wrote in his Book that Linux-HA version 2 can handle up to 16 Nodes. Is this also true for pacemaker? The algorithms in pacemaker place no theoretical limit on the number of supportable nodes. However in practice: - corosync/heartbeat may impose some (iirc, corosync isn't recommended much past 32) - the CIB and PE may struggle in the presence of hundreds of resources On the plus side, recent profiling of the CIB and PE made them much more efficient and able to handle the workload: http://theclusterguy.clusterlabs.org/post/1241986422/large-cluster-performance And we're working on the infrastructure needed to support resource management on non-cluster nodes. --- Mit freundlichen Grüßen / Best regards Andreas Vogelsang Westfälische Wilhelms-Universität Münster IVV 4 - Naturwissenschaften Raum 230, Institutsgruppe 1 Wilhelm-Klemm-Straße 10 48149 Münster, Germany Tel.: +49 (0)251/83-39130 Fax.: +49 (0)251/83-33669 E-Mail: a.vogels...@uni-muenster.de ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
