Hi Klaus, I do it by the weekend of the next week and write a patch using queue.
In addition, please tell me your opinion. Many thanks! Hideo Yamauchi. ----- Original Message ----- > From: Klaus Wenninger <kwenn...@redhat.com> > To: "users@clusterlabs.org" <users@clusterlabs.org> > Cc: > Date: 2016/5/14, Sat 00:51 > Subject: Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts > > On 05/13/2016 04:59 PM, renayama19661...@ybb.ne.jp wrote: >> i Klaus, >> >> After all we want transmission order. >> I think that I am going to use meta_attribute which you suggested and am > enough. >> >> (snip) >> <alert id="alert-1" >> path="/xxx/xxxx/xxxxxxx.sh"> >> <meta_attributes id="meta-alert-1-queue"> >> <nvpair id="alert-1-queue" name="queue" > value="alert1-queue" /> > Maybe we find a name that somehow implies what the purpose of the > queue is. Something like "serialization-queue" coming to my mind > although it is a little bit of an alliteration of course. >> </meta_attributes> >> </alert> >> <alert id="alert-2" >> path="/xxx/xxxx/xxxxxxx.sh"> >> >> (snip) >> >> I intend to write the correction that included a cue in this > meta_attribute, what do you think? >> I think that processing when queue was changed is troublesome, but think > that I can write the patch somehow. > The current implementation for the alerts-feature doesn't directly use > data from cib-diffs coming > in, but just triggers a query for the whole section, purges all local > data and feeds it in again from > what the query returns. > Thus I would just drain the queues - actively or just wait - before > deleting them together with > all the other local data. The alerts section is not expected to be > altered frequently during operation. >> >> We intend to make the correction that pcmk_snmp_helper.shkeeps versatility. > I tried to ask around a little bit to find out which tools were widely > used to collect and process > traps and how these are coping with traps arriving out of order because of > whatever reason (local reordering done by the scheduler - as we have it > here, > different delays on the network from different trap-sources, loss of > order on the > network, ...). > Unfortunately I didn't get real answers. > But maybe somebody here on the list can give a hint on that topic. > Anyway I found OID hrSystemDate (1.3.6.1.2.1.25.1.2) - part of MIB-2. > We could at least add this one to the pacemaker-mib-OIDs in > pcmk_snmp_helper.sh > and feed it with a correct timestamp (created by crmd). > >> >> We want to use this function in Pacemaker1.1.15. >> >> Best Regards, >> Hideo Yamauch. >> >> >> >> ----- Original Message ----- >>> From: "renayama19661...@ybb.ne.jp" > <renayama19661...@ybb.ne.jp> >>> To: "kwenn...@redhat.com" <kwenn...@redhat.com>; > "users@clusterlabs.org" <users@clusterlabs.org>; Cluster Labs - > All topics related to open-source clustering welcomed > <users@clusterlabs.org> >>> Cc: >>> Date: 2016/5/12, Thu 06:28 >>> Subject: Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts >>> >>> Hi Klaus, >>> >>> Thank you for comment. >>> >>> I confirm your comment. >>> I think that I ask you a question again. >>> >>> >>> Many thanks! >>> Hideo Yamauchi. >>> >>> >>> ----- Original Message ----- >>>> From: Klaus Wenninger <kwenn...@redhat.com> >>>> To: users@clusterlabs.org >>>> Cc: >>>> Date: 2016/5/11, Wed 14:13 >>>> Subject: Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts >>>> >>>> On 05/10/2016 11:19 PM, renayama19661...@ybb.ne.jp wrote: >>>>> Hi All, >>>>> >>>>> After all our member needs the control of the turn of the > transmission >>> of >>>> the SNMP trap. >>>>> We make a patch of the control of the turn of the > transmission and >>> intend >>>> to send it. >>>>> Probably, with the patch, we add the "ordered" > attribute >>> that we >>>> sent by an email before. >>>> Actually I still don't think that simple serialization of the > calling >>> of >>>> the snmptrap-tool >>>> is a good solution to tackle the problem of loosing the order of > traps >>>> arriving at >>>> some management station: >>>> >>>> - makes things worse in case of traps coming from multiple nodes >>>> - doesn't help when the order is lost on the network. >>>> >>>> Anyway I see 2 other scenarios where a certain degree of > serialization >>> might >>>> be helpful: >>>> >>>> - alert agent-scripts that can't handle being called > concurrently >>>> - performance issues that might arise on some systems that lack > the >>>> performance-headroom needed and/or the agent-scripts in place >>>> require significant effort and/or there are a lot of > resources/events >>>> that trigger a vast amount of alerts being handled in parallel >>>> >>>> So I could imagine the introduction of a meta-atribute that > specifies a >>>> queue >>>> to be used for serialization. >>>> >>>> - 'none' is default and leads to the behavior we have at > the >>> moment. >>>> - any other queue-name leads to the instantiation of an additional > queue >>>> >>>> This approach should allow merely any kind of serialization you > can think >>> of >>>> with as little impact as needed. >>>> e.g. if the agent doesn't cope with concurrent calls you use a > queue >>> per >>>> agent leading to all recipients being handled in a serialized way > (and of >>>> course the different alerts as well). And all the other agents are > running >>>> in parallel. >>>> e.g. you can have a separate queue for a single recipient leading > to >>>> the alerts being sent there being serialized. >>>> e.g. if the performance impact should be kept at a minimal level > you >>>> would use a single queue for all agents and all recipients >>>> >>>>> >>>>> Best Regards, >>>>> Hideo Yamauchi. >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "renayama19661...@ybb.ne.jp" >>>> <renayama19661...@ybb.ne.jp> >>>>>> To: "kwenn...@redhat.com" > <kwenn...@redhat.com>; >>>> "users@clusterlabs.org" <users@clusterlabs.org>; > Cluster >>> Labs - >>>> All topics related to open-source clustering welcomed >>>> <users@clusterlabs.org> >>>>>> Cc: >>>>>> Date: 2016/4/28, Thu 22:43 >>>>>> Subject: Re: [ClusterLabs] Coming in 1.1.15: Event-driven > alerts >>>>>> >>>>>> Hi Klaus, >>>>>> >>>>>> Because the script is performed the effectiveness of in > async, I >>> think >>>> that it >>>>>> is difficult to set "uptime" by the method of > the >>> sample. >>>>>> After all we may request the transmission of the order. >>>>>> #The patch before mine only controls a practice turn of > the async >>> and >>>> is not a >>>>>> thing giving load of crmd. >>>>>> >>>>>> Japan begins a rest for one week from tomorrow. >>>>>> I discuss after vacation with a member. >>>>>> >>>>>> Best Regards, >>>>>> Hideo Yamauchi. >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: Klaus Wenninger <kwenn...@redhat.com> >>>>>>> To: users@clusterlabs.org >>>>>>> Cc: >>>>>>> Date: 2016/4/28, Thu 03:14 >>>>>>> Subject: Re: [ClusterLabs] Coming in 1.1.15: > Event-driven >>> alerts >>>>>>> On 04/27/2016 04:19 PM, renayama19661...@ybb.ne.jp > wrote: >>>>>>>> Hi All, >>>>>>>> >>>>>>>> We have a request for a new SNMP function. >>>>>>>> >>>>>>>> >>>>>>>> The order of traps is not right. >>>>>>>> >>>>>>>> The turn of the trap is not sometimes followed. >>>>>>>> This is because the handling of notice carries > out >>>> "path" in >>>>>>> async. >>>>>>>> I think that it is necessary to wait for > completion of >>> the >>>> practice at >>>>>>> "path" unit of "alerts". >>>>>>>> >>>>>>>> The turn of the trap is different from the real > stop >>> order of >>>> the >>>>>> resource. >>>>>>> Writing the alerts in a local list and having the >>> alert-scripts >>>> called >>>>>>> in a serialized manner >>>>>>> would lead to the snmptrap-tool creating timestamps > in the >>> order >>>> of the >>>>>>> occurrence >>>>>>> of the alerts. >>>>>>> Having the snmp-manager order the traps by timestamp > this >>> would >>>> indeed >>>>>>> lead to >>>>>>> seeing them in the order they had occured. >>>>>>> >>>>>>> But this approach has a number of drawbacks: >>>>>>> >>>>>>> - it works just when the traps are coming from one > node as >>> there >>>> is no >>>>>>> way to serialize >>>>>>> over nodes - at least none that would work under > all >>>> circumstances we >>>>>>> want alerts >>>>>>> to be delivered >>>>>>> >>>>>>> - it distorts the timestamps created even more from > the >>> points in >>>> time >>>>>>> when the >>>>>>> alert had been triggered - making the result in a >>>> multi-node-scenario >>>>>>> even worse and >>>>>>> making it hard to correlate with other sources of >>> information >>>> like >>>>>>> logfiles >>>>>>> >>>>>>> - if you imagine a scenario with multiple mechanisms > of >>> delivering >>>> an >>>>>>> alert + multiple >>>>>>> recipients we couldn't use a single list but > we would >>> need >>>> something >>>>>> more >>>>>>> complicated to prevent unneeded delays, delays > coming from >>> one >>>> of the >>>>>>> delivery >>>>>>> methods not working properly due to e.g. a > recipient that >>> is not >>>>>>> reachable, ... >>>>>>> (all solvable of course but if it doesn't > solve your >>> problem >>>> in the >>>>>>> first place why the effort) >>>>>>> >>>>>>> The alternative approach taken doesn't create > the >>> timestamps >>>> in the >>>>>>> scripts but >>>>>>> provides timestamps to the scripts already. >>>>>>> This way it doesn't matter if the execution of > the script >>> is >>>> delayed. >>>>>>> >>>>>>> A short example how this approach could be used with > >>> snmp-traps: >>>>>>> edit pcmk_snmp_helper.sh: >>>>>>> >>>>>>> ... >>>>>>> starttickfile="/var/run/starttick" >>>>>>> >>>>>>> # hack to have a reference >>>>>>> # can have it e.g. in an attribute to be visible > throughout >>> the >>>> cluster >>>>>>> if [ ! -f ${starttickfile} ] ; then >>>>>>> echo ${CRM_alert_timestamp} > > ${starttickfile} >>>>>>> fi >>>>>>> >>>>>>> starttick=`cat ${starttickfile}` >>>>>>> ticks=`eval ${CRM_alert_timestamp} - ${starttick}` >>>>>>> >>>>>>> if [[ ${CRM_alert_rc} != 0 && > ${CRM_alert_task} == >>>>>> "monitor" >>>>>>> ]] || [[ >>>>>>> ${CRM_alert_task} != "monitor" ]] ; then >>>>>>> # This trap is compliant with PACEMAKER MIB >>>>>>> # >>>>>>> >>>> > https://github.com/ClusterLabs/pacemaker/blob/master/extra/PCMK-MIB.txt >>>>>>> /usr/bin/snmptrap -v 2c -c public > ${CRM_alert_recipient} >>>> ${ticks} >>>>>>> PACEMAKER-MIB::pacemakerNotificationTrap \ >>>>>>> PACEMAKER-MIB::pacemakerNotificationNode s >>>>>> "${CRM_alert_node}" >>>>>>> \ >>>>>>> PACEMAKER-MIB::pacemakerNotificationResource > s >>>>>>> "${CRM_alert_rsc}" \ >>>>>>> > PACEMAKER-MIB::pacemakerNotificationOperation s >>>>>>> "${CRM_alert_task}" \ >>>>>>> > PACEMAKER-MIB::pacemakerNotificationDescription s >>>>>>> "${CRM_alert_desc}" \ >>>>>>> PACEMAKER-MIB::pacemakerNotificationStatus i > >>>>>>> "${CRM_alert_status}" \ >>>>>>> > PACEMAKER-MIB::pacemakerNotificationReturnCode i >>>> ${CRM_alert_rc} >>>>>> \ >>>>>>> > PACEMAKER-MIB::pacemakerNotificationTargetReturnCode >>> i >>>>>>> ${CRM_alert_target_rc} && exit 0 || exit 1 >>>>>>> fi >>>>>>> >>>>>>> exit 0 >>>>>>> ... >>>>>>> >>>>>>> add a section to the cib: >>>>>>> >>>>>>> cibadmin --create --xml-text > '<configuration> >>>> <alerts> >>>>>> <alert >>>>>>> id="snmp_traps" >>>>>>> >>>> > path="/usr/share/pacemaker/tests/pcmk_snmp_helper.sh"> >>>>>>> <meta_attributes > id="meta_snmp_traps"> >>> <nvpair >>>>>>> id="snmp_timestamp" >>>>>>> name="tstamp_format" > value="%s%02N"/> >>>>>>> </meta_attributes> <recipient >>>>>>> id="trap_destination" >>>> value="192.168.123.3"/> >>>>>>> </alert> </alerts> >>>>>>> </configuration>' >>>>>>> >>>>>>> >>>>>>> This should solve the issue of correct order after > being >>> sorted by >>>>>>> timestamps >>>>>>> without having the ugly side-effects as described > above. >>>>>>> >>>>>>> I hope I understood your scenario correctly and this > small >>> example >>>>>>> points out how I roughly would suggest to cope with > the >>> issue. >>>>>>> Regards, >>>>>>> Klaus >>>>>>>> ---- >>>>>>>> [root@rh72-01 ~]# grep Operation > /var/log/ha-log | grep >>> stop >>>>>>>> Apr 25 18:48:48 rh72-01 crmd[28897]: notice: > Operation >>>>>> prmDummy1_stop_0: >>>>>>> ok (node=rh72-01, call=33, rc=0, cib-update=56, >>> confirmed=true) >>>>>>>> Apr 25 18:48:48 rh72-01 crmd[28897]: notice: > Operation >>>>>> prmDummy3_stop_0: >>>>>>> ok (node=rh72-01, call=37, rc=0, cib-update=57, >>> confirmed=true) >>>>>>>> Apr 25 18:48:48 rh72-01 crmd[28897]: notice: > Operation >>>>>> prmDummy4_stop_0: >>>>>>> ok (node=rh72-01, call=39, rc=0, cib-update=58, >>> confirmed=true) >>>>>>>> Apr 25 18:48:48 rh72-01 crmd[28897]: notice: > Operation >>>>>> prmDummy2_stop_0: >>>>>>> ok (node=rh72-01, call=35, rc=0, cib-update=59, >>> confirmed=true) >>>>>>>> Apr 25 18:48:48 rh72-01 crmd[28897]: notice: > Operation >>>>>> prmDummy5_stop_0: >>>>>>> ok (node=rh72-01, call=41, rc=0, cib-update=60, >>> confirmed=true) >>>>>>>> Apr 25 18:48:50 snmp-manager snmptrapd[6865]: > 2016-04-25 >>>> 18:48:50 >>>>>>> <UNKNOWN> [UDP: >>>>>>> >>> > [192.168.28.170]:40613->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance > > >>> >>>>>>> = Timeticks: (25512486) 2 days, >>>> 22:52:04.86#011SNMPv2-MIB::snmpTrapOID.0 = >>>>>> OID: >>>>>> >>> > PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode > > >>> >>>>>>> = STRING: >>>>>> >>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = >>>>>>> STRING: >>>>>> >>> "prmDummy3"#011PACEMAKER-MIB::pacemakerNotificationOperation >>>> = >>>>>>> STRING: >>>> > "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription >>>>>> = >>>>>>> STRING: >>>> "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = >>>>>> INTEGER: >>>>>>> 0#011PACEMAKER-MIB::pacemakerNotificationReturnCode > = >>> INTEGER: >>>>>>> > 0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = >>>> INTEGER: 0 >>>>>>>> Apr 25 18:48:50 snmp-manager snmptrapd[6865]: > 2016-04-25 >>>> 18:48:50 >>>>>>> <UNKNOWN> [UDP: >>>>>>> >>> > [192.168.28.170]:39581->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance > > >>> >>>>>>> = Timeticks: (25512489) 2 days, >>>> 22:52:04.89#011SNMPv2-MIB::snmpTrapOID.0 = >>>>>> OID: >>>>>> >>> > PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode > > >>> >>>>>>> = STRING: >>>>>> >>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = >>>>>>> STRING: >>>>>> >>> "prmDummy4"#011PACEMAKER-MIB::pacemakerNotificationOperation >>>> = >>>>>>> STRING: >>>> > "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription >>>>>> = >>>>>>> STRING: >>>> "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = >>>>>> INTEGER: >>>>>>> 0#011PACEMAKER-MIB::pacemakerNotificationReturnCode > = >>> INTEGER: >>>>>>> > 0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = >>>> INTEGER: 0 >>>>>>>> Apr 25 18:48:50 snmp-manager snmptrapd[6865]: > 2016-04-25 >>>> 18:48:50 >>>>>>> <UNKNOWN> [UDP: >>>>>>> >>> > [192.168.28.170]:37166->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance > > >>> >>>>>>> = Timeticks: (25512490) 2 days, >>>> 22:52:04.90#011SNMPv2-MIB::snmpTrapOID.0 = >>>>>> OID: >>>>>> >>> > PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode > > >>> >>>>>>> = STRING: >>>>>> >>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = >>>>>>> STRING: >>>>>> >>> "prmDummy1"#011PACEMAKER-MIB::pacemakerNotificationOperation >>>> = >>>>>>> STRING: >>>> > "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription >>>>>> = >>>>>>> STRING: >>>> "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = >>>>>> INTEGER: >>>>>>> 0#011PACEMAKER-MIB::pacemakerNotificationReturnCode > = >>> INTEGER: >>>>>>> > 0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = >>>> INTEGER: 0 >>>>>>>> Apr 25 18:48:50 snmp-manager snmptrapd[6865]: > 2016-04-25 >>>> 18:48:50 >>>>>>> <UNKNOWN> [UDP: >>>>>>> >>> > [192.168.28.170]:53502->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance > > >>> >>>>>>> = Timeticks: (25512494) 2 days, >>>> 22:52:04.94#011SNMPv2-MIB::snmpTrapOID.0 = >>>>>> OID: >>>>>> >>> > PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode > > >>> >>>>>>> = STRING: >>>>>> >>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = >>>>>>> STRING: >>>>>> >>> "prmDummy2"#011PACEMAKER-MIB::pacemakerNotificationOperation >>>> = >>>>>>> STRING: >>>> > "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription >>>>>> = >>>>>>> STRING: >>>> "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = >>>>>> INTEGER: >>>>>>> 0#011PACEMAKER-MIB::pacemakerNotificationReturnCode > = >>> INTEGER: >>>>>>> > 0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = >>>> INTEGER: 0 >>>>>>>> Apr 25 18:48:50 snmp-manager snmptrapd[6865]: > 2016-04-25 >>>> 18:48:50 >>>>>>> <UNKNOWN> [UDP: >>>>>>> >>> > [192.168.28.170]:45956->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance > > >>> >>>>>>> = Timeticks: (25512497) 2 days, >>>> 22:52:04.97#011SNMPv2-MIB::snmpTrapOID.0 = >>>>>> OID: >>>>>> >>> > PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode > > >>> >>>>>>> = STRING: >>>>>> >>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = >>>>>>> STRING: >>>>>> >>> "prmDummy5"#011PACEMAKER-MIB::pacemakerNotificationOperation >>>> = >>>>>>> STRING: >>>> > "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription >>>>>> = >>>>>>> STRING: >>>> "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = >>>>>> INTEGER: >>>>>>> 0#011PACEMAKER-MIB::pacemakerNotificationReturnCode > = >>> INTEGER: >>>>>>> > 0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = >>>> INTEGER: 0 >>>>>>>> ---- >>>>>>>> >>>>>>>> I think that there is "timestamp" > attribute >>> for >>>> async by >>>>>> this >>>>>>> change. >>>>>>>> The order of traps may be important to a user. >>>>>>>> I suggest addition to "alert" element > with >>>>>> "orderd" >>>>>>> attribute. >>>>>>>> * orderd >>>>>>>> false : The present processing. >>>>>>>> true : Control the transmission order of > the trap. >>>>>>>> >>>>>>>> ---- >>>>>>>> <configuration> >>>>>>>> <alerts> >>>>>>>> <alert id="notify_9" >>>>>>>> >>>> path="/usr/share/pacemaker/tests/pcmk_alert_sample1.sh" >>>>>>> ordered="true"> >>>>>>>> (snip) >>>>>>>> </alert> >>>>>>>> <alert id="notify_9" >>>>>>>> >>>> path="/usr/share/pacemaker/tests/pcmk_alert_sample2.sh" >>>>>>> ordered="false"> >>>>>>>> (snip) >>>>>>>> </alert> >>>>>>>> </alerts> >>>>>>>> </configuration> >>>>>>>> >>>>>>>> ---- >>>>>>>> >>>>>>>> I send a patch to cope with this problem > before. >>>>>>>> The former patch may be useful for the > correction. >>>>>>>> * > https://github.com/ClusterLabs/pacemaker/pull/847 >>>>>>>> >>>>>>>> I intend to write the patch if everybody agrees > to >>>> "ordered" >>>>>>> attribute. >>>>>>>> Best Regards, >>>>>>>> Hideo Yamauchi. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list: Users@clusterlabs.org >>>>>>>> http://clusterlabs.org/mailman/listinfo/users >>>>>>>> >>>>>>>> Project Home: http://www.clusterlabs.org >>>>>>>> Getting started: >>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>>>> Bugs: http://bugs.clusterlabs.org >>>>>>> _______________________________________________ >>>>>>> Users mailing list: Users@clusterlabs.org >>>>>>> http://clusterlabs.org/mailman/listinfo/users >>>>>>> >>>>>>> Project Home: http://www.clusterlabs.org >>>>>>> Getting started: >>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>>> Bugs: http://bugs.clusterlabs.org >>>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list: Users@clusterlabs.org >>>>>> http://clusterlabs.org/mailman/listinfo/users >>>>>> >>>>>> Project Home: http://www.clusterlabs.org >>>>>> Getting started: >>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>> Bugs: http://bugs.clusterlabs.org >>>>>> >>>>> _______________________________________________ >>>>> Users mailing list: Users@clusterlabs.org >>>>> http://clusterlabs.org/mailman/listinfo/users >>>>> >>>>> Project Home: http://www.clusterlabs.org >>>>> Getting started: >>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>> Bugs: http://bugs.clusterlabs.org >>>> >>>> _______________________________________________ >>>> Users mailing list: Users@clusterlabs.org >>>> http://clusterlabs.org/mailman/listinfo/users >>>> >>>> Project Home: http://www.clusterlabs.org >>>> Getting started: > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>> Bugs: http://bugs.clusterlabs.org >>>> >>> _______________________________________________ >>> Users mailing list: Users@clusterlabs.org >>> http://clusterlabs.org/mailman/listinfo/users >>> >>> Project Home: http://www.clusterlabs.org >>> Getting started: > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>> Bugs: http://bugs.clusterlabs.org >>> > > > _______________________________________________ > Users mailing list: Users@clusterlabs.org > http://clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > _______________________________________________ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org