On Tue, Aug 21, 2007 at 10:27:10PM -0700, Brian Lynch wrote: > After launching to our production environment, crm_mon is no longer > returning the status. Crm_resources and and crm_standby seem similarly > afflicted. I've attached the log file from one system and the cib for > the environment. Here is a snippet of the log. I am assuming a switch > to classic transmissions may be warranted, but hate to lose the speed.
> > > Aug 21 22:13:49 avol01 cib: [26431]: ERROR: msg2wirefmt_ll: msg too > big(264245)for netstring fmt I'm afraid that your cluster is too big. This is a known issue. One easy remedy, in case you'll always use compression is to just increase the limit from 256k to 512k or 1mb (look for MAXMSG in clplumbing/ipc.h). This is hardcoded, so you'll have to recompile heartbeat. See also: http://old.linux-foundation.org/developer_bugzilla/show_bug.cgi?id=1339 > Aug 21 22:13:49 avol01 cib: [26431]: ERROR: hamsg2ipcmsg() failure > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: send_ipc_message: Could not > send IPC message to 15439 > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: crm_log_message_adv: > #========= IPC[outbound] message start ==========# > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: MSG: Dumping message with 7 > fields > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: MSG[0] : [t=cib] > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: MSG[1] : > [cib_clientid=63976b95-6cb6-47aa-af86-eaba018d4333] > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: MSG[2] : [cib_callopt=256] > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: MSG[3] : [cib_callid=2] > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: MSG[4] : [cib_op=cib_query] > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: MSG[5] : [cib_rc=0] > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: MSG[6] : > [(5)cib_calldata=0x6d70a8(214857 264083)] > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: <cib generated="true" > have_quorum="true" ignore_dtd="false" num_peers="8" ccm_transition="10" > dc_uuid="1944a895-8ada-49fe-8d3e-8fee07475580" cib_feature_revision="1" > admin_epoch="2" epoch="50" num_updates="5"/> > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: send_via_callback_channel: > Delivery of reply to client crm_mon/4651a7b2-9714-441b-a549-465454d048c4 > failed > > Aug 21 22:13:49 avol01 cib: [26431]: WARN: do_local_notify: A-Sync reply > to 15439 failed: reply failed > > Aug 21 22:15:21 avol01 cib: [26431]: ERROR: msg2wirefmt_ll: msg too > big(264215)for netstring fmt > > Aug 21 22:15:21 avol01 cib: [26431]: ERROR: hamsg2ipcmsg() failure > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: send_ipc_message: Could not > send IPC message to 15450 > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: crm_log_message_adv: > #========= IPC[outbound] message start ==========# > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: MSG: Dumping message with 7 > fields > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: MSG[0] : [t=cib] > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: MSG[1] : [cib_clientid=15450] > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: MSG[2] : [cib_callopt=4352] > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: MSG[3] : [cib_callid=2] > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: MSG[4] : [cib_op=cib_query] > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: MSG[5] : [cib_rc=0] > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: MSG[6] : > [(5)cib_calldata=0x105f468(214857 264083)] > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: <cib generated="true" > have_quorum="true" ignore_dtd="false" num_peers="8" ccm_transition="10" > dc_uuid="1944a895-8ada-49fe-8d3e-8fee07475580" cib_feature_revision="1" > admin_epoch="2" epoch="50" num_updates="5"/> > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: send_via_callback_channel: > Delivery of reply to client 15450/15450 failed > > Aug 21 22:15:21 avol01 cib: [26431]: WARN: do_local_notify: Sync reply > to 15450 failed: reply failed > > Aug 22 05:16:23 avol01 sshd[15458]: Connection closed by > ::ffff:10.4.99.243 > > Aug 21 22:18:06 avol01 cibadmin: [15472]: info: Invoked: cibadmin -Q -o > nodes > > Aug 21 22:18:18 avol01 crm_resource: [15475]: info: Invoked: > crm_resource -l > > Aug 21 22:18:18 avol01 cib: [26431]: ERROR: msg2wirefmt_ll: msg too > big(264215)for netstring fmt > > Aug 21 22:18:18 avol01 cib: [26431]: ERROR: hamsg2ipcmsg() failure > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: send_ipc_message: Could not > send IPC message to 15475 > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: crm_log_message_adv: > #========= IPC[outbound] message start ==========# > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: MSG: Dumping message with 7 > fields > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: MSG[0] : [t=cib] > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: MSG[1] : [cib_clientid=15475] > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: MSG[2] : [cib_callopt=4352] > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: MSG[3] : [cib_callid=2] > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: MSG[4] : [cib_op=cib_query] > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: MSG[5] : [cib_rc=0] > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: MSG[6] : > [(5)cib_calldata=0x105f468(214857 264083)] > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: <cib generated="true" > have_quorum="true" ignore_dtd="false" num_peers="8" ccm_transition="10" > dc_uuid="1944a895-8ada-49fe-8d3e-8fee07475580" cib_feature_revision="1" > admin_epoch="2" epoch="50" num_updates="5"/> > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: send_via_callback_channel: > Delivery of reply to client 15475/15475 failed > > Aug 21 22:18:18 avol01 cib: [26431]: WARN: do_local_notify: Sync reply > to 15475 failed: reply failed > > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
