We spent quite a while tracking down where this error was coming from; I
think we have managed to track it down and have a hacked up patch that
fixes it for us. We'd love it if you could check if our understanding of
how clvmd and openais are interacting is correct and comment on our patch.
The "Cannot allocate memory" error is coming from _lock_resource in CLVM's
clvmd-openais.c. We print the "lock_resource '%s', flags=%d, mode=%d\n"
debugging message, so we know we get into the function. Then we exit
without printing any other messages, so we don't get as far as the
"lock_resource returning %d, lock_id=%llx\n" line. Thus we must be failing
in the call to saLckResourceLock, and then the debugging code is calling
ais_to_errno(err) which is turning the actual error into ENOMEM.
ais_to_errno maps SA_AIS_ERR_NO_MEMORY, SA_AIS_ERR_NO_RESOURCES, and
SA_AIS_ERR_NO_SECTIONS to ENOMEM. Looking in the OpenAIS code now, we
found via debugging messages that we make it into
message_handler_req_exec_lck_resourcelock in services/lck.c. Two of those
possible error codes can be generated here; SA_AIS_ERR_NO_MEMORY can be
generated if a malloc call fails, but assuming that doesn't (we have
plenty of free memory, so it's unlikely), the only other error code that
is generated by this function is SA_AIS_ERR_NO_RESOURCES from this snippet
of code:
if (global_lock_count == LCK_MAX_NUM_LOCKS) {
error = SA_AIS_ERR_NO_RESOURCES;
goto error_exit;
}
So indeed, if we bump the value of this constant in include/ipc_lck.h from
4096 to a higher number (we just used 524288, which is 4096*128), the
"Cannot allocate memory" errors disappear. As far as I can tell, this is
the only place that this constant is ever used in clvmd or openais (with
the exception of the asynchronous version of *_lck_resourcelock, and an
un-called function to query the current value of the constant).
Is it safe to bump this constant? Why was the value 4096 chosen? We found
a comment in the changelog about changing the maximum number of locks from
256 to 4096.
One thing we're confused about is why increasing the number of nodes in
the cluster increases the number of locks needed for CLVM.
The patch that we built our openais with is:
--- include/ipc_lck.old.h 2010-05-29 17:05:43.000000000 -0400
+++ include/ipc_lck.h 2010-05-29 17:06:22.000000000 -0400
@@ -75,7 +75,7 @@
* can be obtained via the library call saLckLimitGet
* by passing the appropriate limitId (see saLck.h).
*/
-#define LCK_MAX_NUM_LOCKS 4096
+#define LCK_MAX_NUM_LOCKS 524288
struct req_lib_lck_resourceopen {
coroipc_request_header_t header __attribute__((aligned(8)));
Thanks,
--Quentin
On Wed, 28 Apr 2010, Quentin Smith wrote:
> Nothing seems out of the ordinary; clvmd has a vsize of 51 MB, corosync has a
> vsize of 154 MB, both are using under 10 MB of real memory.
>
> http://web.mit.edu/quentin/Public/d-o-memory.txt
>
> --Quentin
>
> On Wed, 28 Apr 2010, Steven Dake wrote:
>
>> a list of processes with their memory consumption would be helpful.
>>
>> ps -auxx > memory.txt
>>
>> On Wed, 2010-04-28 at 16:12 -0400, Quentin Smith wrote:
>>> On Wed, 28 Apr 2010, Christine Caulfield wrote:
>>>
>>>>> (Needless to say, the servers all have plenty of RAM free...)
>>>>>
>>>>
>>>> There was a memory leak in clvmd up to 2.0.56 that might be having an
>>>> impact
>>>> here if there are a lot of clvmd operation being done. eg. monitoring
>>>> software that polls active LVs frequently.
>>>>
>>>> That's all I can think of off-hand that might cause that
>>>
>>> If it's running out of memory, it's not using up regular RAM - these
>>> machines have 32 GB of memory, and they all have at least several gigs of
>>> free memory. Also, this happens immediately after restarting clvmd.
>>>
>>> I turned up clvmd's logging (with clvmd -C -d 2); the lines relevant to
>>> this request don't seem to have much in the way of useful debugging:
>>>
>>> Got new connection on fd 5
>>> Read on local socket 5, len = 28
>>> creating pipe, [9, 10]
>>> Creating pre&post thread
>>> in sub thread: client = 0x69f010
>>> Sub thread ready for work.
>>> doing PRE command LOCK_VG 'V_xenvg' at 1 (client=0x69f010)
>>> lock_resource 'V_xenvg-1', flags=0, mode=1
>>> Created pre&post thread, state = 0
>>> Writing status 12 down pipe 10
>>> Waiting for next pre command
>>> read on PIPE 9: 4 bytes: status: 12
>>> background routine status was 12, sock_client=0x69f010
>>> Send local reply
>>>
>>> Is there another debugging knob I should turn up? Maybe corosync/openais
>>> logging?
>>>
>>> Thanks,
>>> --Quentin
>>> _______________________________________________
>>> Openais mailing list
>>> [email protected]
>>> https://lists.linux-foundation.org/mailman/listinfo/openais
>>
>>
>
_______________________________________________
Openais mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/openais
