Hi, Justin. We are using 0.12.1.
I thought it could be caused by our version of OTP (we had been using R14A) so I downgraded to R13B04. It didn't help though. Unfortunately, there's nothing interesting in logs, neither in Riak's nor in ours. I think that most likely there are some web requests that hang for some reason. Moreover, since we haven't been getting this problem before the last week I guess there's some problem in my own code that I have released recently. However, for now I got some more urgent tasks to finish so I just added a cron job that restarts Riak every hour. Guess I'll have to handle this problem in a week or so on. Anyway, thanks for getting in touch. As soon as I have time to investigate the problem I'll let you know the result. On Mon, Oct 18, 2010 at 6:11 PM, Justin Sheehy <[email protected]> wrote: > Hi, Dmitry. > > What version of Riak are you using? And is there anything interesting > in the error logs? > > -Justin > > > > > On Thu, Oct 14, 2010 at 7:53 AM, Dmitry Demeshchuk <[email protected]> > wrote: >> A small update. I've just encountered the same problem. Just about 3-4 >> hours have passed. >> >> lsof | wc -l showed only about 2k descriptors for all users. That's >> even more weird as the 32k descriptors limit is per user. So, we >> haven't reached the limit so far. >> >> On Thu, Oct 14, 2010 at 3:48 PM, Dmitry Demeshchuk <[email protected]> >> wrote: >>> Greetings. >>> >>> We have recently started to get the emfile errors. ulimit -n is 32767. >>> Restarting Riak helps for several hours and then we run out of >>> descriptors again. >>> >>> Some time later after restart I performed lsof and found the following >>> descriptors: >>> >>> kondemand 154 root cwd unknown >>> /proc/154/cwd (readlink: Permission denied) >>> kondemand 154 root rtd unknown >>> /proc/154/root (readlink: Permission denied) >>> kondemand 154 root txt unknown >>> /proc/154/exe (readlink: Permission denied) >>> kondemand 154 root NOFD >>> /proc/154/fd (opendir: Permission denied) >>> kondemand 155 root cwd unknown >>> /proc/155/cwd (readlink: Permission denied) >>> kondemand 154 root cwd unknown >>> /proc/154/cwd (readlink: Permission denied) >>> kondemand 154 root rtd unknown >>> /proc/154/root (readlink: Permission denied) >>> kondemand 154 root txt unknown >>> /proc/154/exe (readlink: Permission denied) >>> kondemand 154 root NOFD >>> /proc/154/fd (opendir: Permission denied) >>> kondemand 155 root cwd unknown >>> /proc/155/cwd (readlink: Permission denied) >>> kondemand 155 root rtd unknown >>> /proc/155/root (readlink: Permission denied) >>> kondemand 155 root txt unknown >>> /proc/155/exe (readlink: Permission denied) >>> kondemand 155 root NOFD >>> /proc/155/fd (opendir: Permission denied) >>> kondemand 156 root cwd unknown >>> /proc/156/cwd (readlink: Permission denied) >>> kondemand 156 root rtd unknown >>> /proc/156/root (readlink: Permission denied) >>> kondemand 156 root txt unknown >>> /proc/156/exe (readlink: Permission denied) >>> kondemand 156 root NOFD >>> /proc/156/fd (opendir: Permission denied) >>> kondemand 157 root cwd unknown >>> /proc/157/cwd (readlink: Permission denied) >>> kondemand 157 root rtd unknown >>> /proc/157/root (readlink: Permission denied) >>> kondemand 157 root txt unknown >>> /proc/157/exe (readlink: Permission denied) >>> kondemand 157 root NOFD >>> /proc/157/fd (opendir: Permission denied) >>> kondemand 158 root cwd unknown >>> /proc/158/cwd (readlink: Permission denied) >>> kondemand 158 root rtd unknown >>> /proc/158/root (readlink: Permission denied) >>> kondemand 158 root txt unknown >>> /proc/158/exe (readlink: Permission denied) >>> >>> Also, the following couple of descriptors is opened several times at >>> the same time: >>> >>> bash 20176 dem mem REG 252,0 256316 1179925 >>> /usr/lib/locale/en_US.utf8/LC_CTYPE >>> bash 20176 dem mem REG 252,0 54 1179926 >>> /usr/lib/locale/en_US.utf8/LC_NUMERIC >>> bash 20176 dem mem REG 252,0 2454 1179927 >>> /usr/lib/locale/en_US.utf8/LC_TIME >>> bash 20176 dem mem REG 252,0 966938 1179928 >>> /usr/lib/locale/en_US.utf8/LC_COLLATE >>> bash 20176 dem mem REG 252,0 286 1179929 >>> /usr/lib/locale/en_US.utf8/LC_MONETARY >>> bash 20176 dem mem REG 252,0 52 1179930 >>> /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES >>> bash 20176 dem mem REG 252,0 34 1179931 >>> /usr/lib/locale/en_US.utf8/LC_PAPER >>> bash 20176 dem mem REG 252,0 77 1179932 >>> /usr/lib/locale/en_US.utf8/LC_NAME >>> bash 20176 dem mem REG 252,0 155 1179933 >>> /usr/lib/locale/en_US.utf8/LC_ADDRESS >>> bash 20176 dem mem REG 252,0 59 1179934 >>> /usr/lib/locale/en_US.utf8/LC_TELEPHONE >>> bash 20176 dem mem REG 252,0 23 1179935 >>> /usr/lib/locale/en_US.utf8/LC_MEASUREMENT >>> bash 20176 dem mem REG 252,0 26048 917676 >>> /usr/lib/gconv/gconv-modules.cache >>> bash 20176 dem mem REG 252,0 373 1179936 >>> /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION >>> >>> Version of Riak is 0.12.1. There was a similar problem once and the >>> user was advised to make sure to use 0.12.1 >>> >>> Any ideas? >>> >>> -- >>> Best regards, >>> Dmitry Demeshchuk >>> >> >> >> >> -- >> Best regards, >> Dmitry Demeshchuk >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > -- Best regards, Dmitry Demeshchuk _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
