On Sep 19, 2013, at 6:10 PM, Gregory Farnum <[email protected]> wrote:
> On Wed, Sep 18, 2013 at 11:43 PM, Dan Van Der Ster > <[email protected]> wrote: >> >> On Sep 18, 2013, at 11:50 PM, Gregory Farnum <[email protected]> >> wrote: >> >>> On Wed, Sep 18, 2013 at 6:33 AM, Dan Van Der Ster >>> <[email protected]> wrote: >>>> Hi, >>>> We just finished debugging a problem with RBD-backed Glance image creation >>>> failures, and thought our workaround would be useful for others. >>>> Basically, we found that during an image upload, librbd on the glance api >>>> server was consuming many many processes, eventually hitting the 1024 >>>> nproc limit of non-root users in RHEL. The failure occurred when uploading >>>> to pools with 2048 PGs, but didn't fail when uploading to pools with 512 >>>> PGs (we're guessing that librbd is opening one thread per accessed-PG, and >>>> not closing those threads until the whole processes completes.) >>>> >>>> If you hit this same problem (and you run RHEL like us), you'll need to >>>> modify at least /etc/security/limits.d/90-nproc.conf (adding your non-root >>>> user that should be allowed > 1024 procs), and then also possibly run >>>> ulimit -u in the init script of your client process. Ubuntu should have >>>> some similar limits. >>> >>> Did your pools with 2048 PGs have a significantly larger number of >>> OSDs in them? Or are both pools on a pool with a lot of OSDs relative >>> to the PG counts? >> >> 1056 OSDs at the moment. >> >> Uploading a 14GB image we observed up to ~1500 threads. >> >> We set the glance client to allow 4096 processes for now. >> >> >>> The PG count shouldn't matter for this directly, but RBD (and other >>> clients) will create a couple messenger threads for each OSD it talks >>> to, and while they'll eventually shut down on idle it doesn't >>> proactively close them. I'd expect this to be a problem around 500 >>> OSDs. >> >> A couple, is that the upper limit? Should we be safe with ulimit -u 2*nOSDs >> +1 ?? > > The messenger currently generates 2 threads per daemon it communicates > with (although they will go away after a long enough idle period). > 2*nOSD+1 won't quite be enough as there's the monitor connection and a > handful of internal threads (I don't remember the exact numbers > off-hand). > > So far this hasn't been a problem for anybody and I doubt you'll see > issues, but at some point we will need to switch the messenger to use > epoll instead of a thread per socket. :) So we are the first ;) Anyway, for now Glance is using ulimit -u 4096, and this is not a big problem since we are in close contact with the people running those tests. But in future when the user base of Ceph grows here at our lab, it would be more friendly if the Ceph clients would fit under the default RHEL/Ubuntu ulimits. Cheers, Dan > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
