Re: max rbd devices
On Tue, Jul 28, 2015 at 5:19 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: Actually, its a 4.1rc8 kernel. And modinfo shows that the single_major parameter defaults to false. $ sudo modinfo rbd filename: /lib/modules/4.1.0-040100rc8-generic/kernel/drivers/block/rbd.ko license:GPL description:RADOS Block Device (RBD) driver author: Jeff Garzik j...@garzik.org author: Yehuda Sadeh yeh...@hq.newdream.net author: Sage Weil s...@newdream.net author: Alex Elder el...@inktank.com srcversion: C0EF017464E250E25E466E8 depends:libceph intree: Y vermagic: 4.1.0-040100rc8-generic SMP mod_unload modversions signer: Build time autogenerated kernel key sig_key:FC:1C:37:16:92:C8:E7:E5:17:9C:06:C5:5F:C4:DF:7E:19:00:A9:0F sig_hashalgo: sha512 parm: single_major:Use a single major number for all rbd devices (default: false) (bool) Yes, it defaults to false. But if rbd.ko isn't loaded by the time you do rbd map (which it shouldn't), rbd cli tool will load it for you with single_major=Y. That's why I asked if you loaded it by hand. Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: max rbd devices
Actually, its a 4.1rc8 kernel. And modinfo shows that the single_major parameter defaults to false. $ sudo modinfo rbd filename: /lib/modules/4.1.0-040100rc8-generic/kernel/drivers/block/rbd.ko license:GPL description:RADOS Block Device (RBD) driver author: Jeff Garzik j...@garzik.org author: Yehuda Sadeh yeh...@hq.newdream.net author: Sage Weil s...@newdream.net author: Alex Elder el...@inktank.com srcversion: C0EF017464E250E25E466E8 depends:libceph intree: Y vermagic: 4.1.0-040100rc8-generic SMP mod_unload modversions signer: Build time autogenerated kernel key sig_key:FC:1C:37:16:92:C8:E7:E5:17:9C:06:C5:5F:C4:DF:7E:19:00:A9:0F sig_hashalgo: sha512 parm: single_major:Use a single major number for all rbd devices (default: false) (bool) On Tue, Jul 28, 2015 at 7:58 AM, Ilya Dryomov idryo...@gmail.com wrote: On Tue, Jul 28, 2015 at 2:55 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: We are on 3.16 and Ceph 0.94.2, so I will try that parameter out and see how In this case, rbd map should have loaded the module with single_major=Y for you. Did you by any chance load it by hand? Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: max rbd devices
OK, I misunderstood. It is not loaded by hand. We have permanent mappings listed in /etc/ceph/rbdmap which then get mapped automatically when the system boots. So, in that case, shouldn't single_major=Y since it essentially is just calling rbd map from an init script? On Tue, Jul 28, 2015 at 10:24 AM, Ilya Dryomov idryo...@gmail.com wrote: On Tue, Jul 28, 2015 at 5:19 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: Actually, its a 4.1rc8 kernel. And modinfo shows that the single_major parameter defaults to false. $ sudo modinfo rbd filename: /lib/modules/4.1.0-040100rc8-generic/kernel/drivers/block/rbd.ko license:GPL description:RADOS Block Device (RBD) driver author: Jeff Garzik j...@garzik.org author: Yehuda Sadeh yeh...@hq.newdream.net author: Sage Weil s...@newdream.net author: Alex Elder el...@inktank.com srcversion: C0EF017464E250E25E466E8 depends:libceph intree: Y vermagic: 4.1.0-040100rc8-generic SMP mod_unload modversions signer: Build time autogenerated kernel key sig_key:FC:1C:37:16:92:C8:E7:E5:17:9C:06:C5:5F:C4:DF:7E:19:00:A9:0F sig_hashalgo: sha512 parm: single_major:Use a single major number for all rbd devices (default: false) (bool) Yes, it defaults to false. But if rbd.ko isn't loaded by the time you do rbd map (which it shouldn't), rbd cli tool will load it for you with single_major=Y. That's why I asked if you loaded it by hand. Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: max rbd devices
On Tue, Jul 28, 2015 at 5:59 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: Interesting to note - /etc/modules contains a line for rbd, which means it gets loaded at boot time and NOT via the rbd cli, so perhaps this is why it is loaded with the flag set to N. Yeah, that's probably it - try removing it and reboot. Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: max rbd devices
On Tue, Jul 28, 2015 at 5:35 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: OK, I misunderstood. It is not loaded by hand. We have permanent mappings listed in /etc/ceph/rbdmap which then get mapped automatically when the system boots. So, in that case, shouldn't single_major=Y since it essentially is just calling rbd map from an init script? Let's step back a second. What is the output of $ cat /sys/module/rbd/parameters/single_major right now? Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: max rbd devices
$ cat /sys/module/rbd/parameters/single_major N On Tue, Jul 28, 2015 at 10:44 AM, Ilya Dryomov idryo...@gmail.com wrote: On Tue, Jul 28, 2015 at 5:35 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: OK, I misunderstood. It is not loaded by hand. We have permanent mappings listed in /etc/ceph/rbdmap which then get mapped automatically when the system boots. So, in that case, shouldn't single_major=Y since it essentially is just calling rbd map from an init script? Let's step back a second. What is the output of $ cat /sys/module/rbd/parameters/single_major right now? Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: max rbd devices
Yup, thats the issue. We have some automated configuration stuff that added that line, I'll either remove it or add the single_major=Y to it. Either way, thanks for the help. On Tue, Jul 28, 2015 at 11:06 AM, Ilya Dryomov idryo...@gmail.com wrote: On Tue, Jul 28, 2015 at 5:59 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: Interesting to note - /etc/modules contains a line for rbd, which means it gets loaded at boot time and NOT via the rbd cli, so perhaps this is why it is loaded with the flag set to N. Yeah, that's probably it - try removing it and reboot. Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: max rbd devices
Interesting to note - /etc/modules contains a line for rbd, which means it gets loaded at boot time and NOT via the rbd cli, so perhaps this is why it is loaded with the flag set to N. On Tue, Jul 28, 2015 at 10:46 AM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: $ cat /sys/module/rbd/parameters/single_major N On Tue, Jul 28, 2015 at 10:44 AM, Ilya Dryomov idryo...@gmail.com wrote: On Tue, Jul 28, 2015 at 5:35 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: OK, I misunderstood. It is not loaded by hand. We have permanent mappings listed in /etc/ceph/rbdmap which then get mapped automatically when the system boots. So, in that case, shouldn't single_major=Y since it essentially is just calling rbd map from an init script? Let's step back a second. What is the output of $ cat /sys/module/rbd/parameters/single_major right now? Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Fwd: max rbd devices
We are on 3.16 and Ceph 0.94.2, so I will try that parameter out and see how it goes. Thanks for the tip! On Tue, Jul 28, 2015 at 2:38 AM, Ilya Dryomov idryo...@gmail.com wrote: On Mon, Jul 27, 2015 at 11:38 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: I found a thread from 2012 talking about the maximum number of RBD devices that can be mapped, it seems to cap out at around 228. There was discussion about possibly fixing this by using a registered/fixed major block device and bumping the minor device number for each new mapping. http://www.spinics.net/lists/ceph-devel/msg09362.html Did this ever progress any further or was an official bug ever filed? Im seeing this issue now and was just wondering if there are workarounds (other than not creating that many mappings). What are your kernel and ceph versions? It's been fixed in 3.14 with the addition of single_major parameter to rbd.ko. It's off by default because old userspace doesn't know how to deal with it. If you are on 3.14 or later and your userspace is new enough (0.75 or later IIRC), rbd map will load with single_major=Y. The new theoretical limit is 65536 rbd devices, but I must confess I haven't tried to map more than ~4000. Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: max rbd devices
On Mon, Jul 27, 2015 at 11:38 PM, Wyllys Ingersoll wyllys.ingers...@keepertech.com wrote: I found a thread from 2012 talking about the maximum number of RBD devices that can be mapped, it seems to cap out at around 228. There was discussion about possibly fixing this by using a registered/fixed major block device and bumping the minor device number for each new mapping. http://www.spinics.net/lists/ceph-devel/msg09362.html Did this ever progress any further or was an official bug ever filed? Im seeing this issue now and was just wondering if there are workarounds (other than not creating that many mappings). What are your kernel and ceph versions? It's been fixed in 3.14 with the addition of single_major parameter to rbd.ko. It's off by default because old userspace doesn't know how to deal with it. If you are on 3.14 or later and your userspace is new enough (0.75 or later IIRC), rbd map will load with single_major=Y. The new theoretical limit is 65536 rbd devices, but I must confess I haven't tried to map more than ~4000. Thanks, Ilya -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
max rbd devices
I found a thread from 2012 talking about the maximum number of RBD devices that can be mapped, it seems to cap out at around 228. There was discussion about possibly fixing this by using a registered/fixed major block device and bumping the minor device number for each new mapping. http://www.spinics.net/lists/ceph-devel/msg09362.html Did this ever progress any further or was an official bug ever filed? Im seeing this issue now and was just wondering if there are workarounds (other than not creating that many mappings). thanks, Wyllys Ingersoll -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
