Can you check the value of kernel.pid_max. This may have to be increased for larger OSD counts, it may have some bearing?
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of John Hogenmiller (yt) Sent: Friday, February 12, 2016 8:52 AM To: ceph-users@lists.ceph.com Subject: [ceph-users] ceph-disk activate fails (after 33 osd drives) I have 7 servers, each containing 60 x 6TB drives in jbod mode. When I first started, I only activated a couple drives on 3 nodes as Ceph OSDs. Yesterday, I went to expand to the remaining nodes as well as prepare and activate all the drives. ceph-disk prepare worked just fine. However, ceph-disk activate-all managed to only activate 33 drives and failed on the rest. This is consistent all 7 nodes (existing and newly installed). At the end of the day, I have 33 Ceph OSDs activated per server and can't activate any more. I did have to bump up the pg_num and pgp_num on the pool in order to accommodate the drives that did activate. I don't know if having a low pg number during the mass influx of OSDs caused an issue or not within the pool. I don't think so because I can only set the pg_num to a maximum value determined by the number of known OSDs. But maybe you have to expand slowly, increase pg's, expand osds, increase pgs in a slow fashion. I certainly have not seen anything to suggest a magic "33/node limit", and I've seen references to servers with up to 72 Ceph OSDs on them. I then attempted to activate individual ceph osd's and got the same set of errors. I even wiped a drive, re-ran `ceph-disk prepare` and `ceph-disk activate` to have it fail in the same way. status: ``` root@ljb01:/home/ceph/rain-cluster# ceph status cluster 4ebe7995-6a33-42be-bd4d-20f51d02ae45 health HEALTH_OK monmap e5: 5 mons at {hail02-r01-06=172.29.4.153:6789/0,hail02-r01-08=172.29.4.155:6789/0,rain02-r01-01=172.29.4.148:6789/0,rain02-r01-03=172.29.4.150:6789/0,rain02-r01-04=172.29.4.151:6789/0<http://172.29.4.153:6789/0,hail02-r01-08=172.29.4.155:6789/0,rain02-r01-01=172.29.4.148:6789/0,rain02-r01-03=172.29.4.150:6789/0,rain02-r01-04=172.29.4.151:6789/0>} election epoch 12, quorum 0,1,2,3,4 rain02-r01-01,rain02-r01-03,rain02-r01-04,hail02-r01-06,hail02-r01-08 osdmap e1116: 420 osds: 232 up, 232 in flags sortbitwise pgmap v397198: 10872 pgs, 14 pools, 101 MB data, 8456 objects 38666 MB used, 1264 TB / 1264 TB avail 10872 active+clean ``` Here is what I get when I run ceph-disk prepare on a blank drive: ``` root@rain02-r01-01:/etc/ceph# ceph-disk prepare /dev/sdbh1 The operation has completed successfully. The operation has completed successfully. meta-data=/dev/sdbh1 isize=2048 agcount=6, agsize=268435455 blks = sectsz=512 attr=2, projid32bit=0 data = bsize=4096 blocks=1463819665, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 The operation has completed successfully. root@rain02-r01-01:/etc/ceph# parted /dev/sdh print Model: ATA HUS726060ALA640 (scsi) Disk /dev/sdh: 6001GB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 2 1049kB 5369MB 5368MB ceph journal 1 5370MB 6001GB 5996GB xfs ceph data ``` And finally the errors from attempting to activate the drive. ``` root@rain02-r01-01:/etc/ceph# ceph-disk activate /dev/sdbh1 got monmap epoch 5 2016-02-12 12:53:43.340526 7f149bc71940 -1 journal FileJournal::_open: unable to setup io_context (0) Success 2016-02-12 12:53:43.340748 7f1493f83700 -1 journal io_submit to 0~4096 got (22) Invalid argument 2016-02-12 12:53:43.341186 7f149bc71940 -1 filestore(/var/lib/ceph/tmp/mnt.KRphD_) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory os/FileJournal.cc: In function 'int FileJournal::write_aio_bl(off64_t&, ceph::bufferlist&, uint64_t)' thread 7f1493f83700 time 2016-02-12 12:53:43.341355 os/FileJournal.cc: 1469: FAILED assert(0 == "io_submit got unexpected error") ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f149b767f2b] 2: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d] 3: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63] 4: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394] 5: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd] 6: (()+0x8182) [0x7f1499d87182] 7: (clone()+0x6d) [0x7f14980ce47d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2016-02-12 12:53:43.345434 7f1493f83700 -1 os/FileJournal.cc: In function 'int FileJournal::write_aio_bl(off64_t&, ceph::bufferlist&, uint64_t)' thread 7f1493f83700 time 2016-02-12 12:53:43.341355 os/FileJournal.cc: 1469: FAILED assert(0 == "io_submit got unexpected error") ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f149b767f2b] 2: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d] 3: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63] 4: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394] 5: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd] 6: (()+0x8182) [0x7f1499d87182] 7: (clone()+0x6d) [0x7f14980ce47d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -4> 2016-02-12 12:53:43.340526 7f149bc71940 -1 journal FileJournal::_open: unable to setup io_context (0) Success -3> 2016-02-12 12:53:43.340748 7f1493f83700 -1 journal io_submit to 0~4096 got (22) Invalid argument -1> 2016-02-12 12:53:43.341186 7f149bc71940 -1 filestore(/var/lib/ceph/tmp/mnt.KRphD_) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory 0> 2016-02-12 12:53:43.345434 7f1493f83700 -1 os/FileJournal.cc: In function 'int FileJournal::write_aio_bl(off64_t&, ceph::bufferlist&, uint64_t)' thread 7f1493f83700 time 2016-02-12 12:53:43.3 41355 os/FileJournal.cc: 1469: FAILED assert(0 == "io_submit got unexpected error”) ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f149b767f2b] 2: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d] 3: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63] 4: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394] 5: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd] 6: (()+0x8182) [0x7f1499d87182] 7: (clone()+0x6d) [0x7f14980ce47d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (Aborted) ** in thread 7f1493f83700 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7f149b67b2ca] 2: (()+0x10340) [0x7f1499d8f340] 3: (gsignal()+0x39) [0x7f149800acc9] 4: (abort()+0x148) [0x7f149800e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f1498915535] 6: (()+0x5e6d6) [0x7f14989136d6] 7: (()+0x5e703) [0x7f1498913703] 8: (()+0x5e922) [0x7f1498913922] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7f149b768118] 10: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d] 11: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63] 12: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394] 13: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd] 14: (()+0x8182) [0x7f1499d87182] 15: (clone()+0x6d) [0x7f14980ce47d] 2016-02-12 12:53:43.348498 7f1493f83700 -1 *** Caught signal (Aborted) ** in thread 7f1493f83700 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7f149b67b2ca] 2: (()+0x10340) [0x7f1499d8f340] 3: (gsignal()+0x39) [0x7f149800acc9] 4: (abort()+0x148) [0x7f149800e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f1498915535] 6: (()+0x5e6d6) [0x7f14989136d6] 7: (()+0x5e703) [0x7f1498913703] 8: (()+0x5e922) [0x7f1498913922] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7f149b768118] 10: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d] 11: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63] 12: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394] 13: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd] 14: (()+0x8182) [0x7f1499d87182] 15: (clone()+0x6d) [0x7f14980ce47d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 0> 2016-02-12 12:53:43.348498 7f1493f83700 -1 *** Caught signal (Aborted) ** in thread 7f1493f83700 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) 1: (()+0x7d02ca) [0x7f149b67b2ca] 2: (()+0x10340) [0x7f1499d8f340] 3: (gsignal()+0x39) [0x7f149800acc9] 4: (abort()+0x148) [0x7f149800e0d8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f1498915535] 6: (()+0x5e6d6) [0x7f14989136d6] 7: (()+0x5e703) [0x7f1498913703] 8: (()+0x5e922) [0x7f1498913922] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x7f149b768118] 10: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned long)+0x5ad) [0x7f149b5fe27d] 11: (FileJournal::do_aio_write(ceph::buffer::list&)+0x263) [0x7f149b602e63] 12: (FileJournal::write_thread_entry()+0x4e4) [0x7f149b607394] 13: (FileJournal::Writer::entry()+0xd) [0x7f149b44bddd] 14: (()+0x8182) [0x7f1499d87182] 15: (clone()+0x6d) [0x7f14980ce47d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ERROR:ceph-disk:Failed to activate Traceback (most recent call last): File "/usr/sbin/ceph-disk", line 3576, in <module> main(sys.argv[1:]) File "/usr/sbin/ceph-disk", line 3532, in main main_catch(args.func, args) File "/usr/sbin/ceph-disk", line 3554, in main_catch func(args) File "/usr/sbin/ceph-disk", line 2424, in main_activate dmcrypt_key_dir=args.dmcrypt_key_dir, File "/usr/sbin/ceph-disk", line 2197, in mount_activate (osd_id, cluster) = activate(path, activate_key_template, init) File "/usr/sbin/ceph-disk", line 2360, in activate keyring=keyring, File "/usr/sbin/ceph-disk", line 1950, in mkfs '--setgroup', get_ceph_user(), File "/usr/sbin/ceph-disk", line 349, in command_check_call return subprocess.check_call(arguments) File "/usr/lib/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd)subprocess.CalledProcessError: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '165', '--monmap', '/var/lib/ceph/tmp/mnt.KRphD_/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.KRphD_', '--osd-journal', '/var/lib/ceph/tmp/mnt.K root@rain02-r01-01:/etc/ceph# ls -l /var/lib/ceph/tmp/ total 0 -rw-r--r-- 1 root root 0 Feb 12 12:58 ceph-disk.activate.lock -rw-r--r-- 1 root root 0 Feb 12 12:58 ceph-disk.prepare.lock ```
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
