but disks pass all the tests with smartctl, badblocks and there isn't any error on disks. because the ssd has contain WAL/DB of OSDs it's difficult to test it on other cluster nodes
On Wed, Feb 21, 2018 at 4:58 PM, <[email protected]> wrote: > Could the problem be related with some faulty hardware (RAID-controller, > port, cable) but not disk? Does "faulty" disk works OK on other server? > > Behnam Loghmani wrote on 21/02/18 16:09: > >> Hi there, >> >> I changed the SSD on the problematic node with the new one and >> reconfigure OSDs and MON service on it. >> but the problem occurred again with: >> >> "rocksdb: submit_transaction error: Corruption: block checksum mismatch >> code = 2" >> >> I get fully confused now. >> >> >> >> On Tue, Feb 20, 2018 at 5:16 PM, Behnam Loghmani < >> [email protected] <mailto:[email protected]>> wrote: >> >> Hi Caspar, >> >> I checked the filesystem and there isn't any error on filesystem. >> The disk is SSD and it doesn't any attribute related to Wear level in >> smartctl and filesystem is >> mounted with default options and no discard. >> >> my ceph structure on this node is like this: >> >> it has osd,mon,rgw services >> 1 SSD for OS and WAL/DB >> 2 HDD >> >> OSDs are created by ceph-volume lvm. >> >> the whole SSD is on 1 vg. >> OS is on root lv >> OSD.1 DB is on db-a >> OSD.1 WAL is on wal-a >> OSD.2 DB is on db-b >> OSD.2 WAL is on wal-b >> >> output of lvs: >> >> data-a data-a -wi-a----- >> data-b data-b -wi-a----- >> db-a vg0 -wi-a----- >> db-b vg0 -wi-a----- >> root vg0 -wi-ao---- >> wal-a vg0 -wi-a----- >> wal-b vg0 -wi-a----- >> >> after making a heavy write on the radosgw, OSD.1 and OSD.2 has >> stopped with "block checksum >> mismatch" error. >> Now on this node MON and OSDs services has stopped working with this >> error >> >> I think my issue is related to this bug: >> http://tracker.ceph.com/issues/22102 >> <http://tracker.ceph.com/issues/22102> >> >> I ran >> #ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-1 --deep 1 >> but it returns the same error: >> >> *** Caught signal (Aborted) ** >> in thread 7fbf6c923d00 thread_name:ceph-bluestore- >> 2018-02-20 16:44:30.128787 7fbf6c923d00 -1 abort: Corruption: block >> checksum mismatch >> ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) >> luminous (stable) >> 1: (()+0x3eb0b1) [0x55f779e6e0b1] >> 2: (()+0xf5e0) [0x7fbf61ae15e0] >> 3: (gsignal()+0x37) [0x7fbf604d31f7] >> 4: (abort()+0x148) [0x7fbf604d48e8] >> 5: (RocksDBStore::get(std::string const&, char const*, unsigned >> long, >> ceph::buffer::list*)+0x1ce) [0x55f779d2b5ce] >> 6: (BlueStore::Collection::get_onode(ghobject_t const&, >> bool)+0x545) [0x55f779cd8f75] >> 7: (BlueStore::_fsck(bool, bool)+0x1bb5) [0x55f779cf1a75] >> 8: (main()+0xde0) [0x55f779baab90] >> 9: (__libc_start_main()+0xf5) [0x7fbf604bfc05] >> 10: (()+0x1bc59f) [0x55f779c3f59f] >> 2018-02-20 16:44:30.131334 7fbf6c923d00 -1 *** Caught signal >> (Aborted) ** >> in thread 7fbf6c923d00 thread_name:ceph-bluestore- >> >> ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) >> luminous (stable) >> 1: (()+0x3eb0b1) [0x55f779e6e0b1] >> 2: (()+0xf5e0) [0x7fbf61ae15e0] >> 3: (gsignal()+0x37) [0x7fbf604d31f7] >> 4: (abort()+0x148) [0x7fbf604d48e8] >> 5: (RocksDBStore::get(std::string const&, char const*, unsigned >> long, >> ceph::buffer::list*)+0x1ce) [0x55f779d2b5ce] >> 6: (BlueStore::Collection::get_onode(ghobject_t const&, >> bool)+0x545) [0x55f779cd8f75] >> 7: (BlueStore::_fsck(bool, bool)+0x1bb5) [0x55f779cf1a75] >> 8: (main()+0xde0) [0x55f779baab90] >> 9: (__libc_start_main()+0xf5) [0x7fbf604bfc05] >> 10: (()+0x1bc59f) [0x55f779c3f59f] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> needed to interpret this. >> >> -1> 2018-02-20 16:44:30.128787 7fbf6c923d00 -1 abort: >> Corruption: block checksum mismatch >> 0> 2018-02-20 16:44:30.131334 7fbf6c923d00 -1 *** Caught signal >> (Aborted) ** >> in thread 7fbf6c923d00 thread_name:ceph-bluestore- >> >> ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) >> luminous (stable) >> 1: (()+0x3eb0b1) [0x55f779e6e0b1] >> 2: (()+0xf5e0) [0x7fbf61ae15e0] >> 3: (gsignal()+0x37) [0x7fbf604d31f7] >> 4: (abort()+0x148) [0x7fbf604d48e8] >> 5: (RocksDBStore::get(std::string const&, char const*, unsigned >> long, >> ceph::buffer::list*)+0x1ce) [0x55f779d2b5ce] >> 6: (BlueStore::Collection::get_onode(ghobject_t const&, >> bool)+0x545) [0x55f779cd8f75] >> 7: (BlueStore::_fsck(bool, bool)+0x1bb5) [0x55f779cf1a75] >> 8: (main()+0xde0) [0x55f779baab90] >> 9: (__libc_start_main()+0xf5) [0x7fbf604bfc05] >> 10: (()+0x1bc59f) [0x55f779c3f59f] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> needed to interpret this. >> >> >> >> Could you please help me to recover this node or find a way to prove >> SSD disk problem. >> >> Best regards, >> Behnam Loghmani >> >> >> >> >> On Mon, Feb 19, 2018 at 1:35 PM, Caspar Smit <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Behnam, >> >> I would firstly recommend running a filesystem check on the >> monitor disk first to see if >> there are any inconsistencies. >> >> Is the disk where the monitor is running on a spinning disk or >> SSD? >> >> If SSD you should check the Wear level stats through smartctl. >> Maybe trim (discard) enabled on the filesystem mount? (discard >> could cause >> problems/corruption in combination with certain SSD firmwares) >> >> Caspar >> >> 2018-02-16 23:03 GMT+01:00 Behnam Loghmani < >> [email protected] >> <mailto:[email protected]>>: >> >> I checked the disk that monitor is on it with smartctl and it >> didn't return any error >> and it doesn't have any Current_Pending_Sector. >> Do you recommend any disk checks to make sure that this disk >> has problem and then I can >> send the report to the provider for replacing the disk >> >> On Sat, Feb 17, 2018 at 1:09 AM, Gregory Farnum < >> [email protected] >> <mailto:[email protected]>> wrote: >> >> The disk that the monitor is on...there isn't anything >> for you to configure about a >> monitor WAL though so I'm not sure how that enters into >> it? >> >> On Fri, Feb 16, 2018 at 12:46 PM Behnam Loghmani < >> [email protected] >> <mailto:[email protected]>> wrote: >> >> Thanks for your reply >> >> Do you mean, that's the problem with the disk I use >> for WAL and DB? >> >> On Fri, Feb 16, 2018 at 11:33 PM, Gregory Farnum < >> [email protected] >> <mailto:[email protected]>> wrote: >> >> >> On Fri, Feb 16, 2018 at 7:37 AM Behnam Loghmani < >> [email protected] >> <mailto:[email protected]>> wrote: >> >> Hi there, >> >> I have a Ceph cluster version 12.2.2 on >> CentOS 7. >> >> It is a testing cluster and I have set it up >> 2 weeks ago. >> after some days, I see that one of the three >> mons has stopped(out of >> quorum) and I can't start it anymore. >> I checked the mon service log and the output >> shows this error: >> >> """ >> mon.XXXXXX@-1(probing) e4 preinit clean up >> potentially inconsistent >> store state >> rocksdb: submit_transaction_sync error: >> Corruption: block checksum mismatch >> >> This bit is the important one. Your disk is bad >> and it’s feeding back >> corrupted data. >> >> >> >> >> code = 2 Rocksdb transaction: >> 0> 2018-02-16 17:37:07.041812 >> 7f45a1e52e40 -1 >> /home/jenkins-build/build/work >> space/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL >> E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUI >> LD/ceph-12.2.2/src/mon/MonitorDBStore.h: In >> function 'void >> >> MonitorDBStore::clear(std::set<std::basic_string<char> >> >&)' thread >> 7f45a1e52e40 time 2018-02-16 17:37:07.040846 >> /home/jenkins-build/build/work >> space/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL >> E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12. >> 2.2/rpm/el7/BUILD/ceph-12.2.2/src/mon/MonitorDBStore.h: >> 581: FAILE >> D assert(r >= 0) >> """ >> >> the only solution I found is to remove this >> mon from quorum and remove >> all mon data and re-add this mon to quorum >> again. >> and ceph goes to the healthy status again. >> >> but now after some days this mon has stopped >> and I face the same problem >> again. >> >> My cluster setup is: >> 4 osd hosts >> total 8 osds >> 3 mons >> 1 rgw >> >> this cluster has setup with ceph-volume lvm >> and wal/db separation on >> logical volumes. >> >> Best regards, >> Behnam Loghmani >> >> >> ______________________________ >> _________________ >> ceph-users mailing list >> [email protected] <mailto: >> [email protected]> >> http://lists.ceph.com/listinfo >> .cgi/ceph-users-ceph.com >> <http://lists.ceph.com/listinf >> o.cgi/ceph-users-ceph.com> >> >> >> >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] <mailto:[email protected]> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> >> >> >> >> >> >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
