Hi John, On Fri, 1 Aug 2008, John A. Khvatov wrote: > Hi all, > After a long copying a large number of files, I get it from kernel: > > [...] > > Probably it happened because disk space for MDS, ODSs and MON log files ended. > I have no logs with debug information from this deamons, sorry.
Ceph doesn't gracefully handle disk full situations yet, which explains the cosd failures (that's on the todo list for the next release). Also, at least one cause of the cosd assertion failure you saw there is fixed in the latest unstable branch. The client BUG is a sanity check, and suggests the dentry reference counting is off. Please let me know if/how you are able to reproduce it. I'll start some test runs, but I haven't seen that one before in my environment... thanks- sage > > After it, I clean disk spase and restarted deamons, but OSDs not running: > 080801.172323.838078 osd2 5 pg[2x0.33( empty ec=3 les=5 5/5/5) r=0 > (log bound mismatch, actual=[5'1,5'87] len=87) pct 0'0] read_log 4928 > 5'88 m 10000004ef8.00000000 by client2.1:90428 > 080801.172324.419781 osd2 5 load_pgs loaded pg[2x0.33( empty ec=3 > les=5 5/5/5) r=0 (log bound mismatch, actual=[5'1,5'88] len=88) pct > 0'0] log(0'0,0'0] > 080801.172324.426867 osd2 5 pg[2x1.12( empty ec=3 les=5 5/4/4) r=0 > pct 0'0] read_log [0,0) > 080801.172324.426887 osd2 5 load_pgs loaded pg[2x1.12( empty ec=3 > les=5 5/4/4) r=1] log(0'0,0'0] > 080801.172324.437768 osd2 5 pg[4x0.a1( empty ec=3 les=5 5/5/5) r=0 > pct 0'0] read_log [0,0) > 080801.172324.437788 osd2 5 load_pgs loaded pg[4x0.a1( empty ec=3 > les=5 5/5/5) r=0 pct 0'0] log(0'0,0'0] > 080801.172324.444511 osd2 5 pg[2x1.18( empty ec=3 les=5 5/4/4) r=0 > pct 0'0] read_log [0,0) > 080801.172324.444530 osd2 5 load_pgs loaded pg[2x1.18( empty ec=3 > les=5 5/4/4) r=1] log(0'0,0'0] > 080801.172324.453107 osd2 5 pg[2x1.19( empty ec=3 les=5 5/4/4) r=0 > pct 0'0] read_log [0,0) > 080801.172324.453126 osd2 5 load_pgs loaded pg[2x1.19( empty ec=3 > les=5 5/4/4) r=1] log(0'0,0'0] > 080801.172324.453444 osd2 5 pg[2x1.1d( empty ec=3 les=5 5/4/4) r=0 > pct 0'0] read_log [0,0) > 080801.172324.453464 osd2 5 load_pgs loaded pg[2x1.1d( empty ec=3 > les=5 5/4/4) r=1] log(0'0,0'0] > 080801.172324.453941 osd2 5 pg[3x1.7d( empty ec=3 les=5 5/5/5) r=0 > pct 0'0] read_log [0,0) > 080801.172324.453961 osd2 5 load_pgs loaded pg[3x1.7d( empty ec=3 > les=5 5/5/5) r=0 pct 0'0] log(0'0,0'0] > cosd: osd/PG.cc:1250: void PG::read_log(ObjectStore*): Assertion `r == > sizeof(ondisklog.bottom)' failed. > > > Cmon running successully and write to log: > 080801.173300.365849 mon0(leader).osd e5 e5: 4 osds: 3 up, 3 in > 080801.173300.365868 mon0(leader).mds e6 e6: 1 nodes: 1 up:replay > 080801.173300.365891 mon0(leader).pg v2612 v2612: 2304 pgs: 2304 > active+clean; 59 GB used, 835 GB / 894 GB free > > although all osds is down. > > -- > John A. Khvatov. > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Ceph-devel mailing list > Ceph-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/ceph-devel > > ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ Ceph-devel mailing list Ceph-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ceph-devel
