在 2009-08-20四的 11:17 -0700，Sage Weil写道： > Hi Zhang, > > On Thu, 20 Aug 2009, Wen,Zhang wrote: > > I test the unstable branch, but get some new problems. > > I run iozone on 7 clients,but 4 clients fail: > > [ 750.287165] ceph read_partial_message f79c4780 data crc 3503132903 != > > exp. 1576047669 > > [ 750.287165] ceph osd0 192.168.1.211:6800 bad crc > > [ 755.582780] ceph osd0 192.168.1.211:6800 connection reset > > [ 755.582831] reset on osd0 > > I think this problem was fixed yesterday.. I forgot to mention in my email > that you may need to re-pull the latest unstable code. Can you do a 'git > pull' and verify it's working? I just ran a few iozones with your > arguments without problems. > > We'll do a v0.13 release shortly that includes these fixes. > > sage
I tried the lasted unstable code and it still has the 'data crc' problem sometimes. And I get a new problem as below. [ 529.473008] BUG: unable to handle kernel NULL pointer dereference at 00000002 [ 529.473107] IP: [<f8cf332c>] :ceph:crush_choose+0x77b/0xbc1 [ 529.473185] *pdpt = 0000000037960001 *pde = 0000000000000000 [ 529.473189] Oops: 0002 [#1] SMP [ 529.473271] Modules linked in: ceph nfs lockd nfs_acl sunrpc ipv6 loop iTCO_wdt serio_raw psmouse parport_pc evdev rng_core intel_agp pcspkr parport snd_hda_intel agpgart button snd_pcm snd_timer snd soundcore snd_page_alloc ext3 jbd mbcache sd_mod ide_pci_generic ata_piix piix ide_core ata_generic libata scsi_mod dock ehci_hcd uhci_hcd usbcore atl1e thermal processor fan thermal_sys [ 529.477001] [ 529.477001] Pid: 169, comm: pdflush Not tainted (2.6.26 #2) [ 529.477001] EIP: 0060:[<f8cf332c>] EFLAGS: 00010246 CPU: 0 [ 529.477001] EIP is at crush_choose+0x77b/0xbc1 [ceph] [ 529.477001] EAX: 00000002 EBX: f6dd0d80 ECX: f6dc0004 EDX: 00000000 [ 529.477001] ESI: 00000001 EDI: 00000002 EBP: 00000000 ESP: f7831c48 [ 529.477001] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 529.477001] Process pdflush (pid: 169, ti=f7830000 task=f747ed60 task.ti=f7830000) [ 529.477001] Stack: 4e67c653 f788d248 f6dc7a60 f6dd0d80 f6dc76c0 00000000 00000000 00000000 [ 529.477001] 00000000 00000000 04a4017f 00000000 004ffff8 00000000 00008000 c01760ea [ 529.477001] f7964a04 0000003b 00000010 000000d0 000000d0 f7401140 f740e800 f7403ad0 [ 529.477001] Call Trace: [ 529.477001] [<c01760ea>] cache_alloc_refill+0x62/0x44e [ 529.477001] [<f8cf3a2d>] crush_do_rule+0x2bb/0x396 [ceph] [ 529.477001] [<f8cf1177>] ceph_calc_pg_primary+0xef/0x11f [ceph] [ 529.477001] [<f8cf11d2>] ceph_calc_object_layout+0x2b/0x6b [ceph] [ 529.477001] [<f8cefe24>] map_osds+0x3e/0x9a [ceph] [ 529.477001] [<f8cefe91>] send_request+0x11/0xbc [ceph] [ 529.477001] [<f8cf08d0>] ceph_osdc_start_request+0x12a/0x164 [ceph] [ 529.477001] [<f8ce3988>] ceph_writepages_start+0x736/0x830 [ceph] [ 529.477001] [<f8ce3252>] ceph_writepages_start+0x0/0x830 [ceph] [ 529.477001] [<c015e86a>] do_writepages+0x20/0x30 [ 529.477001] [<c01913e6>] __writeback_single_inode+0x156/0x30c [ 529.477001] [<c019192e>] sync_sb_inodes+0x21f/0x317 [ 529.477001] [<c0191c20>] writeback_inodes+0x6b/0xb3 [ 529.477001] [<c015edee>] background_writeout+0x79/0xa8 [ 529.477001] [<c015f3c1>] pdflush+0x128/0x1c4 [ 529.477001] [<c015ed75>] background_writeout+0x0/0xa8 [ 529.477001] [<c015f299>] pdflush+0x0/0x1c4 [ 529.477001] [<c0133586>] kthread+0x38/0x5d [ 529.477001] [<c013354e>] kthread+0x0/0x5d [ 529.477001] [<c0104593>] kernel_thread_helper+0x7/0x10 [ 529.477001] ======================= [ 529.477001] Code: d0 89 44 24 48 f6 44 24 48 01 0f 84 ee fc ff ff d1 7c 24 48 8b 4c 24 0c 8b 5c 24 48 8b 41 10 8b 04 98 89 44 24 24 e9 b8 02 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 529.477001] EIP: [<f8cf332c>] crush_choose+0x77b/0xbc1 [ceph] SS:ESP 0068:f7831c48 [ 529.481854] ---[ end trace 989d125087067ae2 ]--- > > > > ÿÿ 2009-08-19ÿÿÿÿ 13:59 -0700ÿÿSage Weilÿÿÿÿÿÿ > > On Wed, 19 Aug 2009, Sage Weil wrote: > > > On Wed, 19 Aug 2009, Wen,Zhang wrote: > > > > Hi all, > > > > I write on one client but sometimes cannot read correctly on another. > > > > e.g,i write "hi\n" to a file,but read "hi\nhi\nhi\n...". > > > > > > Are you using v0.12, or the latest from git? (This should be fixed in > > > the > > > 'unstable' branch.) > > > > > > > And another problem is I always get kernel panic when i run iozone on > > > > client. > > > > I'm not seeing problems with the current unstable branch. What iozone > > args were you running with? > > > > sage > > > > > > > > > > I'll take a look, thanks! > > > sage > > > > > > > > > > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] ------------[ cut here ]------------ > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] invalid opcode: 0000 [#1] SMP > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] Process ceph-msgr/0 (pid: 2353, ti=f7844000 > > > > task=f7739200 task.ti=f7844000) > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] Stack: 00000008 f8f68f75 c3014ddc c3014ddc > > > > 00000000 f7845f6c 00000000 f7843b3c > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] f7843a08 f7843a58 f7843b38 f7843a4c > > > > f7843aa1 f7843a6c f7843a8a f7843b30 > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] f7843ad0 f7843a1c f7843a50 00000000 > > > > 00000000 00000000 00000000 ffffffff > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] Call Trace: > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] [<f8f68f75>] con_work+0xcb8/0x172d [ceph] > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] [<c011f134>] hrtick_set+0x7b/0xe6 > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] [<c02bba02>] schedule+0x6f3/0x735 > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] [<f8f682bd>] con_work+0x0/0x172d [ceph] > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] [<c012fc34>] run_workqueue+0x73/0xed > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] [<c012fd65>] worker_thread+0xb7/0xc3 > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574121] [<c013281e>] autoremove_wake_function+0x0/0x2d > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574183] [<c012fcae>] worker_thread+0x0/0xc3 > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574183] [<c01325ba>] kthread+0x38/0x5d > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574183] [<c0132582>] kthread+0x0/0x5d > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574183] [<c0104593>] kernel_thread_helper+0x7/0x10 > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574183] ======================= > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574183] Code: 2d 00 20 70 00 25 00 00 c0 ff 29 c2 89 d0 > > > > c1 e8 0c 8b 14 85 84 52 40 c0 4a 89 14 85 84 52 40 c0 85 d2 74 07 31 c0 > > > > 4a 75 15 eb 04 <0f> 0b eb fe 31 c0 81 3d 68 47 35 c0 68 47 35 c0 0f 95 > > > > c0 fe 05 > > > > > > > > Message from sysl...@kl22 at Aug 19 03:36:50 ... > > > > kernel:[ 1594.574183] EIP: [<c0162deb>] kunmap_high+0x4e/0x84 SS:ESP > > > > 0068:f7845eb8 > > > > > > > > > > > > > > > > BTW I patched the kernel according to the wiki. > > > > > > > > > > > > -- > > > > Wen,Zhang <wenz.zh...@gmail.com> > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 > > > > 30-Day > > > > trial. Simplify your report design, integration and deployment - and > > > > focus on > > > > what you do best, core application coding. Discover what's new with > > > > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > > > _______________________________________________ > > > > Ceph-devel mailing list > > > > Ceph-devel@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/ceph-devel > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 > > > 30-Day > > > trial. Simplify your report design, integration and deployment - and > > > focus on > > > what you do best, core application coding. Discover what's new with > > > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > > _______________________________________________ > > > Ceph-devel mailing list > > > Ceph-devel@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/ceph-devel > > > > > > > -- > Wen,Zhang <wenz.zh...@gmail.com> > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > Ceph-devel mailing list > Ceph-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/ceph-devel -- Wen,Zhang <wenz.zh...@gmail.com> ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Ceph-devel mailing list Ceph-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ceph-devel
