Hi Wen,
I left iozone running continuously over the weekend and wasn't able to
reproduce either of these problems. Can you send me the output of
'git-rev-parse HEAD', and email (off-list) or post the output of 'objdump
-rdS ceph.ko' (it's big)?
I'm testing on x86_64, so it may just be a matter of me getting a 32bit
host set up for testing.
Thanks!
sage
On Fri, 21 Aug 2009, Wen,Zhang wrote:
> ÿÿ 2009-08-20ÿÿÿÿ 11:17 -0700ÿÿSage Weilÿÿÿÿÿÿ
> Hi Zhang,
>
> On Thu, 20 Aug 2009, Wen,Zhang wrote:
> > I test the unstable branch, but get some new problems.
> > I run iozone on 7 clients,but 4 clients fail:
> > [ 750.287165] ceph read_partial_message f79c4780 data crc 3503132903 !=
> > exp. 1576047669
> > [ 750.287165] ceph osd0 192.168.1.211:6800 bad crc
> > [ 755.582780] ceph osd0 192.168.1.211:6800 connection reset
> > [ 755.582831] reset on osd0
>
> I think this problem was fixed yesterday.. I forgot to mention in my email
> that you may need to re-pull the latest unstable code. Can you do a 'git
> pull' and verify it's working? I just ran a few iozones with your
> arguments without problems.
>
> We'll do a v0.13 release shortly that includes these fixes.
>
> sage
I tried the lasted unstable code and it still has the 'data crc' problem
sometimes.
And I get a new problem as below.
[ 529.473008] BUG: unable to handle kernel NULL pointer dereference at
00000002
[ 529.473107] IP: [<f8cf332c>] :ceph:crush_choose+0x77b/0xbc1
[ 529.473185] *pdpt = 0000000037960001 *pde = 0000000000000000
[ 529.473189] Oops: 0002 [#1] SMP
[ 529.473271] Modules linked in: ceph nfs lockd nfs_acl sunrpc ipv6
loop iTCO_wdt serio_raw psmouse parport_pc evdev rng_core intel_agp
pcspkr parport snd_hda_intel agpgart button snd_pcm snd_timer snd
soundcore snd_page_alloc ext3 jbd mbcache sd_mod ide_pci_generic
ata_piix piix ide_core ata_generic libata scsi_mod dock ehci_hcd
uhci_hcd usbcore atl1e thermal processor fan thermal_sys
[ 529.477001]
[ 529.477001] Pid: 169, comm: pdflush Not tainted (2.6.26 #2)
[ 529.477001] EIP: 0060:[<f8cf332c>] EFLAGS: 00010246 CPU: 0
[ 529.477001] EIP is at crush_choose+0x77b/0xbc1 [ceph]
[ 529.477001] EAX: 00000002 EBX: f6dd0d80 ECX: f6dc0004 EDX: 00000000
[ 529.477001] ESI: 00000001 EDI: 00000002 EBP: 00000000 ESP: f7831c48
[ 529.477001] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 529.477001] Process pdflush (pid: 169, ti=f7830000 task=f747ed60
task.ti=f7830000)
[ 529.477001] Stack: 4e67c653 f788d248 f6dc7a60 f6dd0d80 f6dc76c0
00000000 00000000 00000000
[ 529.477001] 00000000 00000000 04a4017f 00000000 004ffff8
00000000 00008000 c01760ea
[ 529.477001] f7964a04 0000003b 00000010 000000d0 000000d0
f7401140 f740e800 f7403ad0
[ 529.477001] Call Trace:
[ 529.477001] [<c01760ea>] cache_alloc_refill+0x62/0x44e
[ 529.477001] [<f8cf3a2d>] crush_do_rule+0x2bb/0x396 [ceph]
[ 529.477001] [<f8cf1177>] ceph_calc_pg_primary+0xef/0x11f [ceph]
[ 529.477001] [<f8cf11d2>] ceph_calc_object_layout+0x2b/0x6b [ceph]
[ 529.477001] [<f8cefe24>] map_osds+0x3e/0x9a [ceph]
[ 529.477001] [<f8cefe91>] send_request+0x11/0xbc [ceph]
[ 529.477001] [<f8cf08d0>] ceph_osdc_start_request+0x12a/0x164 [ceph]
[ 529.477001] [<f8ce3988>] ceph_writepages_start+0x736/0x830 [ceph]
[ 529.477001] [<f8ce3252>] ceph_writepages_start+0x0/0x830 [ceph]
[ 529.477001] [<c015e86a>] do_writepages+0x20/0x30
[ 529.477001] [<c01913e6>] __writeback_single_inode+0x156/0x30c
[ 529.477001] [<c019192e>] sync_sb_inodes+0x21f/0x317
[ 529.477001] [<c0191c20>] writeback_inodes+0x6b/0xb3
[ 529.477001] [<c015edee>] background_writeout+0x79/0xa8
[ 529.477001] [<c015f3c1>] pdflush+0x128/0x1c4
[ 529.477001] [<c015ed75>] background_writeout+0x0/0xa8
[ 529.477001] [<c015f299>] pdflush+0x0/0x1c4
[ 529.477001] [<c0133586>] kthread+0x38/0x5d
[ 529.477001] [<c013354e>] kthread+0x0/0x5d
[ 529.477001] [<c0104593>] kernel_thread_helper+0x7/0x10
[ 529.477001] =======================
[ 529.477001] Code: d0 89 44 24 48 f6 44 24 48 01 0f 84 ee fc ff ff d1
7c 24 48 8b 4c 24 0c 8b 5c 24 48 8b 41 10 8b 04 98 89 44 24 24 e9 b8 02
00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 529.477001] EIP: [<f8cf332c>] crush_choose+0x77b/0xbc1 [ceph] SS:ESP
0068:f7831c48
[ 529.481854] ---[ end trace 989d125087067ae2 ]---
>
>
>
> ÿÿ 2009-08-19ÿÿÿÿ 13:59 -0700ÿÿSage Weilÿÿÿÿÿÿ
> > On Wed, 19 Aug 2009, Sage Weil wrote:
> > > On Wed, 19 Aug 2009, Wen,Zhang wrote:
> > > > Hi all,
> > > > I write on one client but sometimes cannot read correctly on another.
> > > > e.g,i write "hi\n" to a file,but read "hi\nhi\nhi\n...".
> > >
> > > Are you using v0.12, or the latest from git? (This should be fixed in
> > > the
> > > 'unstable' branch.)
> > >
> > > > And another problem is I always get kernel panic when i run iozone on
> > > > client.
> >
> > I'm not seeing problems with the current unstable branch. What iozone
> > args were you running with?
> >
> > sage
> >
> >
> > >
> > > I'll take a look, thanks!
> > > sage
> > >
> > >
> > >
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] ------------[ cut here ]------------
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] invalid opcode: 0000 [#1] SMP
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] Process ceph-msgr/0 (pid: 2353, ti=f7844000
> > > > task=f7739200 task.ti=f7844000)
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] Stack: 00000008 f8f68f75 c3014ddc c3014ddc
> > > > 00000000 f7845f6c 00000000 f7843b3c
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] f7843a08 f7843a58 f7843b38 f7843a4c
> > > > f7843aa1 f7843a6c f7843a8a f7843b30
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] f7843ad0 f7843a1c f7843a50 00000000
> > > > 00000000 00000000 00000000 ffffffff
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] Call Trace:
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] [<f8f68f75>] con_work+0xcb8/0x172d [ceph]
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] [<c011f134>] hrtick_set+0x7b/0xe6
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] [<c02bba02>] schedule+0x6f3/0x735
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] [<f8f682bd>] con_work+0x0/0x172d [ceph]
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] [<c012fc34>] run_workqueue+0x73/0xed
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] [<c012fd65>] worker_thread+0xb7/0xc3
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574121] [<c013281e>] autoremove_wake_function+0x0/0x2d
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574183] [<c012fcae>] worker_thread+0x0/0xc3
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574183] [<c01325ba>] kthread+0x38/0x5d
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574183] [<c0132582>] kthread+0x0/0x5d
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574183] [<c0104593>] kernel_thread_helper+0x7/0x10
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574183] =======================
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574183] Code: 2d 00 20 70 00 25 00 00 c0 ff 29 c2 89 d0
> > > > c1 e8 0c 8b 14 85 84 52 40 c0 4a 89 14 85 84 52 40 c0 85 d2 74 07 31 c0
> > > > 4a 75 15 eb 04 <0f> 0b eb fe 31 c0 81 3d 68 47 35 c0 68 47 35 c0 0f 95
> > > > c0 fe 05
> > > >
> > > > Message from sysl...@kl22 at Aug 19 03:36:50 ...
> > > > kernel:[ 1594.574183] EIP: [<c0162deb>] kunmap_high+0x4e/0x84 SS:ESP
> > > > 0068:f7845eb8
> > > >
> > > >
> > > >
> > > > BTW I patched the kernel according to the wiki.
> > > >
> > > >
> > > > --
> > > > Wen,Zhang <wenz.zh...@gmail.com>
> > > >
> > > >
> > > > ------------------------------------------------------------------------------
> > > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> > > > 30-Day
> > > > trial. Simplify your report design, integration and deployment - and
> > > > focus on
> > > > what you do best, core application coding. Discover what's new with
> > > > Crystal Reports now. http://p.sf.net/sfu/bobj-july
> > > > _______________________________________________
> > > > Ceph-devel mailing list
> > > > Ceph-devel@lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/ceph-devel
> > > >
> > > >
> > >
> > > ------------------------------------------------------------------------------
> > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> > > 30-Day
> > > trial. Simplify your report design, integration and deployment - and
> > > focus on
> > > what you do best, core application coding. Discover what's new with
> > > Crystal Reports now. http://p.sf.net/sfu/bobj-july
> > > _______________________________________________
> > > Ceph-devel mailing list
> > > Ceph-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/ceph-devel
> > >
> > >
> --
> Wen,Zhang <wenz.zh...@gmail.com>
>
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now. http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Ceph-devel mailing list
> Ceph-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ceph-devel
--
Wen,Zhang <wenz.zh...@gmail.com>
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Ceph-devel mailing list
Ceph-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ceph-devel
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Ceph-devel mailing list
Ceph-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ceph-devel
