On Oct 5, 2007, at 12:00 PM, Anthony Tong wrote:
I'm getting file holes with pvfs 2.6.3 on linux 2.6 (rhel4 kernels
i386,
and a vanilla 2.6 as well) on a test system and can consistently
reproduce
them. Holes are about 16k+ of zeros.
Simple setup: 4 io servers, 1 metadata, over TCP, these nodes also
mount
the filesystem.
I am writing gigabyte files sequentially from a client via the vfs
interface
I finally had some time to do some debugging this morning and here's
what I have found so far. "io,client" is the gossip mask on for
pvfs2-client-core-threaded.
Hi Anthony,
Are you able to reproduce the problem with pvfs2-client-core? Also,
are you running pvfs2-client-core-threaded directly or through pvfs2-
client --threaded?
From the traces you've included below, it looks like you're mounting/
unmounting the filesystem over and over between each IO. Any reason
to do that?
-sam
For the first instance of the hole, I see "Posted UNKNOWN" in
the log. The offset (1301371504) corresponds with where the first
hole is in my test file.
Snippet from output of cmp -l good.file corrupt.file
1301371505 127 0
1301371506 376 0
1301371509 115 0
1301371510 221 0
1301371511 132 0
... (and so forth till)..
1301438544 110 0
1571986033 7 0
Searching for other "Posted UNKNOWN" messages and if there's a
file_req_off nearby, it corresponds for other holes as well.
Gossip snippets
[D 11:09:40.531507] * mem req size is 67040, file_req size is 67040
(bytes)
[D 11:09:40.531534] bstream_size = 325343856, datafile nr=1,
ct=4, file_req_off = 1301371504
[D 11:09:40.531712] posted flow for context 0xb4bfd720
[D 11:09:40.531790] preposting write ack for context 0xb4bfd720.
[D 11:09:41.563356] Posted UNKNOWN (waiting for test)
[D 11:09:41.563558] Posted UNKNOWN (waiting for test)
[D 11:09:41.640702] get_config state: server_get_config_setup_msgpair
[D 11:09:41.641900] Posted PVFS_SYS_FS_ADD (waiting for test)
[D 11:09:41.644099] * Adding new dynamic mount point <DYNAMIC-1> [7,0]
[D 11:09:41.644148] PINT_server_config_mgr_add_config: adding
config 0x84e6680
[D 11:09:41.644177] mapped fs_id 1867692515 => config 0x84e6680
[D 11:09:41.644218] Set min handle recycle time to 360 seconds
[D 11:09:41.644249] Reloading handle mappings for fs_id 1867692515
[D 11:09:41.644472] PVFS_isys_io entered [1048186]
[D 11:09:41.644548] (0x84f0c68) io state: io_init
[D 11:09:41.644582] (0x84f0c68) getattr_setup_msgpair
[D 11:09:41.644702] Posted PVFS_SYS_IO (waiting for test)
[D 11:09:41.645097] trying to add object reference to acache
[D 11:09:41.645138] (0x84f0c68) getattr state: getattr_cleanup
[D 11:09:41.645169] (0x84f0c68) io state: io_datafile_setup_msgpairs
[D 11:09:41.645201] - io_find_target_datafiles called
[D 11:09:41.645279] io_find_target_datafiles: datafile[1] might
have data (out=1)
[D 11:09:41.645319] io_find_target_datafiles: datafile[2] might
have data (out=2)
...
[D 11:09:55.609389] * mem req size is 100272, file_req size is
100272 (bytes)
[D 11:09:55.609417] bstream_size = 393019392, datafile nr=0,
ct=4, file_req_off = 1571986032
[D 11:09:55.609526] posted flow for context 0xb55fd318
[D 11:09:55.609554] preposting write ack for context 0xb55fd318.
[D 11:09:56.627065] Posted UNKNOWN (waiting for test)
[D 11:09:56.627238] Posted UNKNOWN (waiting for test)
[D 11:09:56.693300] get_config state: server_get_config_setup_msgpair
[D 11:09:56.694529] Posted PVFS_SYS_FS_ADD (waiting for test)
[D 11:09:56.700558] * Adding new dynamic mount point <DYNAMIC-1> [7,0]
[D 11:09:56.700620] PINT_server_config_mgr_add_config: adding
config 0x83c6680
[D 11:09:56.700650] mapped fs_id 1867692515 => config 0x83c6680
[D 11:09:56.700692] Set min handle recycle time to 360 seconds
[D 11:09:56.700735] Reloading handle mappings for fs_id 1867692515
[D 11:09:56.700954] PVFS_isys_io entered [1048186]
[D 11:09:56.701033] (0x83d0c68) io state: io_init
[D 11:09:56.701066] (0x83d0c68) getattr_setup_msgpair
[D 11:09:56.701188] Posted PVFS_SYS_IO (waiting for test)
[D 11:09:56.701589] trying to add object reference to acache
[D 11:09:56.701631] (0x83d0c68) getattr state: getattr_cleanup
[D 11:09:56.701663] (0x83d0c68) io state: io_datafile_setup_msgpairs
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
