I have a lustre MGS/MDT hosting one lustre filesystem with three OSTs
attached. When trying to attach a forth OST I see the following on the OST's
console and the mount command times out with "mount.lustre: mount
/dev/lustre2/ost at /mnt/data/ost failed: Input/output error Is the MGS
running?".
I am able to mount the lustre filesystem on this un-attachable OST node as a
client and am able to ping the MGS/MDT and vice versa.
# mkfs.lustre --reformat --fsname tmonster --ost
[EMAIL PROTECTED]'-N 1200000' /dev/lustre2/ost
...
# mount -t lustre /dev/lustre2/ost /mnt/data/ost
mount.lustre: mount /dev/lustre2/ost at /mnt/data/ost failed: Input/output
error
Is the MGS running?
# dmesg
Lustre: OBD class driver, [EMAIL PROTECTED]
Lustre Version: 1.6.4.2
Build Version:
1.6.4.2-19691231190000-PRISTINE-.cache.build.BUILD.lustre-kernel-2.6.18.lustre.linux-2.6.18-8.1.14.el5_lustre.1.6.4.2smp
Lustre: Added LNI [EMAIL PROTECTED] [8/256]
Lustre: Accept secure, port 988
Lustre: Lustre Client File System; [EMAIL PROTECTED]
kjournald starting. Commit interval 5 seconds
LDISKFS FS on dm-2, internal journal
LDISKFS-fs: mounted filesystem with ordered data mode.
LustreError: 2535:0:(obd_mount.c:247:ldd_parse()) cannot open
CONFIGS/mountdata: rc = -2
LustreError: 2535:0:(obd_mount.c:1252:server_kernel_mount()) premount parse
options failed: rc = -2
LustreError: 2535:0:(obd_mount.c:1533:server_fill_super()) Unable to mount
device /dev/lustre2/ost: -2
LustreError: 2535:0:(obd_mount.c:1924:lustre_fill_super()) Unable to mount
(-2)
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
bond0: no IPv6 routers present
eth0: no IPv6 routers present
eth1: no IPv6 routers present
kjournald starting. Commit interval 5 seconds
LDISKFS FS on dm-2, internal journal
LDISKFS-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
LDISKFS FS on dm-2, internal journal
LDISKFS-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
LDISKFS FS on dm-2, internal journal
LDISKFS-fs: mounted filesystem with ordered data mode.
LDISKFS-fs: file extents enabled
LDISKFS-fs: mballoc enabled
LustreError: 3157:0:(client.c:975:ptlrpc_expire_one_request()) @@@ timeout
(sent at 1203640178, 100s ago) [EMAIL PROTECTED] x2/t0 o253->
[EMAIL PROTECTED]@tcp_0:26 lens 4672/4672 ref 1 fl Rpc:/0/0 rc 0/-22
LustreError: 166-1: [EMAIL PROTECTED]: Connection to service MGS via nid
[EMAIL PROTECTED] was lost; in progress operations using this service will
fail.
LustreError: 3157:0:(obd_mount.c:954:server_register_target()) registration
with the MGS failed (-5)
LustreError: 3157:0:(obd_mount.c:1054:server_start_targets()) Required
registration failed for tmonster-OSTffff: -5
LustreError: 15f-b: Communication error with the MGS. Is the MGS running?
Lustre: [EMAIL PROTECTED]: Reactivating import
LustreError: 3157:0:(obd_mount.c:1570:server_fill_super()) Unable to start
targets: -5
Lustre: [EMAIL PROTECTED]: Connection restored to service MGS using nid
[EMAIL PROTECTED]
LustreError: 3157:0:(obd_mount.c:1368:server_put_super()) no obd
tmonster-OSTffff
LustreError: 3157:0:(obd_mount.c:119:server_deregister_mount())
tmonster-OSTffff not registered
LustreError: 11-0: an error occurred while communicating with
[EMAIL PROTECTED] The mgs_disconnect operation failed with -107
LDISKFS-fs: mballoc: 0 blocks 0 reqs (0 success)
LDISKFS-fs: mballoc: 0 extents scanned, 0 goal hits, 0 2^N hits, 0 breaks, 0
lost
LDISKFS-fs: mballoc: 0 generated and it took 0
LDISKFS-fs: mballoc: 0 preallocated, 0 discarded
Lustre: server umount tmonster-OSTffff complete
LustreError: 3157:0:(obd_mount.c:1924:lustre_fill_super()) Unable to mount
(-5)
MGS/MDT dmesg: (some of these are certainly unrelated to the OST's mount
cmd)
Lustre: 2807:0:(ldlm_lib.c:519:target_handle_reconnect()) tmonster-MDT0000:
3d7d98f5-470c-4188-8023-6c0023150148 reconnecting
Lustre: 2807:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 2 previous
similar messages
Lustre: 2806:0:(mds_reint.c:362:mds_steal_ack_locks()) Stealing 1 locks from
rs ffff81006ce30000 x895702.t458401 o101 NID [EMAIL PROTECTED]
Lustre: 2813:0:(service.c:751:ptlrpc_server_handle_reply()) All locks stolen
from rs ffff81006ce30000 x895702.t458401 o101 NID [EMAIL PROTECTED]
Lustre: 2797:0:(mds_reint.c:362:mds_steal_ack_locks()) Stealing 1 locks from
rs ffff81007f93d000 x666557.t458402 o101 NID [EMAIL PROTECTED]
Lustre: 2817:0:(service.c:751:ptlrpc_server_handle_reply()) All locks stolen
from rs ffff81007f93d000 x666557.t458402 o101 NID [EMAIL PROTECTED]
Lustre: 2688:0:(router.c:167:lnet_notify()) Ignoring prediction from
[EMAIL PROTECTED] of [EMAIL PROTECTED] down 7854805405 seconds in the future
Lustre: 2780:0:(ldlm_lib.c:519:target_handle_reconnect()) MGS:
8eba281a-43bd-3fa2-2491-fbab892dc02c reconnecting
Lustre: 2780:0:(ldlm_lib.c:519:target_handle_reconnect()) Skipped 2 previous
similar messages
Lustre: MGS: haven't heard from client 8eba281a-43bd-3fa2-2491-fbab892dc02c
(at [EMAIL PROTECTED]) in 72 seconds. I think it's dead, and I am evicting
it.
LustreError: 2780:0:(mgs_handler.c:515:mgs_handle()) lustre_mgs: operation
251 on unconnected MGS
LustreError: 2780:0:(ldlm_lib.c:1442:target_send_reply_msg()) @@@ processing
error (-107) [EMAIL PROTECTED] x7/t0 o251-><?>@<?>:-1 lens 128/0 ref 0
fl Interpret:/0/0 rc -107/0
LustreError: 2780:0:(ldlm_lib.c:1442:target_send_reply_msg()) Skipped 2
previous similar messages
Mounting the desired lustre filesystem as client on the OST that is having
problems --
# mount -t lustre [EMAIL PROTECTED]:/tmonster /mnt/tmonster
# df -h /mnt/tmonster
Filesystem Size Used Avail Use% Mounted on
[EMAIL PROTECTED]:/tmonster 2.8T 181G 2.6T 7% /mnt/tmonster
I have replaced this OSTs hardware (utilizing same boot/OST disks in
different blade) to no avail. Any help is highly appreciated.
Thanks,
Abe
_______________________________________________
Lustre-discuss mailing list
[email protected]
http://lists.lustre.org/mailman/listinfo/lustre-discuss
