yum install libibverbs tells me [r...@storage0 ~]# yum install libibverbs Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * addons: mirror.umoss.org * base: mirror.vcu.edu * extras: mirror.atlanticmetro.net * updates: holmes.umflint.edu Setting up Install Process Package libibverbs-1.1.3-2.el5.x86_64 already installed and latest version Package libibverbs-1.1.3-2.el5.i386 already installed and latest version Nothing to do
Perhaps it is installed but turned off somehow? .. Lana ([email protected]) On Tue, Oct 19, 2010 at 7:15 PM, Craig Carl <[email protected]> wrote: > Lana - > Looks like you have the IPoIB stack installed, but not support for > ibverbs. Let's try this - > > # yum install libibverbs > # service glusterd restart > > Thanks, > > Craig > > -- > Craig Carl > Senior Systems Engineer; Gluster, Inc. > Cell - (408) 829-9953 (California, USA) > Office - (408) 770-1884 > Gtalk - [email protected] > Twitter - @gluster > Installing Gluster Storage Platform, the movie! > http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ > > > ________________________________ > From: "Lana Deere" <[email protected]> > To: "Craig Carl" <[email protected]> > Cc: [email protected], [email protected] > Sent: Tuesday, October 19, 2010 4:02:11 PM > Subject: Re: [Gluster-users] hanging "df" (3.1, infiniband) > > They show up in ibhosts and I can ping or ssh via IPoIB to them, but > perhaps they are not completely configured properly. Or perhaps I > have mixed some references to the regular Ethernet into the > configuration for rdma? Anyway, here are the outputs you requested: > > [r...@storage0 ~]# lsmod > Module Size Used by > iptable_filter 36161 0 > ip_tables 55201 1 iptable_filter > x_tables 50505 1 ip_tables > fuse 83057 1 > autofs4 63049 3 > hidp 83521 2 > rfcomm 104937 0 > l2cap 89409 10 hidp,rfcomm > bluetooth 118853 5 hidp,rfcomm,l2cap > lockd 101553 0 > sunrpc 199945 2 lockd > cpufreq_ondemand 42449 8 > acpi_cpufreq 47937 0 > freq_table 38977 2 cpufreq_ondemand,acpi_cpufreq > ib_iser 69569 0 > libiscsi2 77765 1 ib_iser > scsi_transport_iscsi2 74073 2 ib_iser,libiscsi2 > scsi_transport_iscsi 35017 1 scsi_transport_iscsi2 > ib_srp 67465 0 > rds 401393 0 > ib_sdp 144285 0 > ib_ipoib 113057 0 > ipoib_helper 35537 2 ib_ipoib > ipv6 435489 77 ib_ipoib > xfrm_nalgo 43333 1 ipv6 > crypto_api 42945 1 xfrm_nalgo > rdma_ucm 47681 0 > rdma_cm 68437 4 ib_iser,rds,ib_sdp,rdma_ucm > ib_ucm 50121 0 > ib_uverbs 68720 2 rdma_ucm,ib_ucm > ib_umad 50153 0 > ib_cm 72809 4 ib_srp,ib_ipoib,rdma_cm,ib_ucm > iw_cm 43465 1 rdma_cm > ib_addr 41929 1 rdma_cm > ib_sa 74953 4 ib_srp,ib_ipoib,rdma_cm,ib_cm > mlx4_ib 94461 0 > ib_mad 70629 4 ib_umad,ib_cm,ib_sa,mlx4_ib > ib_core 104901 15 > ib_iser,ib_srp,rds,ib_sdp,ib_ipoib,rdma_ucm,rdma_cm,ib_ucm,ib_uverbs,ib_umad,ib_cm,iw_cm,ib_sa,mlx4_ib,ib_mad > xfs 508625 1 > loop 48721 0 > dm_mirror 54737 0 > dm_multipath 56921 0 > scsi_dh 42177 1 dm_multipath > raid456 152417 1 > xor 38865 1 raid456 > video 53197 0 > backlight 39873 1 video > sbs 49921 0 > power_meter 47053 0 > hwmon 36553 1 power_meter > i2c_ec 38593 1 sbs > dell_wmi 37601 0 > wmi 41985 1 dell_wmi > button 40545 0 > battery 43849 0 > asus_acpi 50917 0 > acpi_memhotplug 40517 0 > ac 38729 0 > parport_pc 62313 0 > lp 47121 0 > parport 73165 2 parport_pc,lp > mlx4_en 107985 0 > joydev 43969 0 > i2c_i801 41813 0 > igb 122709 0 > i2c_core 56641 2 i2c_ec,i2c_i801 > 8021q 57425 1 igb > shpchp 70893 0 > mlx4_core 152773 2 mlx4_ib,mlx4_en > serio_raw 40517 0 > dca 41221 1 igb > sg 70377 0 > pcspkr 36289 0 > dm_raid45 99657 0 > dm_message 36289 1 dm_raid45 > dm_region_hash 46145 1 dm_raid45 > dm_log 44993 3 dm_mirror,dm_raid45,dm_region_hash > dm_mod 101649 4 dm_mirror,dm_multipath,dm_raid45,dm_log > dm_mem_cache 38977 1 dm_raid45 > mpt2sas 159337 12 > scsi_transport_sas 66753 1 mpt2sas > ahci 69705 6 > libata 209489 1 ahci > sd_mod 56513 32 > scsi_mod 196953 10 > ib_iser,libiscsi2,scsi_transport_iscsi2,ib_srp,scsi_dh,sg,mpt2sas,scsi_transport_sas,libata,sd_mod > raid1 56001 3 > ext3 168913 2 > jbd 94769 1 ext3 > uhci_hcd 57433 0 > ohci_hcd 56309 0 > ehci_hcd 66125 0 > [r...@storage0 ~]# ibv_devinfo > libibverbs: Warning: no userspace device-specific driver found for > /sys/class/infiniband_verbs/uverbs0 > No IB devices found > [r...@storage0 ~]# lspci > 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22) > 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI > Express Root Port 1 (rev 22) > 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI > Express Root Port 3 (rev 22) > 00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express > Root Port 5 (rev 22) > 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI > Express Root Port 7 (rev 22) > 00:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI > Express Root Port 9 (rev 22) > 00:13.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC > Interrupt Controller (rev 22) > 00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management > Registers (rev 22) > 00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch > Pad Registers (rev 22) > 00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status > and RAS Registers (rev 22) > 00:14.3 PIC: Intel Corporation 5520/5500/X58 I/O Hub Throttle Registers (rev > 22) > 00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset > QuickData Technology Device (rev 22) > 00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset > QuickData Technology Device (rev 22) > 00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset > QuickData Technology Device (rev 22) > 00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset > QuickData Technology Device (rev 22) > 00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset > QuickData Technology Device (rev 22) > 00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset > QuickData Technology Device (rev 22) > 00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset > QuickData Technology Device (rev 22) > 00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset > QuickData Technology Device (rev 22) > 00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #4 > 00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #5 > 00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #6 > 00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 > EHCI Controller #2 > 00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #1 > 00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #2 > 00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB > UHCI Controller #3 > 00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 > EHCI Controller #1 > 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) > 00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface > Controller > 00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA > AHCI Controller > 00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network > Connection (rev 01) > 01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network > Connection (rev 01) > 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic > SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02) > 05:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe > 2.0 5GT/s - IB QDR / 10GigE] (rev b0) > 06:01.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW > WPCM450 (rev 0a) > [r...@storage0 ~]# /etc/init.d/openibd status > Low level hardware support loaded: > mlx4_ib > > Upper layer protocol modules: > ib_iser ib_srp rds ib_sdp ib_ipoib > > User space access modules: > rdma_ucm ib_ucm ib_uverbs ib_umad > > Connection management modules: > rdma_cm ib_cm iw_cm > > Configured IPoIB interfaces: ib0 > Currently active IPoIB interfaces: ib0 > [r...@storage0 ~]# > > > > .. Lana ([email protected]) > > > > > > > On Tue, Oct 19, 2010 at 6:48 PM, Craig Carl <[email protected]> wrote: >> Lana - >> The first couple of lines of the log identify our problem - >> >> [2010-10-19 07:47:49.315416] C [rdma.c:3817:rdma_init] rpc-transport/rdma: >> No IB devices found >> [2010-10-19 07:47:49.315438] E [rdma.c:4744:init] rdma.management: Failed >> to >> initialize IB Device >> [2010-10-19 07:47:49.315452] E [rpc-transport.c:965:rpc_transport_load] >> rpc-transport: 'rdma' initialization failed >> >> Are you sure your IB cards are working? Can you send the output of - >> >> # lsmod >> # ibv_devinfo >> # lspci >> # /etc/init.d/openibd status >> >> >> >> Thanks, >> >> Craig >> >> -- >> Craig Carl >> Senior Systems Engineer; Gluster, Inc. >> Cell - (408) 829-9953 (California, USA) >> Office - (408) 770-1884 >> Gtalk - [email protected] >> Twitter - @gluster >> Installing Gluster Storage Platform, the movie! >> http://rackerhacker.com/2010/08/11/one-month-with-glusterfs-in-production/ >> >> >> ________________________________ >> From: "Lana Deere" <[email protected]> >> To: "Craig Carl" <[email protected]> >> Cc: [email protected], [email protected] >> Sent: Tuesday, October 19, 2010 3:29:41 PM >> Subject: Re: [Gluster-users] hanging "df" (3.1, infiniband) >> >> For the last little while I've been using storage0 as both client and >> server, so those files are both client and server files at the same >> time. If it would be helpful, I could go back to using a different >> host as client (but then 'df' will hang instead of reporting the >> Transport message). >> >> [r...@storage0 ~]# cat /etc/glusterd/.cmd_log_history >> [2010-10-19 07:54:36.244333] peer probe : on host storage1:24007 >> [2010-10-19 07:54:36.249891] peer probe : on host storage1:24007 FAILED >> [2010-10-19 07:54:43.745558] peer probe : on host storage2:24007 >> [2010-10-19 07:54:43.750752] peer probe : on host storage2:24007 FAILED >> [2010-10-19 07:54:48.915378] peer probe : on host storage3:24007 >> [2010-10-19 07:54:48.920595] peer probe : on host storage3:24007 FAILED >> [2010-10-19 07:59:49.737251] Volume create : on volname: RaidData >> attempted >> [2010-10-19 07:59:49.737314] Volume create : on volname: RaidData >> type:DEFAULT count:4 bricks: storage0:/data storage1:/data >> storage2:/data storage3:/data >> [2010-10-19 07:59:49.737631] Volume create : on volname: RaidData SUCCESS >> [2010-10-19 08:01:36.909963] volume start : on volname: RaidData SUCCESS >> >> The /var/log file was pretty big, so I put it on pastebin: >> http://pastebin.com/m6WbHPUp >> >> >> .. Lana ([email protected]) >> >> >> >> >> >> >> On Tue, Oct 19, 2010 at 6:10 PM, Craig Carl <[email protected]> wrote: >>> Lana - >>> Can you also post the contents of >>> >>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log >>> and >>> /etc/glusterd/.cmd_log_history >>> >>> on both the client and server to the list? >> > _______________________________________________ Gluster-users mailing list [email protected] http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
