[OmniOS-discuss] Ang: RE: Re: zfs send via SRP or other RDMA enabled protocol

Johan Kragsterman Sun, 02 Nov 2014 10:35:58 -0800

-----Rune Tipsmark <[email protected]> skrev: -----

Till: Johan Kragsterman <[email protected]>
Från: Rune Tipsmark <[email protected]>
Datum: 2014-11-02 19:11
Kopia: David Bomba <[email protected]>, "[email protected]" <[email protected]>
Ärende: RE: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Hi Johan

Got two ZFS boxes (ZFS00 recv, ZFS10 send), both with IB and all configured and views made for vSphere which works just fine..

What I can't figure out is how to share a LUN with the other zfs box... see pasted info below...

ZFS00: (the box I want to receive my zfs snapshot)
The below are all ESX servers, cannot see the other ZFS box

root@zfs00:/pool03# stmfadm list-target -v
Target: eui.0002C90300095E7C
   Operational Status: Online
   Provider Name : srpt
   Alias : -
   Protocol : SRP
   Sessions : 8
   Initiator: eui.0002C903000F397C
   Alias: 8102c90300095e7e:0002c903000f397c
   Logged in since: Sun Nov 2 02:09:41 2014
   Initiator: eui.0002C903000F397B
   Alias: 8102c90300095e7d:0002c903000f397b
   Logged in since: Sun Nov 2 02:09:40 2014
   Initiator: eui.0002C903000D3D04
   Alias: 8102c90300095e7e:0002c903000d3d04
   Logged in since: Sat Nov 1 21:14:47 2014
   Initiator: eui.0002C90300104F47
   Alias: 8102c90300095e7d:0002c90300104f47
   Logged in since: Sat Nov 1 21:12:54 2014
   Initiator: eui.0002C903000D3D03
   Alias: 8102c90300095e7d:0002c903000d3d03
   Logged in since: Sat Nov 1 21:12:32 2014
   Initiator: eui.0002C90300104F48
   Alias: 8102c90300095e7e:0002c90300104f48
   Logged in since: Sat Nov 1 21:10:45 2014
   Initiator: eui.0002C903000A48FA
   Alias: 8102c90300095e7e:0002c903000a48fa
   Logged in since: Sat Nov 1 21:10:40 2014
   Initiator: eui.0002C903000D3CA0
   Alias: 8102c90300095e7e:0002c903000d3ca0
   Logged in since: Sat Nov 1 21:10:39 2014
Target: iqn.2010-09.org.napp-it:1394106801
   Operational Status: Online
   Provider Name : iscsit
   Alias : 03.06.2014
   Protocol : iSCSI
   Sessions : 0

root@zfs00:/pool03# stmfadm list-lu -v
LU Name: 600144F007780B7F00005455EDD50002
   Operational Status: Online
   Provider Name : sbd
   Alias : /pool03/LU11
   View Entry Count : 1
   Data File : /pool03/LU11
   Meta File : not set
   Size : 2199023255552
   Block Size : 512
   Management URL : not set
   Vendor ID : SUN
   Product ID : COMSTAR
   Serial Num : not set
   Write Protect : Disabled
   Writeback Cache : Disabled
   Access State : Active

root@zfs00:/pool03# stmfadm list-hg -v
Host Group: ESX
Host Group: Windows
Host Group: ESX-iSER
Host Group: OmniOS
   Member: eui.0002C903000923E6 <<--- The other ZFS box
   Member: iqn.2010-09.org.napp-it:1402013225
   Member: iqn.1986-03.com.sun:01:58cfb38a32ff.5390f58d

root@zfs00:/pool03# stmfadm list-view -l 600144F007780B7F00005455EDD50002
View Entry: 0
   Host group : OmniOS
   Target group : All
   LUN : 0

ZFS10: (the sending box where I want to see the LUN from ZFS00)
No disk show up from ZFS00...

Hi!

You got eui.0002C903000923E6 in host group OmniOS, but you don't have a session from that eui to the target.

Rgrds Johan

root@zfs10:/root# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
   0. c4t5000C50055FC9533d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50055fc9533
   1. c4t5000C50055FE6A63d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50055fe6a63
   2. c4t5000C500625B7EA7d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500625b7ea7
   3. c4t5000C500625B86E3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500625b86e3
   4. c4t5000C500625B886Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500625b886f
   5. c4t5000C500625B8137d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500625b8137
   6. c4t5000C500625B8427d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500625b8427
   7. c4t5000C500625BB773d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500625bb773
   8. c4t5000C500625BC2C3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500625bc2c3
   9. c4t5000C500625BD3EBd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500625bd3eb
   10. c4t5000C50057085A6Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50057085a6b
   11. c4t5000C50057086B67d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50057086b67
   12. c4t5000C50062878C0Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50062878c0b
   13. c4t5000C50062878C43d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50062878c43
   14. c4t5000C500570858EFd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500570858ef
   15. c4t5000C500570870D3d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c500570870d3
   16. c4t5000C5005708351Bd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c5005708351b
   17. c4t5000C5005708296Fd0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c5005708296f
   18. c4t5000C50057089753d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50057089753
   19. c4t5000C50057086307d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50057086307
   20. c4t5000C50062879687d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50062879687
   21. c4t5000C50062879707d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50062879707
   22. c4t5000C50062879723d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50062879723
   23. c4t5000C50062879787d0 <SEAGATE-ST4000NM0023-0004-3.64TB>
   /scsi_vhci/disk@g5000c50062879787
   24. c7t0d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
   /pci@0,0/pci15d9,704@1f,2/disk@0,0
   25. c7t1d0 <ATA-INTELSSDSC2BA10-0270 cyl 15571 alt 2 hd 224 sec 56>
   /pci@0,0/pci15d9,704@1f,2/disk@1,0
   26. c7t4d0 <ATA-InnoLite SATADOM-19-14.91GB>
   /pci@0,0/pci15d9,704@1f,2/disk@4,0
   27. c7t5d0 <ATA-InnoLite SATADOM-19-14.91GB>
   /pci@0,0/pci15d9,704@1f,2/disk@5,0
   28. c10d0 <Unknown-Unknown-0001-149.00GB>
   /pci@0,0/pci8086,3c08@3/pci103c,178b@0
   29. c11d0 <Unknown-Unknown-0001-149.00GB>
   /pci@0,0/pci8086,3c0a@3,2/pci103c,178b@0
   30. c12d0 <Unknown-Unknown-0001-298.02GB>
   /pci@79,0/pci8086,3c04@2/pci10b5,8616@0/pci10b5,8616@5/pci103c,178e@0
   31. c13d0 <Unknown-Unknown-0001-298.02GB>
   /pci@79,0/pci8086,3c04@2/pci10b5,8616@0/pci10b5,8616@6/pci103c,178e@0
   32. c14d0 <Unknown-Unknown-0001-298.02GB>
   /pci@79,0/pci8086,3c08@3/pci10b5,8616@0/pci10b5,8616@5/pci103c,178e@0
   33. c15d0 <Unknown-Unknown-0001-298.02GB>
   /pci@79,0/pci8086,3c08@3/pci10b5,8616@0/pci10b5,8616@6/pci103c,178e@0
Specify disk (enter its number):

-----Original Message-----
From: Johan Kragsterman [mailto:[email protected]]
Sent: Sunday, November 02, 2014 9:56 AM
To: Rune Tipsmark
Cc: David Bomba; [email protected]
Subject: Ang: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

-----"OmniOS-discuss" <[email protected]> skrev: -----
Till: David Bomba <[email protected]>
Från: Rune Tipsmark
Sänt av: "OmniOS-discuss"
Datum: 2014-11-02 07:55
Kopia: "[email protected]" <[email protected]>
Ärende: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

Sounds sensible, how do I do that?

I tried creating a view for a thin lu with my other zfs box, but how do I detect it?

First of all, if you run IB, forget the iscsi stuff, it's only creating un unnecessary IP layer that you don't need, and adds latency to your application.

Did you create SRP target service using COMSTAR?

# svcadm enable -r ibsrp/target

What's the output of "srptadm list-target" ?(on storage box), or you can also use "stmfadm list-target -v"

Do you got all necessary IB stuff, like a storage manager(OpenSM), in place? HCA ports shows up in dladm show-link?

If so, your host system should discover it as a local disk, just with "format", if you have created a view with the right eui.xxxxxxxxxxxx for the initiator HCA.

Rgrds Johan

I also stumbled across something else interesting, wondering if its possible to set up two identical boxes and create a pool with local/remote disks as per this article http://www.ssec.wisc.edu/~scottn/Lustre_ZFS_notes/lustre_zfs_srp_mirror.html

Br,

Rune

From: David Bomba [mailto:[email protected]]
Sent: Saturday, November 01, 2014 6:01 PM
To: Rune Tipsmark
Cc: [email protected]
Subject: Re: [OmniOS-discuss] zfs send via SRP or other RDMA enabled protocol

I usually mount a iSer target and perform ZFS send to the target. This was the best way to exploit the rdma bandwidth to its full potential.

On 2 Nov 2014, at 11:45 am, Rune Tipsmark <[email protected]> wrote:

Hi all,

Is it possible to do zfs send/recv via SRP or some other RMDA enabled protocol? IPoIB is really slow, about 50 MB/sec between two boxes, no disks are more than 10-15% busy.

If not, is there a way I can aggregate say 8 or 16 IPoIB partitions and push throughput to a more reasonable speed…

Br,

Rune

_______________________________________________
OmniOS-discuss mailing list
[email protected]
http://lists.omniti.com/mailman/listinfo/omnios-discuss

_______________________________________________
OmniOS-discuss mailing list
[email protected]
http://lists.omniti.com/mailman/listinfo/omnios-discuss

_______________________________________________
OmniOS-discuss mailing list
[email protected]
http://lists.omniti.com/mailman/listinfo/omnios-discuss

[OmniOS-discuss] Ang: RE: Re: zfs send via SRP or other RDMA enabled protocol

Reply via email to