Re: [Need Help]isboot (iSCSI boot driver) version 0.2.1
Thank you for your advice! 128KB is a default MAXPHYS value. You may rise it in your kernel if you want. I am successfully using 1MB MAXPHYS now. I misunderstood. I thought maxio was able to overwrite in GENERIC kernel. OK, I built custom kernel. There is a result. It's great! The log shows: client request 1 PDU(SCSI OP=0x28) with TL=1MB (2048 blocks). target response 4 PDUs with MRDSL=256KB (SCSI status is embedded in last BHS). --- iSCSI client: FreeBSD 8.1-RC2 amd64 + isboot 0.2.2 (maxio=1M) iSCSI target: FreeBSD 7.3 amd64 + istgt 20100606 (QueueDepth 64) Notes: both side use em(4) with MTU=1500. test.dat was created before rebooting. [r...@fbs8-isboot /usr/tmp]# dd if=./test.dat of=/dev/null bs=1m count=1k 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 9.357166 secs (114750753 bytes/sec) [r...@fbs8-isboot /usr/tmp]# dd if=/dev/zero of=./test0.dat bs=1m count=1k 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 15.100469 secs (71106521 bytes/sec) /usr/src/sys/amd64/conf/GENERIC (added): options MAXPHYS=(1024*1024) /boot/loader.conf: isboot_load="YES" /etc/sysctl.conf: vfs.read_max=2048 kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.recvbuf_inc=524288 net.inet.tcp.inflight.enable=0 --- When enabling log at istgt (previous 16KB and queued 1MB read): istgt_iscsi.c: 430:istgt_iscsi_read_pdu: BHS read 48 istgt_iscsi.c: 242:istgt_iscsi_read: Read 48 bytes (no padding) istgt_iscsi.c:4870:worker: isid=804953420001, tsih=1, cid=1, op=1 istgt_iscsi.c:4352:istgt_iscsi_execute: opcode 1 CDB 28 00 02 c1 80 ff 00 08 (... 00 00 00 00 00 00 00 00 istgt_iscsi.c:2934:istgt_iscsi_op_scsi: I=0, F=1, R=1, W=0, Attr=1, ITT=3bf7, TL=1048576 istgt_iscsi.c:2939:istgt_iscsi_op_scsi: CmdSN=15350, ExpStatSN=15351, StatSN=15351, ExpCmdSN=15350, MaxCmdSN=15412 istgt_lu.c:2227:istgt_lu_execute: LU99: PG=0x0001, Name=iqn.2007-09.jp.ne.peach:isboot1, LUN=0x0002 istgt_lu_disk.c:4162:istgt_lu_disk_queue: Queue(0), CmdSN=15350, OP=0x28, LUN=0x0002 istgt_lu_disk.c:4175:istgt_lu_disk_queue: insert Simple istgt_iscsi.c:3088:istgt_iscsi_op_scsi: Queue OK istgt_iscsi.c:4890:worker: isid=804953420001, tsih=1, cid=1, op=1 complete istgt_iscsi.c:4760:worker: kevent sock 40 (timeout 2ms) istgt_lu_disk.c:3540:istgt_lu_disk_lbread: Read 16384/16384 bytes istgt_lu_disk.c:5726:istgt_lu_disk_execute: SCSI OP=0x28, LUN=0x0002 status=0x0, complete istgt_iscsi.c:3219:istgt_iscsi_task_response: SCSI response CmdSN=15349 istgt_iscsi.c:2776:istgt_iscsi_transfer_in: Transfer 16384 istgt_iscsi.c:2811:istgt_iscsi_transfer_in: Transfer=16384, Offset=0, Len=16384 istgt_iscsi.c:2814:istgt_iscsi_transfer_in: StatSN=15351, DataSN=0, Offset=0, Len=16384 istgt_iscsi.c: 678:istgt_iscsi_write_pdu: BHS write 48 istgt_iscsi.c: 367:istgt_iscsi_write: Write 48 bytes (no padding) istgt_iscsi.c: 744:istgt_iscsi_write_pdu: Data write 16384 istgt_iscsi.c: 367:istgt_iscsi_write: Write 16384 bytes (no padding) istgt_iscsi.c:4760:worker: kevent sock 40 (timeout 2ms) istgt_lu_disk.c:4510:istgt_lu_disk_queue_start: LU99: LUN2 queue end istgt_lu_disk.c:4250:istgt_lu_disk_queue_count: LU99: LUN2 queue(1) istgt_lu_disk.c:4284:istgt_lu_disk_queue_start: LU99: LUN2 queue start istgt_lu_disk.c:4480:istgt_lu_disk_queue_start: LU99: LUN2 Task Read Start istgt_lu_disk.c:4617:istgt_lu_disk_execute: SCSI OP=0x28, LUN=0x0002 istgt_lu_disk.c:5129:istgt_lu_disk_execute: READ_10(lba 46235903, len 2048 blocks) istgt_lu_disk.c:3514:istgt_lu_disk_lbread: Read: max=67108864, lba=46235903, len=2048 istgt_lu_disk.c:3540:istgt_lu_disk_lbread: Read 1048576/1048576 bytes istgt_lu_disk.c:5726:istgt_lu_disk_execute: SCSI OP=0x28, LUN=0x0002 status=0x0, complete istgt_lu_disk.c:4510:istgt_lu_disk_queue_start: LU99: LUN2 queue end istgt_iscsi.c:3219:istgt_iscsi_task_response: SCSI response CmdSN=15350 istgt_iscsi.c:2776:istgt_iscsi_transfer_in: Transfer 1048576 istgt_iscsi.c:2811:istgt_iscsi_transfer_in: Transfer=1048576, Offset=0, Len=262144 istgt_iscsi.c:2814:istgt_iscsi_transfer_in: StatSN=15352, DataSN=0, Offset=0, Len=262144 istgt_iscsi.c: 678:istgt_iscsi_write_pdu: BHS write 48 istgt_iscsi.c: 367:istgt_iscsi_write: Write 48 bytes (no padding) istgt_iscsi.c: 744:istgt_iscsi_write_pdu: Data write 262144 istgt_iscsi.c: 367:istgt_iscsi_write: Write 262144 bytes (no padding) istgt_iscsi.c:2811:istgt_iscsi_transfer_in: Transfer=1048576, Offset=262144, Len=262144 istgt_iscsi.c:2814:istgt_iscsi_transfer_in: StatSN=15352, DataSN=1, Offset=262144, Len=262144 istgt_iscsi.c: 678:istgt_iscsi_write_pdu: BHS write 48 istgt_iscsi.c: 367:istgt_iscsi_write: Write 48 bytes (no padding) istg
Re: [Need Help]isboot (iSCSI boot driver) version 0.2.1
> iscsi initiator paniced over 4 dev, and the code has limitations of > devices, it is hardcoded to 4 > > 9f2ae5be (scottl 2007-07-24 15:35:02 + 39) #define ISCSIDEV > "iscsi" > 9f2ae5be (scottl 2007-07-24 15:35:02 + 40) > 9f2ae5be (scottl 2007-07-24 15:35:02 + 41) #define > ISCSI_MAX_TARGETS 4 //64 > 9f2ae5be (scottl 2007-07-24 15:35:02 + 42) > 9f2ae5be (scottl 2007-07-24 15:35:02 + 43) #define ISCSI_MAX_LUNS > 4 > > when setting this to 64, than paniced the kernel > get ftp://ftp.cs.huji.ac.il/users/danny/freebsd/iscsi-2.2.4.tar.gz but what does this have to to with the Subject:? danny > > On 7/4/10, Alexander Motin wrote: > > Daisuke Aoyama wrote: > Notes/Known Issues/Limitations: > FreeBSD can't use transfer length > 64KB. > >>> > >>> Since 8.0 FreeBSD can use any transfer lengths. 64K is a safety limit > >>> for CAM SIMs that do not report maximum transfer size. If your driver > >>> supports bigger transactions (and even if not), you should fill maxio > >>> field in XPT_PATH_INQ response. > >> > >> I set maxio=1024*1024 in version 0.2.2. As a result, the request (each > >> ccb) > >> have 256 blocks (128KB). I don't know why it is 128KB. > > > > 128KB is a default MAXPHYS value. You may rise it in your kernel if you > > want. I am successfully using 1MB MAXPHYS now. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [Need Help]isboot (iSCSI boot driver) version 0.2.1
iscsi initiator paniced over 4 dev, and the code has limitations of devices, it is hardcoded to 4 9f2ae5be (scottl 2007-07-24 15:35:02 + 39) #define ISCSIDEV"iscsi" 9f2ae5be (scottl 2007-07-24 15:35:02 + 40) 9f2ae5be (scottl 2007-07-24 15:35:02 + 41) #define ISCSI_MAX_TARGETS 4 //64 9f2ae5be (scottl 2007-07-24 15:35:02 + 42) 9f2ae5be (scottl 2007-07-24 15:35:02 + 43) #define ISCSI_MAX_LUNS 4 when setting this to 64, than paniced the kernel On 7/4/10, Alexander Motin wrote: > Daisuke Aoyama wrote: Notes/Known Issues/Limitations: FreeBSD can't use transfer length > 64KB. >>> >>> Since 8.0 FreeBSD can use any transfer lengths. 64K is a safety limit >>> for CAM SIMs that do not report maximum transfer size. If your driver >>> supports bigger transactions (and even if not), you should fill maxio >>> field in XPT_PATH_INQ response. >> >> I set maxio=1024*1024 in version 0.2.2. As a result, the request (each >> ccb) >> have 256 blocks (128KB). I don't know why it is 128KB. > > 128KB is a default MAXPHYS value. You may rise it in your kernel if you > want. I am successfully using 1MB MAXPHYS now. > > -- > Alexander Motin > ___ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [Need Help]isboot (iSCSI boot driver) version 0.2.1
Daisuke Aoyama wrote: >>> Notes/Known Issues/Limitations: >>> FreeBSD can't use transfer length > 64KB. >> >> Since 8.0 FreeBSD can use any transfer lengths. 64K is a safety limit >> for CAM SIMs that do not report maximum transfer size. If your driver >> supports bigger transactions (and even if not), you should fill maxio >> field in XPT_PATH_INQ response. > > I set maxio=1024*1024 in version 0.2.2. As a result, the request (each ccb) > have 256 blocks (128KB). I don't know why it is 128KB. 128KB is a default MAXPHYS value. You may rise it in your kernel if you want. I am successfully using 1MB MAXPHYS now. -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [Need Help]isboot (iSCSI boot driver) version 0.2.1
Updated to 0.2.2 I noticed a bug after writing previous mail. sosend was called from XPT_SCSI_IO with locked mutex. It caused "sleeping thread owns a non-sleepable lock". What's new?: add auto sense. add maxio=1m. modify max tags by iSCSI command window. fix locked sleep problem. Download links: http://www.peach.ne.jp/archives/isboot/isboot-0.2.2.tar.gz Download links(for testing purpose only): http://www.peach.ne.jp/archives/isboot/demo/FreeBSD-7.3-RELEASE-amd64-isboot-0.2.2.iso http://www.peach.ne.jp/archives/isboot/demo/FreeBSD-7.3-RELEASE-i386-isboot-0.2.2.iso http://www.peach.ne.jp/archives/isboot/demo/FreeBSD-8.1-RC2-amd64-isboot-0.2.2.iso http://www.peach.ne.jp/archives/isboot/demo/FreeBSD-8.1-RC2-i386-isboot-0.2.2.iso http://www.peach.ne.jp/archives/isboot/demo/unionfs-mkisboot.sh -- Daisuke Aoyama ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [Need Help]isboot (iSCSI boot driver) version 0.2.1
Hi, Notes/Known Issues/Limitations: FreeBSD can't use transfer length > 64KB. Since 8.0 FreeBSD can use any transfer lengths. 64K is a safety limit for CAM SIMs that do not report maximum transfer size. If your driver supports bigger transactions (and even if not), you should fill maxio field in XPT_PATH_INQ response. I set maxio=1024*1024 in version 0.2.2. As a result, the request (each ccb) have 256 blocks (128KB). I don't know why it is 128KB. It seems CAM tag is used only 2 of 64 tags. I don't know a reason, but MAXPHYS limit? How many concurrent requests you are submitting? If you are testing it with sequential read from file system - make sure you have sysctl vfs.read_max set high enough. MAXPHYS does not affects number of concurrent requests, only size of each one. I used vfs.read_max=32 and run only one dd. Now vfs.read_max=2048. I tried to run dd 4 times at background, then I got 4 x 30MB/s (read) and 4 x 21MB/s (write). Write is still slower than read, but it is enough. I can see 1 or 2 tags when reading, around 10 tags when writing. I guess that reading tag exists only for a short time. (or my execution order is not good) Thank you. Daisuke Aoyama ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: [Need Help]isboot (iSCSI boot driver) version 0.2.1
Hi. Daisuke Aoyama wrote: > I uploaded isboot 0.2.1. > Now isboot supports R2T command and non-immediate mode. > I believe that the compatibility of iSCSI target is improved. > Please check it with your iSCSI target and NIC. > Also this version supports task queue, which improves performance. > My test machine got read 74MB/s, write 60MB/s (at bs=1m). > > Notes/Known Issues/Limitations: > FreeBSD can't use transfer length > 64KB. Since 8.0 FreeBSD can use any transfer lengths. 64K is a safety limit for CAM SIMs that do not report maximum transfer size. If your driver supports bigger transactions (and even if not), you should fill maxio field in XPT_PATH_INQ response. > It seems CAM tag is used only 2 of 64 tags. > I don't know a reason, but MAXPHYS limit? How many concurrent requests you are submitting? If you are testing it with sequential read from file system - make sure you have sysctl vfs.read_max set high enough. MAXPHYS does not affects number of concurrent requests, only size of each one. -- Alexander Motin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
[Need Help]isboot (iSCSI boot driver) version 0.2.1
Hi all, I uploaded isboot 0.2.1. Now isboot supports R2T command and non-immediate mode. I believe that the compatibility of iSCSI target is improved. Please check it with your iSCSI target and NIC. Also this version supports task queue, which improves performance. My test machine got read 74MB/s, write 60MB/s (at bs=1m). Notes/Known Issues/Limitations: FreeBSD can't use transfer length > 64KB. It seems CAM tag is used only 2 of 64 tags. I don't know a reason, but MAXPHYS limit? How to build/upgrade: # cd /usr/src # tar xvf /path/to/iscsi-2.2.4.tar.gz # tar xvf /path/to/isboot-0.2.1.tar.gz # make buildkernel # make installkernel or # cd /usr/src/sys/modules/iscsi/isboot # make obj # make depend # make clean # make all # make install After installation, please reboot the server to take effect. Confirm the log shows "iSCSI boot driver version 0.2.1". What's new?: * support CHAP/Mutual CHAP * add task queue * R2T command and non-immediate mode Performance(without digest): # dd if=/dev/da4 of=/dev/null bs=1m count=1k 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 14.374117 secs (74699672 bytes/sec) # dd if=/dev/zero of=/dev/da4 bs=1m count=1k 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 17.823493 secs (60243064 bytes/sec) Download links: http://www.peach.ne.jp/archives/isboot/isboot-0.2.1.tar.gz Download links(for testing purpose only): http://www.peach.ne.jp/archives/isboot/demo/FreeBSD-7.3-RELEASE-amd64-isboot-0.2.1.iso http://www.peach.ne.jp/archives/isboot/demo/FreeBSD-7.3-RELEASE-i386-isboot-0.2.1.iso http://www.peach.ne.jp/archives/isboot/demo/FreeBSD-8.1-RC2-amd64-isboot-0.2.1.iso http://www.peach.ne.jp/archives/isboot/demo/FreeBSD-8.1-RC2-i386-isboot-0.2.1.iso http://www.peach.ne.jp/archives/isboot/demo/unionfs-mkisboot.sh gPXE + isboot + istgt(for reference): http://lists.freebsd.org/pipermail/freebsd-scsi/2010-June/004432.html Previous versions: http://lists.freebsd.org/pipermail/freebsd-scsi/2010-June/004431.html http://lists.freebsd.org/pipermail/freebsd-scsi/2010-June/004425.html I hope this helps you. Enjoy it. Daisuke Aoyama ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
