Re: [ANNOUNCE 0/7] Open-iSCSI/Linux-iSCSI-5 High-Performance Initiator
James Bottomley wrote: On Wed, 2005-06-01 at 13:08 -0700, Alex Aizman wrote: This is open-iscsi/linux-iscsi-5 Initiator. This submission is ready for inclusion into mainline kernel. OK, I tried to put this into scsi-misc. FIB has taken your netlink number, so I changed it to 32 OK. Hopefully that'll remain. __nlm_put() has had an updated prototype, which I can fix (although I'm not sure you're supposed to be using this function...) I can't fix up the compile errors in iscsi_tcp.c: Yes, the 06/01 submission was for 2.6.11. It will not compile. Here's the latest development tarball 0.3rc7: http://www.open-iscsi.org/bits/open-iscsi-0.3rc7-383.tar.gz (This, as well as all the previous code drops can be downloaded from http://www.open-iscsi.org/index.html#download.) Will work on producing a new patchset for 2.6.13. The only pending item now is code style cleanup. Changelog, etc. Monday 08/01 is the latest. CC [M] drivers/scsi/iscsi_tcp.o drivers/scsi/iscsi_tcp.c: In function `iscsi_hdr_extract': drivers/scsi/iscsi_tcp.c:160: warning: implicit declaration of function `iscsi_cnx_error' drivers/scsi/iscsi_tcp.c:161: error: `ISCSI_ERR_PDU_GATHER_FAILED' undeclared (first use in this function) drivers/scsi/iscsi_tcp.c:161: error: (Each undeclared identifier is reported only once drivers/scsi/iscsi_tcp.c:161: error: for each function it appears in.) drivers/scsi/iscsi_tcp.c: In function `iscsi_tcp_state_change': drivers/scsi/iscsi_tcp.c:1005: error: `ISCSI_ERR_CNX_FAILED' undeclared (first use in this function) drivers/scsi/iscsi_tcp.c: In function `iscsi_sendhdr': drivers/scsi/iscsi_tcp.c:1092: error: `ISCSI_ERR_CNX_FAILED' undeclared (first use in this function) drivers/scsi/iscsi_tcp.c: In function `iscsi_sendpage': drivers/scsi/iscsi_tcp.c:1141: error: `ISCSI_ERR_CNX_FAILED' undeclared (first use in this function) drivers/scsi/iscsi_tcp.c: In function `iscsi_data_xmit_more': drivers/scsi/iscsi_tcp.c:1707: error: `STOP_CNX_RECOVER' undeclared (first use in this function) drivers/scsi/iscsi_tcp.c: At top level: [...] Do you have an updated driver that will work in the current tree? Thanks, James - To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[ANNOUNCE 1b/6] Linux-iSCSI High-Performance Initiator
For the start of this thread please refer to:
http://marc.theaimsgroup.com/?l=linux-kernel&m=111327337005048&w=2
and
http://marc.theaimsgroup.com/?l=linux-kernel&m=111328256211837&w=2
Regards,
The combined open-iscsi and linux-iscsi teams
SCSI LLDD, the 2nd part:
- iscsi_if.c (iSCSI open interface over netlink, iSCSI
generic transport module).
Signed-off-by: Alex Aizman <[EMAIL PROTECTED]>
Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]>
diff -Nru --exclude Kconfig --exclude Makefile
linux-2.6.12-rc2.orig/drivers/scsi/iscsi_if.c
linux-2.6.12-rc2.dima/drivers/scsi/iscsi_if.c
--- linux-2.6.12-rc2.orig/drivers/scsi/iscsi_if.c 1969-12-31
16:00:00.0 -0800
+++ linux-2.6.12-rc2.dima/drivers/scsi/iscsi_if.c 2005-04-11
18:13:12.0 -0700
@@ -0,0 +1,818 @@
+/*
+ * iSCSI Initiator Kernel/User Interface
+ *
+ * Copyright (C) 2004 Dmitry Yusupov, Alex Aizman
+ * maintained by [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published
+ * by the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * See the file COPYING included with this distribution for more details.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+#ifdef CONFIG_SCSI_ISCSI_ATTRS
+#include
+#endif
+#include
+#include
+
+MODULE_AUTHOR("Dmitry Yusupov <[EMAIL PROTECTED]>, "
+ "Alex Aizman <[EMAIL PROTECTED]>");
+MODULE_DESCRIPTION("Open-iSCSI Interface");
+MODULE_LICENSE("GPL");
+
+static struct iscsi_transport *transport_table[ISCSI_TRANSPORT_MAX];
+static struct sock *nls;
+static int daemon_pid;
+DECLARE_MUTEX(callsema);
+
+struct mempool_zone {
+ mempool_t *pool;
+ volatile int allocated;
+ int size;
+ int max;
+ int hiwat;
+ struct list_head freequeue;
+ spinlock_t freelock;
+};
+
+static struct mempool_zone z_reply;
+
+#define Z_REPLY0
+#define Z_SIZE_REPLY NLMSG_SPACE(sizeof(struct iscsi_uevent))
+#define Z_MAX_REPLY8
+#define Z_HIWAT_REPLY 6
+
+#define Z_PDU 1
+#define Z_SIZE_PDU NLMSG_SPACE(sizeof(struct iscsi_uevent) + \
+ sizeof(struct iscsi_hdr) + \
+ DEFAULT_MAX_RECV_DATA_SEGMENT_LENGTH)
+#define Z_MAX_PDU 8
+#define Z_HIWAT_PDU6
+
+#define Z_ERROR2
+#define Z_SIZE_ERROR NLMSG_SPACE(sizeof(struct iscsi_uevent))
+#define Z_MAX_ERROR16
+#define Z_HIWAT_ERROR 12
+
+#define zone_init(_zp, _zone) ({ \
+ (_zp)->pool = mempool_create(Z_MAX_##_zone, \
+ mempool_zone_alloc_skb, mempool_zone_free_skb, \
+ (void*)(_zp)); \
+ if ((_zp)->pool) { \
+ (_zp)->max = Z_MAX_##_zone; \
+ (_zp)->size = Z_SIZE_##_zone; \
+ (_zp)->hiwat = Z_HIWAT_##_zone; \
+ INIT_LIST_HEAD(&(_zp)->freequeue); \
+ spin_lock_init(&(_zp)->freelock); \
+ (_zp)->allocated = 0; \
+ } \
+ (_zp)->pool; \
+})
+
+struct iscsi_if_cnx {
+ struct list_head item; /* item in cnxlist */
+ struct list_head snxitem; /* item in snx->connections */
+ iscsi_cnx_t cp_cnx;
+ iscsi_cnx_t dp_cnx;
+ volatile int active;
+ struct Scsi_Host *host; /* originated shost */
+ struct iscsi_transport *transport;
+ struct mempool_zone z_error;
+ struct mempool_zone z_pdu;
+ struct list_head freequeue;
+};
+LIST_HEAD(cnxlist);
+spinlock_t cnxlock;
+
+struct iscsi_if_snx {
+ struct list_head item; /* item in snxlist */
+ struct list_head connections;
+ iscsi_snx_t cp_snx;
+ iscsi_snx_t dp_snx;
+ struct iscsi_transport *transport;
+};
+LIST_HEAD(snxlist);
+spinlock_t snxlock;
+
+#define H_TYPE_CP 0
+#define H_TYPE_DP 1
+#define H_TYPE_HOST2
+static struct iscsi_if_cnx*
+iscsi_if_find_cnx(uint64_t key, int type)
+{
+ unsigned long flags;
+ struct iscsi_if_cnx *cnx;
+
+ spin_lock_irqsave(&cnxlock, flags);
+ list_for_each_entry(cnx, &cnxlist, item) {
+ if ((type == H_TYPE_DP && cnx->dp_cnx == key) ||
+ (type == H_TYPE_CP && cnx->cp_cnx == key) ||
+ (type == H_TYPE_HOST && cnx->host == iscsi_ptr(key))) {
+ spin_unlock_irqrestore(&cnxlock, flags);
+ return
[ANNOUNCE 5/6] Linux-iSCSI High-Performance Initiator
include/linux/netlink.h changes (added NETLINK_ISCSI) Signed-off-by: Alex Aizman <[EMAIL PROTECTED]> Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]> --- linux-2.6.12-rc2.orig/include/linux/netlink.h 2005-03-01 23:38:25.0 -0800 +++ linux-2.6.12-rc2.dima/include/linux/netlink.h 2005-04-11 18:13:12.0 -0700 @@ -14,6 +14,7 @@ #define NETLINK_SELINUX7 /* SELinux event notifications */ #define NETLINK_ARPD 8 #define NETLINK_AUDIT 9 /* auditing */ +#define NETLINK_ISCSI 10 /* iSCSI Open Interface */ #define NETLINK_ROUTE6 11 /* af_inet6 route comm channel */ #define NETLINK_IP6_FW 13 #define NETLINK_DNRTMSG14 /* DECnet routing messages */
[ANNOUNCE 6/6] Linux-iSCSI High-Performance Initiator
Documentation/scsi/iscsi.txt Signed-off-by: Alex Aizman <[EMAIL PROTECTED]> Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]> diff -Nru linux-2.6.12-rc2.orig/Documentation/scsi/iscsi.txt linux-2.6.12-rc2.dima/Documentation/scsi/iscsi.txt --- linux-2.6.12-rc2.orig/Documentation/scsi/iscsi.txt 1969-12-31 16:00:00.0 -0800 +++ linux-2.6.12-rc2.dima/Documentation/scsi/iscsi.txt 2005-04-11 18:13:12.0 -0700 @@ -0,0 +1,292 @@ += + +Linux* Open-iSCSI + += + +April 11, 2005 + +Contents + + +- 1. In This Release +- 2. Introduction +- 3. Installation +- 4. Open-iSCSI daemon +- 5. Open-iSCSI Configuration Utility +- 6. Configuration +- 7. Getting Started +- 8. TBD +- Appendix A. SendTargets snapshot. + + + +1. In This Release +== + +This file describes the Linux* Open-iSCSI Initiator. The software was +tested on AMD Opteron (TM) and Intel Xeon (TM). + +The latest development release is available at: +http://linux-iscsi.sf.net + +For questions, comments, contributions send e-mail to: [EMAIL PROTECTED] and [EMAIL PROTECTED] + +1.1. Features + +- highly optimized and very small-footprint data path; +- multiple outstanding R2Ts; +- persistent configuration database; +- SendTargets discovery; +- CHAP; +- PDU header Digest; +- multiple sessions; +- multi-connection sessions. + +For the most recent list of features please refer to: +http://www.open-iscsi.org/cgi-bin/wiki.pl/Roadmap + + + +2. Introduction +=== + +Open-iSCSI project is a high-performance, transport independent, +multi-platform implementation of RFC3720 iSCSI. + +Open-iSCSI is partitioned into user and kernel parts. + +The kernel portion of Open-iSCSI is a from-scratch code +licensed under GPL. The kernel part implements iSCSI data path +(that is, iSCSI Read and iSCSI Write), and consists of two +loadable modules: iscsi_if.ko and iscsi_tcp.ko. + +User space contains the entire control plane: configuration +manager, iSCSI Discovery, Login and Logout processing, +connection-level error processing, Nop-In and Nop-Out handling, +and (in the future:) Text processing, iSNS, SLP, Radius, etc. + +The user space Open-iSCSI consists of a daemon process called +iscsid, and a management utility iscsiadm. + + +3. Installation +=== + +As of today, the Open-iSCSI Initiator requires a host running the +Linux operating system with kernel version 2.6.11, or later. You need to +enable "Cryptographic API" under "Cryptographic options" in the +kernel config. You also need to enable "CRC32c CRC algorithm" if +you use header or data digests. They are the kernel options, +CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C, respectively. + +Compilation of the kernel Opne-iSCSI modules requires a path +to kernel sources: + + make KSRC= + +or cross-compilation: + + make KSRC= KARCH="ARCH=um" + + +4. Open-iSCSI daemon + + +The daemon implements control path of iSCSI protocol, plus some management +facilities. For example, the daemon could be configured to automatically +re-start discovery at startup, based on the contents of persistent +iSCSI database (see next section). + +For help, run: + + ./iscsid --help + +Usage: iscsid [OPTION] + + -c, --config=[path] Execute in the config file (/etc/iscsid.conf). + -f, --foregroundrun iscsid in the foreground + -d, --debug debuglevel print debugging information + -u, --uid=uid run as uid, default is current user + -g, --gid=gid run as gid, default is current user group + -h, --help display this help and exit + -v, --version display version and exit + + + +5. Open-iSCSI Configuration Utility +=== + +Open-iSCSI persistent configuration is implemented as a DBM database +available on all Linux installations. + +The database contains two tables: + +- Discovery table (discovery.db); +- Node table (node.db). + +The regular place for iSCSI database files: /var/db/iscsi/*.db + +The iscsiadm utility is a command-line tool to manage (update, delete, +insert, query) the persistent database. + +The utility presents set of operations that a user can perform +on iSCSI nodes, sessions, connections, and discovery records. + +Note that some of the iSCSI Node and iSCSI Discovery operations +do not require iSCSI daemon (iscsid) loaded. + +For help, run: + + ./iscsiadm --help + +Usage: iscsiadm [OPTION] + + -m, --mode specify operational mode op = + -m discovery --type=[type] --portal=[ip:port] --login + perform [type] discovery for target portal with +
[ANNOUNCE 4/6] Linux-iSCSI High-Performance Initiator
drivers/scsi/Makefile changes (added iscsi_if and iscsi_tcp). Signed-off-by: Alex Aizman <[EMAIL PROTECTED]> Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]> --- linux-2.6.12-rc2.orig/drivers/scsi/Makefile 2005-03-01 23:38:19.0 -0800 +++ linux-2.6.12-rc2.dima/drivers/scsi/Makefile 2005-04-11 18:13:12.0 -0700 @@ -30,6 +30,8 @@ obj-$(CONFIG_SCSI_FC_ATTRS)+= scsi_transport_fc.o obj-$(CONFIG_SCSI_ISCSI_ATTRS) += scsi_transport_iscsi.o +obj-$(CONFIG_ISCSI_IF) += iscsi_if.o +obj-$(CONFIG_ISCSI_TCP)+= iscsi_tcp.o obj-$(CONFIG_SCSI_AMIGA7XX)+= amiga7xx.o 53c7xx.o obj-$(CONFIG_A3000_SCSI) += a3000.o wd33c93.o obj-$(CONFIG_A2091_SCSI) += a2091.o wd33c93.o
[ANNOUNCE 3/6] Linux-iSCSI High-Performance Initiator
drivers/scsi/Kconfig changes. Signed-off-by: Alex Aizman <[EMAIL PROTECTED]> Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]> diff -Nru --exclude 'iscsi*' --exclude Makefile linux-2.6.12-rc2.orig/drivers/scsi/Kconfig linux-2.6.12-rc2.dima/drivers/scsi/Kconfig --- linux-2.6.12-rc2.orig/drivers/scsi/Kconfig 2005-04-11 17:13:39.0 -0700 +++ linux-2.6.12-rc2.dima/drivers/scsi/Kconfig 2005-04-11 18:13:12.0 -0700 @@ -185,6 +185,45 @@ there should be no noticeable performance impact as long as you have logging turned off. +config ISCSI_IF + tristate "iSCSI Open Transport Interface" + depends on SCSI && INET + ---help--- + This driver manages multiple iSCSI transports. This module is required + for normal iscsid operation. + + To compile this driver as a module, choose M here: the + module will be called iscsi_if. + + See more detailed information here: + + http://linux-iscsi.sf.net + +config ISCSI_TCP + tristate "iSCSI Initiator over TCP/IP" + depends on ISCSI_IF + default y + select CRYPTO + select CRYPTO_MD5 + select CRYPTO_CRC32C + select SCSI_ISCSI_ATTRS + ---help--- + The iSCSI Driver provides a host with the ability to access storage + through an IP network. The driver uses the iSCSI protocol to transport + SCSI requests and responses over a TCP/IP network between the host + (the "initiator") and "targets". Architecturally, the iSCSI driver + combines with the host's TCP/IP stack, network drivers, and Network + Interface Card (NIC) to provide the same functions as a SCSI or a + Fibre Channel (FC) adapter driver with a Host Bus Adapter (HBA). + + To compile this driver as a module, choose M here: the + module will be called iscsi_tcp. + + The userspace component needed to initialize the driver, documentation, + and sample configuration files can be found here: + + http://linux-iscsi.sf.net + menu "SCSI Transport Attributes" depends on SCSI
[ANNOUNCE 2/6] Linux-iSCSI High-Performance Initiator
Common header files:
- iscsi_ifev.h (user/kernel events).
- iscsi_if.h (user/kernel #defines);
- iscsi_iftrans.h (iscsi transport interface);
- iscsi_proto.h (RFC3720 #defines and types).
Signed-off-by: Alex Aizman <[EMAIL PROTECTED]>
Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]>
diff -Nru linux-2.6.12-rc2.orig/include/scsi/iscsi_ifev.h
linux-2.6.12-rc2.dima/include/scsi/iscsi_ifev.h
--- linux-2.6.12-rc2.orig/include/scsi/iscsi_ifev.h 1969-12-31
16:00:00.0 -0800
+++ linux-2.6.12-rc2.dima/include/scsi/iscsi_ifev.h 2005-04-11
18:13:12.0 -0700
@@ -0,0 +1,118 @@
+/*
+ * iSCSI Kernel/User Interface Events
+ *
+ * Copyright (C) 2005 Dmitry Yusupov, Alex Aizman
+ * maintained by [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published
+ * by the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * See the file COPYING included with this distribution for more details.
+ */
+
+#ifndef ISCSI_IFEV_H
+#define ISCSI_IFEV_H
+
+enum iscsi_uevent_e {
+ ISCSI_UEVENT_UNKNOWN= 0,
+
+ /* down events */
+ ISCSI_UEVENT_CREATE_SESSION = UEVENT_BASE + 1,
+ ISCSI_UEVENT_DESTROY_SESSION= UEVENT_BASE + 2,
+ ISCSI_UEVENT_CREATE_CNX = UEVENT_BASE + 3,
+ ISCSI_UEVENT_DESTROY_CNX= UEVENT_BASE + 4,
+ ISCSI_UEVENT_BIND_CNX = UEVENT_BASE + 5,
+ ISCSI_UEVENT_SET_PARAM = UEVENT_BASE + 6,
+ ISCSI_UEVENT_START_CNX = UEVENT_BASE + 7,
+ ISCSI_UEVENT_STOP_CNX = UEVENT_BASE + 8,
+ ISCSI_UEVENT_SEND_PDU = UEVENT_BASE + 9,
+ ISCSI_UEVENT_TRANS_LIST = UEVENT_BASE + 10,
+
+ /* up events */
+ ISCSI_KEVENT_RECV_PDU = KEVENT_BASE + 1,
+ ISCSI_KEVENT_CNX_ERROR = KEVENT_BASE + 2,
+ ISCSI_KEVENT_IF_ERROR = KEVENT_BASE + 3,
+};
+
+struct iscsi_uevent {
+ uint32_t type; /* k/u events type */
+ uint32_t iferror; /* carries interface or resource errors */
+ uint64_t transport_handle;
+
+ union {
+ /* messages u -> k */
+ struct msg_create_session {
+ uint64_tsession_handle;
+ uint32_tinitial_cmdsn;
+ } c_session;
+ struct msg_destroy_session {
+ uint64_tsession_handle;
+ uint32_tsid;
+ } d_session;
+ struct msg_create_cnx {
+ uint64_tsession_handle;
+ uint64_tcnx_handle;
+ uint32_tcid;
+ uint32_tsid;
+ } c_cnx;
+ struct msg_bind_cnx {
+ uint64_tsession_handle;
+ uint64_tcnx_handle;
+ uint32_ttransport_fd;
+ uint32_tis_leading;
+ } b_cnx;
+ struct msg_destroy_cnx {
+ uint64_tcnx_handle;
+ uint32_tcid;
+ } d_cnx;
+ struct msg_send_pdu {
+ uint32_thdr_size;
+ uint32_tdata_size;
+ uint64_tcnx_handle;
+ } send_pdu;
+ struct msg_set_param {
+ uint64_tcnx_handle;
+ uint32_tparam; /* enum iscsi_param */
+ uint32_tvalue;
+ } set_param;
+ struct msg_start_cnx {
+ uint64_tcnx_handle;
+ } start_cnx;
+ struct msg_stop_cnx {
+ uint64_tcnx_handle;
+ uint32_tflag;
+ } stop_cnx;
+ } u;
+ union {
+ /* messages k -> u */
+ uint64_thandle;
+ int retcode;
+ struct msg_create_session_ret {
+ uint64_thandle;
+ uint32_tsid;
+ } c_session_ret;
+ struct msg_recv_req {
+ uint64_trecv_handle;
+ uint64_tcnx_handle;
+ } recv_req;
+ struct msg_cnx_error {
+
[ANNOUNCE 0/6] Linux-iSCSI High-Performance Initiator
preallocation * added protection from SIGTERM * added protection from OOM-killer using oom_adj == -17 * allocate cache aligned iscsi_taskcache * tx latency: transmitting from queuecommand() if trylock succeeds * reduced context switching in queuecommand * max_xmit_dlength properly negotiated * enable Nagle * netlink reply cleanup * kernel/user IPC cleanup: added and implemented iscsi_ipc callback structure * receive pool for control PDU's per-connection added * using predefined macro for max_sg SG_ALL * kernel IPC transport header file added * picking unique OUI ISID for the same target based on session's SID Regards, Alex Aizman & Dmitry Yusupov - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE 0/6] Open-iSCSI High-Performance Initiator for Linux
Lars Marowsky-Bree wrote: On 2005-03-08T22:25:29, Alex Aizman <[EMAIL PROTECTED]> wrote: There's (or at least was up until today) an ongoing discussion on our mailing list at http://groups-beta.google.com/group/open-iscsi. The short and long of it: the problem can be solved, and it will. Couple simple things we already do: mlockall() to keep the daemon un-swapped, and also looking into potential dependency created by syslog (there's one for 2.4 kernel, not sure if this is an issue for 2.6). BTW, to get around the very same issues, heartbeat does much the same: lock itself into memory, reserve a couple of pages more to spare on stack & heap, run at soft-realtime priority. Heartbeat is good for reliability, etc. WRT "getting paged-out" - non-deterministic (things depend on time), right? syslog(), however, sucks. It does. We went down the path of using our non-blocking IPC library to have all our various components log to ha_logd, which then logs to syslog() or writes to disk or wherever. Found ha_logd under http://linux-ha.org. The latter is extemely interesting in the longer term. In the short term, there's quite a bit of information on this site, need time. That works well in our current development series, and if you want to share code, you can either rip it off (Open Source, we love ya ;) or we can spin off these parts into a sub-package for you to depend on... If it's not a big deal :-) let's do the "sub-package" option. The sfnet is a learning experience; it is by no means a proof that it cannot be done. I'd also argue that it MUST be done, because the current way of "Oh, it's somehow related to block stuff, must be in kernel" leads down to hell. We better figure out good ways around it ;-) Yes, it MUST be done. Sincerely, Lars Marowsky-Brée <[EMAIL PROTECTED]> Alex - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE 0/6] Open-iSCSI High-Performance Initiator for Linux
Matt Mackall wrote: On Tue, Mar 08, 2005 at 09:51:39PM -0800, Alex Aizman wrote: Matt Mackall wrote: How big is the userspace client? Hmm.. x86 executable? source? Anyway, there's about 12,000 lines of user space code, and growing. In the kernel we have approx. 3,300 lines. - 450MB/sec Read on a single connection (2-way 2.4Ghz Opteron, 64KB block size); With what network hardware and drives, please? Neterion's 10GbE adapters. RAM disk on the target side. Ahh. Snipped my question about userspace deadlocks - that was the important one. It is in fact why the sfnet one is written as it is - it originally had a userspace component and turned out to be easy to deadlock under load because of it. There's (or at least was up until today) an ongoing discussion on our mailing list at http://groups-beta.google.com/group/open-iscsi. The short and long of it: the problem can be solved, and it will. Couple simple things we already do: mlockall() to keep the daemon un-swapped, and also looking into potential dependency created by syslog (there's one for 2.4 kernel, not sure if this is an issue for 2.6). The sfnet is a learning experience; it is by no means a proof that it cannot be done. Alex - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE 0/6] Open-iSCSI High-Performance Initiator for Linux
Matt Mackall wrote: How big is the userspace client? Hmm.. x86 executable? source? Anyway, there's about 12,000 lines of user space code, and growing. In the kernel we have approx. 3,300 lines. - 450MB/sec Read on a single connection (2-way 2.4Ghz Opteron, 64KB block size); With what network hardware and drives, please? Neterion's 10GbE adapters. RAM disk on the target side. Alex - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[ANNOUNCE 5/6] Open-iSCSI High-Performance Initiator for Linux
include/linux/netlink.h changes (added new protocol NETLINK_ISCSI) Signed-off-by: Alex Aizman <[EMAIL PROTECTED]> Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]> --- linux-2.6.11.orig/include/linux/netlink.h 2005-03-01 23:38:25.0 -0800 +++ linux-2.6.11.dima/include/linux/netlink.h 2005-03-04 17:50:11.143413101 -0800 @@ -14,6 +14,7 @@ #define NETLINK_SELINUX7 /* SELinux event notifications */ #define NETLINK_ARPD 8 #define NETLINK_AUDIT 9 /* auditing */ +#define NETLINK_ISCSI 10 /* iSCSI Open Interface */ #define NETLINK_ROUTE6 11 /* af_inet6 route comm channel */ #define NETLINK_IP6_FW 13 #define NETLINK_DNRTMSG14 /* DECnet routing messages */
[ANNOUNCE 4/6] Open-iSCSI High-Performance Initiator for Linux
drivers/scsi/Makefile changes. Signed-off-by: Alex Aizman <[EMAIL PROTECTED]> Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]> --- linux-2.6.11.orig/drivers/scsi/Makefile 2005-03-01 23:38:19.0 -0800 +++ linux-2.6.11.dima/drivers/scsi/Makefile 2005-03-04 17:50:11.142413217 -0800 @@ -30,6 +30,8 @@ obj-$(CONFIG_SCSI_FC_ATTRS)+= scsi_transport_fc.o obj-$(CONFIG_SCSI_ISCSI_ATTRS) += scsi_transport_iscsi.o +obj-$(CONFIG_ISCSI_IF) += iscsi_if.o +obj-$(CONFIG_ISCSI_TCP)+= iscsi_tcp.o obj-$(CONFIG_SCSI_AMIGA7XX)+= amiga7xx.o 53c7xx.o obj-$(CONFIG_A3000_SCSI) += a3000.o wd33c93.o obj-$(CONFIG_A2091_SCSI) += a2091.o wd33c93.o
[ANNOUNCE 6/6] Open-iSCSI High-Performance Initiator for Linux
Documentation/scsi/iscsi.txt Signed-off-by: Alex Aizman <[EMAIL PROTECTED]> Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]> diff -Nru linux-2.6.11.orig/Documentation/scsi/iscsi.txt linux-2.6.11.dima/Documentation/scsi/iscsi.txt --- linux-2.6.11.orig/Documentation/scsi/iscsi.txt 1969-12-31 16:00:00.0 -0800 +++ linux-2.6.11.dima/Documentation/scsi/iscsi.txt 2005-03-04 17:50:11.145412869 -0800 @@ -0,0 +1,279 @@ += + +Linux* Open-iSCSI + += + +March 04, 2005 + +Contents + + +- 1. In This Release +- 2. Introduction +- 3. Installation +- 4. Open-iSCSI daemon +- 5. Open-iSCSI Configuration Utility +- 6. Configuration +- 7. Getting Started +- 8. TBD +- Appendix A. SendTargets snapshot. + + + +1. In This Release +== + +This file describes the Linux* Open-iSCSI Initiator. + +The latest development release is available at: +http://www.open-iscsi.org + +For questions, comments, contributions send e-mail to: [EMAIL PROTECTED] + +1.1. Features + +- highly optimized and very small-footprint data path; +- multiple outstanding R2Ts; +- persistent configuration database; +- SendTargets discovery; +- CHAP; +- PDU header Digest; +- multiple sessions; +- multi-connection sessions. + +For the most recent list of features please refer to: +http://www.open-iscsi.org/cgi-bin/wiki.pl/Roadmap + + + +2. Introduction +=== + +Open-iSCSI project is a high-performance, transport independent, +multi-platform implementation of RFC3720 iSCSI. + +Open-iSCSI is partitioned into user and kernel parts. + +The kernel portion of Open-iSCSI is a from-scratch code +licensed under GPL. The kernel part implements iSCSI data path +(that is, iSCSI Read and iSCSI Write), and consists of two +loadable modules: iscsi_if.ko and iscsi_tcp.ko. + +User space contains the entire control plane: configuration +manager, iSCSI Discovery, Login and Logout processing, +connection-level error processing, Nop-In and Nop-Out handling, +and (in the future:) Text processing, iSNS, SLP, Radius, etc. + +The user space Open-iSCSI consists of a daemon process called +iscsid, and a management utility iscsiadm. + + +3. Installation +=== + +You need to enable "Cryptographic API" under "Cryptographic options" +in the kernel config. You also need to enable "CRC32c CRC algorithm" if +you use header or data digests. They are the kernel options, +CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C, respectively. + + +4. Open-iSCSI daemon + + +The daemon implements control path of iSCSI protocol, plus some management +facilities. For example, the daemon could be configured to automatically +re-start discovery at startup, based on the contents of persistent +iSCSI database (see next section). + +For help, run: + + ./iscsid --help + +Usage: iscsid [OPTION] + + -c, --config=[path] Execute in the config file (/etc/iscsid.conf). + -f, --foregroundrun iscsid in the foreground + -d, --debug debuglevel print debugging information + -u, --uid=uid run as uid, default is current user + -g, --gid=gid run as gid, default is current user group + -h, --help display this help and exit + -v, --version display version and exit + + + +5. Open-iSCSI Configuration Utility +=== + +Open-iSCSI persistent configuration is implemented as a DBM database +available on all Linux installations. + +The database contains two tables: + +- Discovery table (discovery.db); +- Node table (node.db). + +The regular place for iSCSI database files: /var/db/iscsi/*.db + +The iscsiadm utility is a command-line tool to manage (update, delete, +insert, query) the persistent database. + +The utility presents set of operations that a user can perform +on iSCSI nodes, sessions, connections, and discovery records. + +Note that some of the iSCSI Node and iSCSI Discovery operations +do not require iSCSI daemon (iscsid) loaded. + +For help, run: + + ./iscsiadm --help + +Usage: iscsiadm [OPTION] + + -m, --mode specify operational mode op = + -m discovery --type=[type] --portal=[ip:port] --login + perform [type] discovery for target portal with + ip-address [ip] and port [port]. Initiate Login for + each discovered target if --login is specified + -m discoverydisplay all discovery records from internal + persistent discovery database + -m discovery --record=[id] --login + perform discovery based on record [id] in database + -m discovery --record=[id] --op=[op] [--name=[name] --value=[
[ANNOUNCE 3/6] Open-iSCSI High-Performance Initiator for Linux
drivers/scsi/Kconfig changes. Signed-off-by: Alex Aizman <[EMAIL PROTECTED]> Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]> diff -Nru --exclude 'iscsi*' --exclude Makefile linux-2.6.11.orig/drivers/scsi/Kconfig linux-2.6.11.dima/drivers/scsi/Kconfig --- linux-2.6.11.orig/drivers/scsi/Kconfig 2005-03-01 23:38:25.0 -0800 +++ linux-2.6.11.dima/drivers/scsi/Kconfig 2005-03-04 17:50:11.14141 -0800 @@ -185,6 +185,44 @@ there should be no noticeable performance impact as long as you have logging turned off. +config ISCSI_IF + tristate "iSCSI Open Transport Interface" + depends on SCSI && INET + ---help--- + To compile this driver as a module, choose M here: the + module will be called iscsi_if. + + This driver manages multiple iSCSI transports. This module is required + for normal iscsid operation. + + See more detailed information here: + + http://www.open-iscsi.org + +config ISCSI_TCP + tristate "iSCSI Initiator over TCP/IP" + depends on ISCSI_IF + default y + select CRYPTO + select CRYPTO_MD5 + select CRYPTO_CRC32C + ---help--- + To compile this driver as a module, choose M here: the + module will be called iscsi_tcp. + + The iSCSI Driver provides a host with the ability to access storage + through an IP network. The driver uses the iSCSI protocol to transport + SCSI requests and responses over a TCP/IP network between the host + (the "initiator") and "targets". Architecturally, the iSCSI driver + combines with the host's TCP/IP stack, network drivers, and Network + Interface Card (NIC) to provide the same functions as a SCSI or a + Fibre Channel (FC) adapter driver with a Host Bus Adapter (HBA). + + The userspace component needed to initialize the driver, documentation, + and sample configuration files can be found here: + + http://www.open-iscsi.org + menu "SCSI Transport Attributes" depends on SCSI
[ANNOUNCE 2/6] Open-iSCSI High-Performance Initiator for Linux
Common header files:
- iscsi_ifev.h (user/kernel events).
- iscsi_if.h (iSCSI open interface over netlink);
- iscsi_proto.h (RFC3720 #defines and types);
Signed-off-by: Alex Aizman <[EMAIL PROTECTED]>
Signed-off-by: Dmitry Yusupov <[EMAIL PROTECTED]>
diff -Nru linux-2.6.11.orig/include/scsi/iscsi_ifev.h
linux-2.6.11.dima/include/scsi/iscsi_ifev.h
--- linux-2.6.11.orig/include/scsi/iscsi_ifev.h 1969-12-31 16:00:00.0
-0800
+++ linux-2.6.11.dima/include/scsi/iscsi_ifev.h 2005-03-04 17:50:11.138413681
-0800
@@ -0,0 +1,105 @@
+/*
+ * iSCSI Kernel/User Interface Events
+ *
+ * Copyright (C) 2005 Dmitry Yusupov, Alex Aizman
+ * maintained by [EMAIL PROTECTED]
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published
+ * by the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * See the file COPYING included with this distribution for more details.
+ */
+
+#ifndef ISCSI_IFEV_H
+#define ISCSI_IFEV_H
+
+typedef enum iscsi_uevent_e {
+ ISCSI_UEVENT_UNKNOWN= 0,
+
+ /* down events */
+ ISCSI_UEVENT_CREATE_SESSION = UEVENT_BASE + 1,
+ ISCSI_UEVENT_DESTROY_SESSION= UEVENT_BASE + 2,
+ ISCSI_UEVENT_CREATE_CNX = UEVENT_BASE + 3,
+ ISCSI_UEVENT_DESTROY_CNX= UEVENT_BASE + 4,
+ ISCSI_UEVENT_BIND_CNX = UEVENT_BASE + 5,
+ ISCSI_UEVENT_SET_PARAM = UEVENT_BASE + 6,
+ ISCSI_UEVENT_START_CNX = UEVENT_BASE + 7,
+ ISCSI_UEVENT_STOP_CNX = UEVENT_BASE + 8,
+ ISCSI_UEVENT_SEND_PDU = UEVENT_BASE + 9,
+
+ /* up events */
+ ISCSI_KEVENT_RECV_PDU = KEVENT_BASE + 1,
+ ISCSI_KEVENT_CNX_ERROR = KEVENT_BASE + 2,
+} iscsi_uevent_e;
+
+struct iscsi_uevent {
+ uint32_t type; /* k/u events type */
+ uint32_t transport_id;
+
+ union {
+ /* messages u -> k */
+ struct msg_create_session {
+ uint64_tsession_handle;
+ uint32_tinitial_cmdsn;
+ } c_session;
+ struct msg_destroy_session {
+ uint64_tsession_handle;
+ } d_session;
+ struct msg_create_cnx {
+ uint64_tsession_handle;
+ uint64_tcnx_handle;
+ uint32_tcid;
+ } c_cnx;
+ struct msg_bind_cnx {
+ uint64_tsession_handle;
+ uint64_tcnx_handle;
+ uint32_ttransport_fd;
+ uint32_tis_leading;
+ } b_cnx;
+ struct msg_destroy_cnx {
+ uint64_tcnx_handle;
+ } d_cnx;
+ struct msg_send_pdu {
+ uint32_thdr_size;
+ uint32_tdata_size;
+ uint64_tcnx_handle;
+ } send_pdu;
+ struct msg_set_param {
+ uint64_tcnx_handle;
+ uint32_tparam; /* iscsi_param_e */
+ uint32_tvalue;
+ } set_param;
+ struct msg_start_cnx {
+ uint64_tcnx_handle;
+ } start_cnx;
+ struct msg_stop_cnx {
+ uint64_tcnx_handle;
+ } stop_cnx;
+ } u;
+ union {
+ /* messages k -> u */
+ uint64_thandle;
+ int retcode;
+ struct msg_create_session_ret {
+ uint64_thandle;
+ uint32_tsid;
+ } c_session_ret;
+ struct msg_recv_req {
+ uint64_trecv_handle;
+ uint64_tcnx_handle;
+ } recv_req;
+ struct msg_cnx_error {
+ uint64_tcnx_handle;
+ uint32_terror; /* iscsi_err_e */
+ } cnxerror;
+ } r;
+};
+
+#endif /* ISCSI_IFEV_H */
diff -Nru linux-2.6.11.orig/include/scsi/iscsi_if.h
linux-2.6.11.dima/include/scsi/iscsi_if.h
--- linux-2.6.11.orig/include/scsi/iscsi_if.h 1969-12-31 16:00:00.0
-0800
+++ linux-2.6.11.dima/include/scsi/iscsi_if.h 2005-03-04 17:50:11.137413797
-0800
@@ -0,0 +1,140 @@
+/*
+ *
[ANNOUNCE 0/6] Open-iSCSI High-Performance Initiator for Linux
This is to announce Open-iSCSI project: High-Performance iSCSI Initiator for Linux. MOTIVATION == Our initial motivations for the project were: (1) implement the right user/kernel split, and (2) design iSCSI data path for performance. Recently we added (3): get accepted into the mainline kernel. As far as user/kernel, the existing iSCSI initiators bloat the kernel with ever-growing control plane code, including but not limited to: iSCSI discovery, Login (Authentication and Operational), session and connection management, connection-level error processing, iSCSI Text, Nop-Out/In, Async Message, iSNS, SLP, Radius... Open-iSCSI puts the entire control plane in the user space. This control plane talks to the data plane via well defined interface over the netlink transport. (Side note: prior to closing on the netlink we considered: sysfs, ioctl, and syscall. Because the entire control plane logic resides in the user space, we needed a real bi-directional transport that could support asynchronous API to transfer iSCSI control PDUs: Login, Logout, Nop-in, Nop-Out, Text, Async Message. Performance. This is the major goal and motivation for this project. As it happens, iSCSI has to compete with Fibre Channel, which is a more entrenched technology in the storage space. In addition, the "soft" iSCSI implementation have to show good results in presence of specialized hardware offloads. Our today's performance numbers are: - 450MB/sec Read on a single connection (2-way 2.4Ghz Opteron, 64KB block size); - 320MB/sec Write on a single connection (2-way 2.4Ghz Opteron, 64KB block size); - 50,000 Read IOPS on a single connection (2-way 2.4Ghz Opteron, 4KB block size). Prior to starting from-scratch the data path code we did evaluate the sfnet Initiator. And eventually decided against patching it. Instead, we reused its Discovery, Login, etc. control plane code. Technically, it was the shortest way to achieve the (1) and (2) goals stated above. We believe that it remains the easiest and the most practical thing on the larger scale of: iSCSI for Linux. STATUS == There's a 100% working code that interoperates with all (count=5) iSCSI targets we could get our hands on. The software was tested on AMD Opteron (TM) and Intel Xeon (TM). Code is available online via either Subversion source control database or the latest development release (i.e., the tarball containing Open-iSCSI sources, including user space, that will build and run on kernels starting 2.6.10). http://www.open-iscsi.org Features: - highly optimized and small-footprint data path; - multiple outstanding R2Ts; - thread-less receive; - sendpage() based transmit; - zero-copy header processing on receive; - no data path memory allocations at runtime; - persistent configuration database; - SendTargets discovery; - CHAP; - DataSequenceInOrder=No; - PDU header Digest; - multiple sessions; - MC/S (note: disabled in the patch); - SCSI-level recovery via Abort Task and session re-open. TODO The near term plan is: test, test, and test. We need to stabilize the existing code, after 5 months of development this seems to be the right thing to do. Other short-term plans include: a) process community feedback, implement comments and apply patches; b) cleanup user side of the iSCSI open interface; use API calls (instead of directly constructing events); c) eliminate runtime control path memory allocations (for Nop-In, Nop-Out, etc.); d) implement Write path optimizations (delayed because of the self-imposed submission deadline); e) oProfile the data path, use the reports for further optimization; f) complete the readme. Comments, code reviews, patches - are greatly appreciated! THANKS == Special thanks to our first reviewers: Christoph Hellwig and Mike Christie. Special thanks to Ming Zhang for help in testing and for insightful questions. Regards, Alex Aizman & Dmitry Yusupov = The following 6 patches alltogether represent the Open-iSCSI Initiator: Patch 1: SCSI LLDD consists of 3 files: - iscsi_if.c (iSCSI open interface over netlink); - iscsi_tcp.[ch] (iSCSI transport over TCP/IP). Patch 2: Common header files: - iscsi_if.h (iSCSI open interface over netlink); - iscsi_proto.h (RFC3720 #defines and types); - iscsi_ifev.h (user/kernel events). Patch 3: drivers/scsi/Kconfig changes. Patch 4: drivers/scsi/Makefile changes. Patch 5: include/linux/netlink.h changes (added new protocol NETLINK_ISCSI) Patch 6: Documentation/scsi/iscsi.txt - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
