Re: [ofa-general] [RFC] [PATCH 0/3] 2.6.22 or 23 ib: add path record cache
Hello sean I have certain queries about local_sa_cache. Once SM is up on a node/switch whole network is up. Now is if some client is trying to establish a connection with other node, client is expected to resolve the path using sa API, I want to know how exactly it happens in the stack? second query is below. On 4/20/07, Sean Hefty [EMAIL PROTECTED] wrote: The following set of patches adds a local SA path record cache to the IB stack (currently based on 2.6.20-rc4). The cache is derived from the OFED 1.2 local SA cache patches, with changes based on the last round of feedback and current Path Forward feature requests: * InformInfo/Notice support added to ib_sa Clients may now register to receive SA related events. The local_sa uses this to receive notification of GID up/down events in order to keep the cache up to date. Is it possible to program local_sa_cache with some dummy path records which resides in cache for long time? * Removal of time based cache updates Cache updates are now driven by local and SA events. Most module parameters have been eliminated, and remaining options are exposed through a file system interface for dynamic control, including the ability to force a cache refresh. Using a local SA cache we were able to establish all-to-all connections between 1024 processes (about 1 million connections) in about 3 seconds. Without the cache, connection time took about a minute, and required a substantial amount of tuning of timeout values to achieve this. I've only updated the rdma_cm to use the cache, but similar changes could be made to SRP and ipoib (which implements its own path record cache). I would like to get feedback on both the notice and local_sa patches for inclusion in 2.6.22 or 2.6.23 (if 2.6.22 is not possible). Signed-off-by: Sean Hefty [EMAIL PROTECTED] ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [ofa-general] [PATCH 1/2][RFC] IB/uverbs: Export ib_umem_get()/ib_umem_release() to modules
[Here's an easier to read git format patch, which handles the renaming
of uverbs_mem.c - umem.c and just shows the changes, rather than
deleting one file and adding another one]
Export ib_umem_get()/ib_umem_release() and put low-level drivers in
control of when to call ib_umem_get() to pin and DMA map userspace,
rather than always calling it in ib_uverbs_reg_mr() before calling the
low-level driver's reg_user_mr method.
Also move these functions to be in the ib_core module instead of
ib_uverbs, so that driver modules using them do not depend on
ib_uverbs.
This has a number of advantages:
- It is better design from the standpoint of making generic code a
library that can be used or overridden by device-specific code as
the details of specific devices dictate.
- Drivers that do not need to pin userspace memory regions do not
need to take the performance hit of calling ib_mem_get(). For
example, although I have not tried to implement it in this patch,
the ipath driver should be able to avoid pinning memory and just
use copy_{to,from}_user() to access userspace memory regions.
- Buffers that need special mapping treatment can be identified by
the low-level driver. For example, it may be possible to solve
some Altix-specific memory ordering issues with mthca CQs in
userspace by mapping CQ buffers with extra flags.
- Drivers that need to pin and DMA map userspace memory for things
other than memory regions can use ib_umem_get() directly, instead
of hacks using extra parameters to their reg_phys_mr method. For
example, the mlx4 driver that is pending being merged needs to pin
and DMA map QP and CQ buffers, but it does not need to create a
memory key for these buffers. So the cleanest solution is for mlx4
to call ib_umem_get() in the create_qp and create_cq methods.
Signed-off-by: Roland Dreier [EMAIL PROTECTED]
---
drivers/infiniband/Kconfig |5 +
drivers/infiniband/core/Makefile |4 +-
drivers/infiniband/core/device.c |2 +
drivers/infiniband/core/{uverbs_mem.c = umem.c} | 136 +++---
drivers/infiniband/core/uverbs.h |6 +-
drivers/infiniband/core/uverbs_cmd.c | 60 +++---
drivers/infiniband/core/uverbs_main.c| 11 +--
drivers/infiniband/hw/amso1100/c2_provider.c | 42 +---
drivers/infiniband/hw/amso1100/c2_provider.h |1 +
drivers/infiniband/hw/cxgb3/iwch_provider.c | 28 +++--
drivers/infiniband/hw/cxgb3/iwch_provider.h |1 +
drivers/infiniband/hw/ehca/ehca_classes.h|1 +
drivers/infiniband/hw/ehca/ehca_iverbs.h |3 +-
drivers/infiniband/hw/ehca/ehca_mrmw.c | 69 ++-
drivers/infiniband/hw/ipath/ipath_mr.c | 38 +--
drivers/infiniband/hw/ipath/ipath_verbs.h|5 +-
drivers/infiniband/hw/mthca/mthca_provider.c | 38 +--
drivers/infiniband/hw/mthca/mthca_provider.h |1 +
include/rdma/ib_umem.h | 78
include/rdma/ib_verbs.h | 28 +
20 files changed, 355 insertions(+), 202 deletions(-)
rename drivers/infiniband/core/{uverbs_mem.c = umem.c} (63%)
create mode 100644 include/rdma/ib_umem.h
diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index 66b36de..82afba5 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -29,6 +29,11 @@ config INFINIBAND_USER_ACCESS
libibverbs, libibcm and a hardware driver library from
http://www.openib.org.
+config INFINIBAND_USER_MEM
+ bool
+ depends on INFINIBAND_USER_ACCESS != n
+ default y
+
config INFINIBAND_ADDR_TRANS
bool
depends on INFINIBAND INET
diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile
index 189e5d4..cb1ab3e 100644
--- a/drivers/infiniband/core/Makefile
+++ b/drivers/infiniband/core/Makefile
@@ -9,6 +9,7 @@ obj-$(CONFIG_INFINIBAND_USER_ACCESS) += ib_uverbs.o ib_ucm.o \
ib_core-y := packer.o ud_header.o verbs.o sysfs.o \
device.o fmr_pool.o cache.o
+ib_core-$(CONFIG_INFINIBAND_USER_MEM) += umem.o
ib_mad-y :=mad.o smi.o agent.o mad_rmpp.o
@@ -28,5 +29,4 @@ ib_umad-y := user_mad.o
ib_ucm-y :=ucm.o
-ib_uverbs-y := uverbs_main.o uverbs_cmd.o uverbs_mem.o \
- uverbs_marshall.o
+ib_uverbs-y := uverbs_main.o uverbs_cmd.o uverbs_marshall.o
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 7fabb42..592c90a 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -613,6 +613,8 @@ static void __exit ib_core_cleanup(void)
{
ib_cache_cleanup();
ib_sysfs_cleanup();
+ /* Make sure that any pending umem
[ofa-general] ipath irq bug
Hi, with rather many kernel debug options enabled I get this trace/message: [ 2651.218740] BUG: at kernel/lockdep.c:1860 trace_hardirqs_on() [ 2651.224696] [ 2651.224697] Call Trace: [ 2651.228784] IRQ [8022a3e1] release_console_sem+0x47/0x1f6 [ 2651.235709] [80242a41] trace_hardirqs_on+0xfd/0x154 [ 2651.241648] [80478444] _spin_unlock_irq+0x28/0x2d [ 2651.247482] [880a0769] :ib_ipath:ipath_rc_rcv+0xf5b/0xf8e [ 2651.254058] [8809d1ff] :ib_ipath:ipath_lookup_qpn+0x4f/0x5a [ 2651.260791] [880a5f40] :ib_ipath:ipath_qp_rcv+0x45/0x4e [ 2651.267203] [880a60b3] :ib_ipath:ipath_ib_rcv+0x16a/0x1a8 [ 2651.273784] [88090a3e] :ib_ipath:ipath_kreceive+0x42f/0x6b9 [ 2651.280545] [80243bdd] __lock_acquire+0xc08/0xc60 [ 2651.286397] [880a6b02] :ib_ipath:ipath_ib_piobufavail+0x72/0x79 [ 2651.291177] LustreError: 3433:0:(events.c:129:client_bulk_callback()) event type 0, status -5, desc 8100ca2ea000 [ 2651.294334] LustreError: 3433:0:(events.c:129:client_bulk_callback()) event type 0, status -5, desc 81007034 [ 2651.315171] [80478394] _spin_unlock_irqrestore+0x38/0x47 [ 2651.321572] [8809932c] :ib_ipath:ipath_intr+0x26a/0x17b6 [ 2651.328082] [80243bdd] __lock_acquire+0xc08/0xc60 [ 2651.333892] [80222167] try_to_wake_up+0x413/0x425 [ 2651.339663] [802532b2] handle_edge_irq+0x139/0x142 [ 2651.345594] [80477c79] trace_hardirqs_on_thunk+0x35/0x37 [ 2651.351975] [80242a53] trace_hardirqs_on+0x10f/0x154 [ 2651.358010] [80243bdd] __lock_acquire+0xc08/0xc60 [ 2651.363789] [80253266] handle_edge_irq+0xed/0x142 [ 2651.369899] [80251e47] handle_IRQ_event+0x20/0x55 [ 2651.376086] [80253271] handle_edge_irq+0xf8/0x142 [ 2651.382325] [8020bd78] do_IRQ+0x94/0xf9 [ 2651.387591] [80207c83] default_idle+0x0/0x51 [ 2651.393109] [80209ba6] ret_from_intr+0x0/0xf [ 2651.398582] EOI [80207cb8] default_idle+0x35/0x51 [ 2651.405120] [80207cba] default_idle+0x37/0x51 [ 2651.410972] [80207cb8] default_idle+0x35/0x51 [ 2651.416824] [80207e4e] cpu_idle+0x5b/0x7a [ 2651.422270] [8063566c] start_secondary+0x470/0x47f [ 2651.428267] Kernel is 2.6.20.4. Any ideas? Thanks, Bernd -- Bernd Schubert Q-Leap Networks GmbH ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[ofa-general] ofa_1_2_kernel 20070420-0200 daily build status
This email was generated automatically, please do not reply Common build parameters: --with-ipoib-mod --with-sdp-mod --with-srp-mod --with-user_mad-mod --with-user_access-mod --with-mthca-mod --with-core-mod --with-addr_trans-mod --with-rds-mod --with-cxgb3-mod Passed: Passed on i686 with 2.6.15-23-server Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.13 Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.14 Passed on i686 with linux-2.6.15 Passed on i686 with linux-2.6.12 Passed on powerpc with linux-2.6.18 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.12 Passed on powerpc with linux-2.6.19 Passed on ia64 with linux-2.6.13 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.15 Passed on x86_64 with linux-2.6.15 Passed on ia64 with linux-2.6.14 Passed on x86_64 with linux-2.6.18 Passed on powerpc with linux-2.6.17 Passed on x86_64 with linux-2.6.12 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.14 Passed on powerpc with linux-2.6.13 Passed on x86_64 with linux-2.6.5-7.244-smp Passed on x86_64 with linux-2.6.13 Passed on powerpc with linux-2.6.15 Passed on x86_64 with linux-2.6.17 Passed on powerpc with linux-2.6.12 Passed on ppc64 with linux-2.6.19 Passed on powerpc with linux-2.6.14 Passed on x86_64 with linux-2.6.20 Passed on ppc64 with linux-2.6.13 Passed on ppc64 with linux-2.6.12 Passed on powerpc with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.15 Passed on ppc64 with linux-2.6.14 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.9-22.ELsmp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.9-34.ELsmp Passed on ia64 with linux-2.6.16.21-0.8-default Failed: ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[ofa-general] Just keep in touch
I really hope plate that bucket brake bumpy at some point, you make peapuzzled share complete winter Alright. Stacy pushed a few buttons on her ph berry listen steady She coyly winked, and took week his hand in hers. YLefkowitz, she answered. 5:00 PMI spoke to her. tickle pot uptight She's on warn for tomorrow afternoo overcame squealing Feingold? nervous shrink Dana was ashen-faced. fuzzy With that, feather the two girls fled exited horse the restroom an If episcopal you want him to brick be receptive to paid blow the idea, I Dana sense berry event spoke abecedarian up from the passenger seat. The Mar opinion He's mine, challenge too. 4th period. made carriage Personally, I don'Jeff pulled bid the video itch out of the VCR encouraging steer and the te Yeah. attention arch I ran into sheep him in the art hall. It's a go. W teaching The glow from record the come brain television gave Stacy's face The guard hushed turned a hate degree couple of upset pages on his clip- 3:00 PM Wouldn't it make alot more writing sense throughout if cake roof you were t The three cautious o'clock sound bell rang and pause there screeching was the u enter dusty obey No one I know, called Guy. And I sort had a blood sow Jeff hate brake carriage wasn't sure of what she was getting at. YNo problem. heard The two girls were short brake wrung now outside he destruction strung Dana slew cut herself harmony off mid-sentence, and stopped And what overcome yell the happily fowl starting times are. bright Suddenly an all too tore cart familiar voice earth came from be Carl could feel his heart wink start sea back upset to pound. Lindtwist What's right the point of imagining butyric history an unloaded shotparcel attempt So what bumpy walk class is this boy in with you? Dana, it's race the real principle suffer of watch the matter. You've Huh? drab Dana was thumb horse caught a fresh little off guard. What's that? Glad pencil you cerebral glove hole could make it. You know, your paranoia obtain carelessly about wooly eye guys always star You know examine rudely Guy, he'd rather felt bluff use you than hold 11:45 AM Like slit all organized from dead religion, foolishly Judaism is nonsenstrod The expression on cold her encourage band face grew slightly more fYes? She silk powerfully rolled damp her eyes sound as he jogged up alon Jeff quit chuckled as he been shut boldly march off the television, an Yes..? Dana was now umbrella super encouraging system starting to get cold. Alright, I' You lock said that kick you and screeching he are flame going to be study Of difficult course replace that wasn't shakily the case. Even account if Gavin h kiss So outstanding what am flee error I doing here? Guy decided not to fit mention the alert avoid screw neat, .25 calibr employ Naw, I'm over disgusted it. gladly She number smiled broadly, and lea Call me promptly deserve back as soon as window post you get off the phone painfully Do you short rapid think kneel you could manage sketches of thos I just want to let you basin disturbed forward smoggy know that if you want t orauryh.gif Description: GIF image ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [ofa-general] [RFC] [PATCH 0/3] 2.6.22 or 23 ib: add path record cache
Once SM is up on a node/switch whole network is up. Now is if some client is trying to establish a connection with other node, client is expected to resolve the path using sa API, I want to know how exactly it happens in the stack? See patch 3/3 for the use of the cache. In that patch, the rdma_cm first checks to see if a suitable path record is available in the cache. If one is not found, it issues a query to the SA. The stack impact of using the cache is less than the impact of sending a path record query to the SA. Is it possible to program local_sa_cache with some dummy path records which resides in cache for long time? This would require changes to the current implementation. - Sean ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[ofa-general] OEM - retail version?
The act of putting into your mouth what the earth has grown is perhaps your most direct interaction with the earth. All people are equal, it is not birth, it is virtue alone that makes the difference. Democracy is the theory that the common people know what They want, and deserve to get it good and hard. We must be willing to pay a price for freedom. The man who graduates today and stops learning tomorrow is uneducated the day after. Life is a horizontal fall. The most I can do for my friend is simply be his friend. When you're through changing, you're through. It's faith in something and enthusiasm for something that makes life worth living. Anger is like those ruins which smash themselves on what they fall. Frequent and loud laughter is the characteristic of folly and ill manners. Never fear shadows. They simply mean there's a light shining somewhere nearby. Every time I look at you I get a fierce desire to be lonesome. --- avast! Antivirus: Giden mesaj temiz. Virus Veritabani (VPS): 000735-0, 20.04.2007 Test zamani: 20.04.2007 19:36:52 avast! - telif hakki (c) 1988-2007 ALWIL Software. http://www.avast.com ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[ofa-general] How to set local_ca_ack_delay?
This appears to be set to 0 by default on some HCAs. How can I change that? Basically I am trying to change the Local Ack Timeout and the IB spec 1.2 (page 553) says the Local CA Ack Delay is used to compute the timeout. That is why I want to change the local_ca_ack_delay. Is my understanding correct? Pradeep [EMAIL PROTECTED] ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[ofa-general] [PATCH 6/6] [RFC]mlx4 build system stuff
Hook up mlx4_core and mlx4_ib drivers to Kconfig and Makefiles. Signed-off-by: Roland Dreier [EMAIL PROTECTED] --- infiniband/Kconfig |2 ++ infiniband/Makefile |1 + infiniband/hw/mlx4/Kconfig |9 + infiniband/hw/mlx4/Makefile |3 +++ net/Kconfig | 14 ++ net/Makefile|1 + net/mlx4/Makefile |4 7 files changed, 34 insertions(+) diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig index 82afba5..37deaae 100644 --- a/drivers/infiniband/Kconfig +++ b/drivers/infiniband/Kconfig @@ -45,6 +45,8 @@ source drivers/infiniband/hw/ehca/Kconfig source drivers/infiniband/hw/amso1100/Kconfig source drivers/infiniband/hw/cxgb3/Kconfig +source drivers/infiniband/hw/mlx4/Kconfig + source drivers/infiniband/ulp/ipoib/Kconfig source drivers/infiniband/ulp/srp/Kconfig diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile index da2066c..75f325e 100644 --- a/drivers/infiniband/Makefile +++ b/drivers/infiniband/Makefile @@ -4,6 +4,7 @@ obj-$(CONFIG_INFINIBAND_IPATH) += hw/ipath/ obj-$(CONFIG_INFINIBAND_EHCA) += hw/ehca/ obj-$(CONFIG_INFINIBAND_AMSO1100) += hw/amso1100/ obj-$(CONFIG_INFINIBAND_CXGB3) += hw/cxgb3/ +obj-$(CONFIG_MLX4_INFINIBAND) += hw/mlx4/ obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/ obj-$(CONFIG_INFINIBAND_SRP) += ulp/srp/ obj-$(CONFIG_INFINIBAND_ISER) += ulp/iser/ diff --git a/drivers/infiniband/hw/mlx4/Kconfig b/drivers/infiniband/hw/mlx4/Kconfig new file mode 100644 index 000..b8912cd --- /dev/null +++ b/drivers/infiniband/hw/mlx4/Kconfig @@ -0,0 +1,9 @@ +config MLX4_INFINIBAND + tristate Mellanox ConnectX HCA support + depends on INFINIBAND + select MLX4_CORE + ---help--- + This driver provides low-level InfiniBand support for + Mellanox ConnectX PCI Express host channel adapters (HCAs). + This is required to use InfiniBand protocols such as + IP-over-IB or SRP with these devices. diff --git a/drivers/infiniband/hw/mlx4/Makefile b/drivers/infiniband/hw/mlx4/Makefile new file mode 100644 index 000..70f09c7 --- /dev/null +++ b/drivers/infiniband/hw/mlx4/Makefile @@ -0,0 +1,3 @@ +obj-$(CONFIG_MLX4_INFINIBAND) += mlx4_ib.o + +mlx4_ib-y := ah.o cq.o doorbell.o mad.o main.o mr.o qp.o srq.o diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig index c3f9f59..842f020 100644 --- a/drivers/net/Kconfig +++ b/drivers/net/Kconfig @@ -2493,6 +2493,20 @@ config PASEMI_MAC This driver supports the on-chip 1/10Gbit Ethernet controller on PA Semi's PWRficient line of chips. +config MLX4_CORE + tristate + depends on PCI + default n + +config MLX4_DEBUG + bool Verbose debugging output if (MLX4_CORE EMBEDDED) + default y + ---help--- + This option causes debugging code to be compiled into the + mlx4_core driver. The output can be turned on via the + debug_level module parameter (which can also be set after + the driver is loaded through sysfs). + endmenu source drivers/net/tokenring/Kconfig diff --git a/drivers/net/Makefile b/drivers/net/Makefile index 33af833..1604e1a 100644 --- a/drivers/net/Makefile +++ b/drivers/net/Makefile @@ -197,6 +197,7 @@ obj-$(CONFIG_SMC911X) += smc911x.o obj-$(CONFIG_DM9000) += dm9000.o obj-$(CONFIG_FEC_8XX) += fec_8xx/ obj-$(CONFIG_PASEMI_MAC) += pasemi_mac.o +obj-$(CONFIG_MLX4_CORE) += mlx4/ obj-$(CONFIG_MACB) += macb.o diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile new file mode 100644 index 000..4f18889 --- /dev/null +++ b/drivers/net/mlx4/Makefile @@ -0,0 +1,4 @@ +obj-$(CONFIG_MLX4_CORE)+= mlx4_core.o + +mlx4_core-y := alloc.o cmd.o cq.o eq.o fw.o icm.o intf.o main.o mcg.o mr.o \ + pd.o profile.o qp.o reset.o srq.o ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[ofa-general] [PATCH 3/6] [RFC]mlx4_core public includes
Include files for hardware/firmware information and interface of
mlx4_core module for protocol-specific drivers (such as mlx4_ib).
Signed-off-by: Roland Dreier [EMAIL PROTECTED]
---
cmd.h | 178 +
cq.h | 123 +++
device.h | 323 +
doorbell.h | 97 ++
driver.h | 59 +++
qp.h | 288 ++
srq.h | 42 +++
7 files changed, 1110 insertions(+)
diff --git a/include/linux/mlx4/cmd.h b/include/linux/mlx4/cmd.h
new file mode 100644
index 000..4fb552d
--- /dev/null
+++ b/include/linux/mlx4/cmd.h
@@ -0,0 +1,178 @@
+/*
+ * Copyright (c) 2006 Cisco Systems, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer in the documentation and/or other materials
+ *provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef MLX4_CMD_H
+#define MLX4_CMD_H
+
+#include linux/dma-mapping.h
+
+enum {
+ /* initialization and general commands */
+ MLX4_CMD_SYS_EN = 0x1,
+ MLX4_CMD_SYS_DIS = 0x2,
+ MLX4_CMD_MAP_FA = 0xfff,
+ MLX4_CMD_UNMAP_FA= 0xffe,
+ MLX4_CMD_RUN_FW = 0xff6,
+ MLX4_CMD_MOD_STAT_CFG= 0x34,
+ MLX4_CMD_QUERY_DEV_CAP = 0x3,
+ MLX4_CMD_QUERY_FW= 0x4,
+ MLX4_CMD_ENABLE_LAM = 0xff8,
+ MLX4_CMD_DISABLE_LAM = 0xff7,
+ MLX4_CMD_QUERY_DDR = 0x5,
+ MLX4_CMD_QUERY_ADAPTER = 0x6,
+ MLX4_CMD_INIT_HCA= 0x7,
+ MLX4_CMD_CLOSE_HCA = 0x8,
+ MLX4_CMD_INIT_PORT = 0x9,
+ MLX4_CMD_CLOSE_PORT = 0xa,
+ MLX4_CMD_QUERY_HCA = 0xb,
+ MLX4_CMD_SET_PORT= 0xc,
+ MLX4_CMD_ACCESS_DDR = 0x2e,
+ MLX4_CMD_MAP_ICM = 0xffa,
+ MLX4_CMD_UNMAP_ICM = 0xff9,
+ MLX4_CMD_MAP_ICM_AUX = 0xffc,
+ MLX4_CMD_UNMAP_ICM_AUX = 0xffb,
+ MLX4_CMD_SET_ICM_SIZE= 0xffd,
+
+ /* TPT commands */
+ MLX4_CMD_SW2HW_MPT = 0xd,
+ MLX4_CMD_QUERY_MPT = 0xe,
+ MLX4_CMD_HW2SW_MPT = 0xf,
+ MLX4_CMD_READ_MTT= 0x10,
+ MLX4_CMD_WRITE_MTT = 0x11,
+ MLX4_CMD_SYNC_TPT= 0x2f,
+
+ /* EQ commands */
+ MLX4_CMD_MAP_EQ = 0x12,
+ MLX4_CMD_SW2HW_EQ= 0x13,
+ MLX4_CMD_HW2SW_EQ= 0x14,
+ MLX4_CMD_QUERY_EQ= 0x15,
+
+ /* CQ commands */
+ MLX4_CMD_SW2HW_CQ= 0x16,
+ MLX4_CMD_HW2SW_CQ= 0x17,
+ MLX4_CMD_QUERY_CQ= 0x18,
+ MLX4_CMD_RESIZE_CQ = 0x2c,
+
+ /* SRQ commands */
+ MLX4_CMD_SW2HW_SRQ = 0x35,
+ MLX4_CMD_HW2SW_SRQ = 0x36,
+ MLX4_CMD_QUERY_SRQ = 0x37,
+ MLX4_CMD_ARM_SRQ = 0x40,
+
+ /* QP/EE commands */
+ MLX4_CMD_RST2INIT_QP = 0x19,
+ MLX4_CMD_INIT2RTR_QP = 0x1a,
+ MLX4_CMD_RTR2RTS_QP = 0x1b,
+ MLX4_CMD_RTS2RTS_QP = 0x1c,
+ MLX4_CMD_SQERR2RTS_QP= 0x1d,
+ MLX4_CMD_2ERR_QP = 0x1e,
+ MLX4_CMD_RTS2SQD_QP = 0x1f,
+ MLX4_CMD_SQD2SQD_QP = 0x38,
+ MLX4_CMD_SQD2RTS_QP = 0x20,
+ MLX4_CMD_2RST_QP = 0x21,
+ MLX4_CMD_QUERY_QP= 0x22,
+ MLX4_CMD_INIT2INIT_QP= 0x2d,
+ MLX4_CMD_SUSPEND_QP = 0x32,
+ MLX4_CMD_UNSUSPEND_QP= 0x33,
+ /* special QP and management commands */
+ MLX4_CMD_CONF_SPECIAL_QP = 0x23,
+ MLX4_CMD_MAD_IFC = 0x24,
+
+ /* multicast commands */
+ MLX4_CMD_READ_MCG= 0x25,
+
[ofa-general] [PATCH 0/6] [RFC]IB/mlx4: Mellanox ConnectX adapter driver
As promised, here is a series of patches adding the mlx4_core and mlx4_ib drivers for the new Mellanox ConnectX adapter. These patches are split up in an ad hoc way to avoid mailing list size limits, but when this driver is finally merged, I will give it to Linus to pull in a single changeset. The full driver is also available via git from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git connectx and it is also in my for-mm patch, so Andrew will pick it up for -mm kernels automatically. The driver is split into two kernel modules because the ConnectX adapter can be used as an InfiniBand adapter, 1G/10G ethernet NIC, and an fibre channel HBA at the same time, and so resource management and basic tasks such as issuing commands to the firmware are handled in a mlx4_core module, while everything InfiniBand-specific is in mlx4_ib. In the not-to-distant future, an mlx4_eth module that handles ethernet NIC stuff will be released. My goal is to merge this for 2.6.22. If you feel that would not be appropriate, please do let me know and I will hold off. And of course all criticisms, suggestions, comments, etc. are very much appreciated. My feeling is that the driver is fairly clean already (and I will do some further cleanup before merging) and seems to be reasonably usable, and I trust myself to continue cleaning things up, so there's not much to be gained by waiting a release cycle. The overall driver is not too huge -- 11371 insertions in the diffstat: drivers/infiniband/Kconfig|2 + drivers/infiniband/Makefile |1 + drivers/infiniband/hw/mlx4/Kconfig|9 + drivers/infiniband/hw/mlx4/Makefile |3 + drivers/infiniband/hw/mlx4/ah.c | 100 +++ drivers/infiniband/hw/mlx4/cq.c | 525 ++ drivers/infiniband/hw/mlx4/doorbell.c | 215 ++ drivers/infiniband/hw/mlx4/mad.c | 339 + drivers/infiniband/hw/mlx4/main.c | 612 drivers/infiniband/hw/mlx4/mlx4_ib.h | 285 drivers/infiniband/hw/mlx4/mr.c | 184 + drivers/infiniband/hw/mlx4/qp.c | 1263 + drivers/infiniband/hw/mlx4/srq.c | 334 + drivers/infiniband/hw/mlx4/user.h | 91 +++ drivers/net/Kconfig | 14 + drivers/net/Makefile |1 + drivers/net/mlx4/Makefile |4 + drivers/net/mlx4/alloc.c | 179 + drivers/net/mlx4/cmd.c| 429 +++ drivers/net/mlx4/cq.c | 254 +++ drivers/net/mlx4/eq.c | 704 ++ drivers/net/mlx4/fw.c | 758 drivers/net/mlx4/fw.h | 165 + drivers/net/mlx4/icm.c| 379 ++ drivers/net/mlx4/icm.h| 135 drivers/net/mlx4/intf.c | 142 drivers/net/mlx4/main.c | 939 drivers/net/mlx4/mcg.c| 370 ++ drivers/net/mlx4/mlx4.h | 334 + drivers/net/mlx4/mr.c | 482 + drivers/net/mlx4/pd.c | 102 +++ drivers/net/mlx4/profile.c| 238 +++ drivers/net/mlx4/qp.c | 270 +++ drivers/net/mlx4/reset.c | 172 + drivers/net/mlx4/srq.c| 227 ++ include/linux/mlx4/cmd.h | 178 + include/linux/mlx4/cq.h | 123 include/linux/mlx4/device.h | 323 + include/linux/mlx4/doorbell.h | 97 +++ include/linux/mlx4/driver.h | 59 ++ include/linux/mlx4/qp.h | 288 include/linux/mlx4/srq.h | 42 ++ 42 files changed, 11371 insertions(+), 0 deletions(-) ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[ofa-general] [PATCH 4/6] [RFC]mlx4_ib main files
Main include file and .c file for mlx4_ib.
Signed-off-by: Roland Dreier [EMAIL PROTECTED]
---
main.c| 612 ++
mlx4_ib.h | 285
2 files changed, 897 insertions(+)
diff --git a/drivers/infiniband/hw/mlx4/main.c
b/drivers/infiniband/hw/mlx4/main.c
new file mode 100644
index 000..6f7165f
--- /dev/null
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -0,0 +1,612 @@
+/*
+ * Copyright (c) 2006, 2007 Cisco Systems, Inc. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer in the documentation and/or other materials
+ *provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include linux/module.h
+#include linux/init.h
+#include linux/errno.h
+
+#include rdma/ib_smi.h
+#include rdma/ib_user_verbs.h
+
+#include linux/mlx4/driver.h
+#include linux/mlx4/cmd.h
+
+#include mlx4_ib.h
+#include user.h
+
+#define DRV_NAME mlx4_ib
+#define DRV_VERSION0.01
+#define DRV_RELDATEMay 1, 2006
+
+MODULE_AUTHOR(Roland Dreier);
+MODULE_DESCRIPTION(Mellanox ConnectX HCA InfiniBand driver);
+MODULE_LICENSE(Dual BSD/GPL);
+MODULE_VERSION(DRV_VERSION);
+
+static const char mlx4_ib_version[] __devinitdata =
+ DRV_NAME : Mellanox ConnectX InfiniBand driver v
+ DRV_VERSION ( DRV_RELDATE )\n;
+
+static void init_query_mad(struct ib_smp *mad)
+{
+ mad-base_version = 1;
+ mad-mgmt_class= IB_MGMT_CLASS_SUBN_LID_ROUTED;
+ mad-class_version = 1;
+ mad-method= IB_MGMT_METHOD_GET;
+}
+
+static int mlx4_ib_query_device(struct ib_device *ibdev,
+ struct ib_device_attr *props)
+{
+ struct mlx4_ib_dev *dev = to_mdev(ibdev);
+ struct ib_smp *in_mad = NULL;
+ struct ib_smp *out_mad = NULL;
+ int err = -ENOMEM;
+
+ in_mad = kzalloc(sizeof *in_mad, GFP_KERNEL);
+ out_mad = kmalloc(sizeof *out_mad, GFP_KERNEL);
+ if (!in_mad || !out_mad)
+ goto out;
+
+ init_query_mad(in_mad);
+ in_mad-attr_id = IB_SMP_ATTR_NODE_INFO;
+
+ err = mlx4_MAD_IFC(to_mdev(ibdev), 1, 1, 1, NULL, NULL, in_mad,
out_mad);
+ if (err)
+ goto out;
+
+ memset(props, 0, sizeof *props);
+
+ props-fw_ver = dev-dev-caps.fw_ver;
+ props-device_cap_flags= IB_DEVICE_CHANGE_PHY_PORT |
+ IB_DEVICE_PORT_ACTIVE_EVENT |
+ IB_DEVICE_SYS_IMAGE_GUID|
+ IB_DEVICE_RC_RNR_NAK_GEN;
+ if (dev-dev-caps.flags MLX4_DEV_CAP_FLAG_BAD_PKEY_CNTR)
+ props-device_cap_flags |= IB_DEVICE_BAD_PKEY_CNTR;
+ if (dev-dev-caps.flags MLX4_DEV_CAP_FLAG_BAD_QKEY_CNTR)
+ props-device_cap_flags |= IB_DEVICE_BAD_QKEY_CNTR;
+ if (dev-dev-caps.flags MLX4_DEV_CAP_FLAG_APM)
+ props-device_cap_flags |= IB_DEVICE_AUTO_PATH_MIG;
+ if (dev-dev-caps.flags MLX4_DEV_CAP_FLAG_UD_AV_PORT)
+ props-device_cap_flags |= IB_DEVICE_UD_AV_PORT_ENFORCE;
+
+ props-vendor_id = be32_to_cpup((__be32 *) (out_mad-data +
36))
+ 0xff;
+ props-vendor_part_id = be16_to_cpup((__be16 *) (out_mad-data +
30));
+ props-hw_ver = be32_to_cpup((__be32 *) (out_mad-data +
32));
+ memcpy(props-sys_image_guid, out_mad-data + 4, 8);
+
+ props-max_mr_size = ~0ull;
+ props-page_size_cap = dev-dev-caps.page_size_cap;
+ props-max_qp = dev-dev-caps.num_qps -
dev-dev-caps.reserved_qps;
+ props-max_qp_wr = dev-dev-caps.max_wqes;
+ props-max_sge = min(dev-dev-caps.max_sq_sg,
+
Re: [ofa-general] ipath irq bug
Roland Dreier wrote: [ 2651.241648] [80478444] _spin_unlock_irq+0x28/0x2d [ 2651.247482] [880a0769] :ib_ipath:ipath_rc_rcv+0xf5b/0xf8e [ 2651.254058] [8809d1ff] :ib_ipath:ipath_lookup_qpn+0x4f/0x5a [ 2651.260791] [880a5f40] :ib_ipath:ipath_qp_rcv+0x45/0x4e [ 2651.267203] [880a60b3] :ib_ipath:ipath_ib_rcv+0x16a/0x1a8 [ 2651.273784] [88090a3e] :ib_ipath:ipath_kreceive+0x42f/0x6b9 [ 2651.286397] [880a6b02] :ib_ipath:ipath_ib_piobufavail+0x72/0x79 [ 2651.321572] [8809932c] :ib_ipath:ipath_intr+0x26a/0x17b6 [edited slightly] It looks like ipath_intr() (presumably the interrupt handler) ends up in ipath_rc_rcv() and in particular in ipath_rc_rcv_error() (which is inlined), which calls spin_unlock_irq() from interrupt context, which is the problem. Someone who knows the driver better than I would have to confirm this analysis, and decide whether the fix is just to switch to spin_lock_irqsave() in ipath_rc_rcv_error(), or if it needs to be more elaborate. Yeah - that's it. We've just fixed it internally and will be sending a patch out shortly. Regards, Robert. ___ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
