I have more numbers, this time its on Qlogic switches, which do not
handle DR packets forwarding in HW.
Fabric of ~1100 HCAs, ~280 switches.
Current OSM configures LFTs in ~2 seconds.
New algorithm does the same job in 1.4-1.6 seconds (30%-20% speed up),
depending on the max_smps_per_node value.
As in case of IS4 switches, the shortest config time was obtained with
max_smps_per_node=0, which is unlimited pipeline.
-- Yevgeny
Yevgeny Kliteynik wrote:
Sasha,
Hal Rosenstock wrote:
Currently, MADs are pipelined to a single switch at a time which
effectively serializes these requests due to processing at the SMA.
This patch pipelines (stripes) them across the switches first before
proceeding with successive blocks. As a result of this striping,
multiple switches can process the set and respond concurrently
which results in an improvement to the subnet initialization time.
I have some numbers, and they look very good.
I have a small cluster of 17 IS4 switches and 11 HCAs.
To artificially increase the cluster I used LMC=7, including
ExtendedSwitchPort 0 LMC.
With the new code, LFT configuration is more than twice as
fast as with the old code :)
Current ucast manager ran on avarage for ~250msec, with the
new code - 110-120msec.
Routing calculation phase of the ucast manager took ~1200 usec,
the rest was sending the blocks and waiting for no more pending
transactions.
I also didn't see any noticeble difference between various
max_smps_per_node values.
Here are some detailed results of different executions (the
number on the left is timer value in usec):
Current ucast manager (w/o the optimization):
000000 [LFT]: osm_ucast_mgr_process() - START
001131 [LFT]: ucast_mgr_process_tbl() - START
032251 [LFT]: ucast_mgr_process_tbl() - END
032263 [LFT]: osm_ucast_mgr_process() - END
253416 [LFT]: Done wait_for_pending_transactions()
New code, max_smps_per_node=0:
001417 [LFT]: osm_ucast_mgr_process() - START (0 max_smps_per_node)
002690 [LFT]: ucast_mgr_process_tbl() - START
032946 [LFT]: ucast_mgr_process_tbl() - END
032948 [LFT]: osm_ucast_pipeline_tbl() - START
033846 [LFT]: osm_ucast_pipeline_tbl() - END
033858 [LFT]: osm_ucast_mgr_process() - END
108203 [LFT]: Done wait_for_pending_transactions()
New code, max_smps_per_node=1:
007474 [LFT]: osm_ucast_mgr_process() - START (1 max_smps_per_node)
008735 [LFT]: ucast_mgr_process_tbl() - START
040071 [LFT]: ucast_mgr_process_tbl() - END
040074 [LFT]: osm_ucast_pipeline_tbl() - START
040103 [LFT]: osm_ucast_pipeline_tbl() - END
040114 [LFT]: osm_ucast_mgr_process() - END
120097 [LFT]: Done wait_for_pending_transactions()
New code, max_smps_per_node=4:
004137 [LFT]: osm_ucast_mgr_process() - START (4 max_smps_per_node)
005380 [LFT]: ucast_mgr_process_tbl() - START
037436 [LFT]: ucast_mgr_process_tbl() - END
037439 [LFT]: osm_ucast_pipeline_tbl() - START
037495 [LFT]: osm_ucast_pipeline_tbl() - END
037506 [LFT]: osm_ucast_mgr_process() - END
114983 [LFT]: Done wait_for_pending_transactions()
Here's the patch that shows where I added the log messages in current code:
From d7962cc06f5526982bc51c17d53981b6d23256a8 Mon Sep 17 00:00:00 2001
From: Yevgeny Kliteynik <[email protected]>
Date: Mon, 27 Jul 2009 16:12:27 +0300
Subject: [PATCH] [LFT - master] log messages for LFT parallelization
Signed-off-by: Yevgeny Kliteynik <[email protected]>
---
opensm/opensm/osm_state_mgr.c | 9 ++++++++-
opensm/opensm/osm_ucast_mgr.c | 4 ++++
2 files changed, 12 insertions(+), 1 deletions(-)
diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c
index adc39a0..573a2a7 100644
--- a/opensm/opensm/osm_state_mgr.c
+++ b/opensm/opensm/osm_state_mgr.c
@@ -1271,11 +1271,18 @@ _repeat_discovery:
*/
if (!sm->ucast_mgr.cache_valid ||
- osm_ucast_cache_process(&sm->ucast_mgr))
+ osm_ucast_cache_process(&sm->ucast_mgr)) {
+ OSM_LOG(sm->p_log, OSM_LOG_INFO, "[LFT]: "
+ "osm_ucast_mgr_process() - START\n");
osm_ucast_mgr_process(&sm->ucast_mgr);
+ OSM_LOG(sm->p_log, OSM_LOG_INFO, "[LFT]: "
+ "osm_ucast_mgr_process() - END\n");
+ }
if (wait_for_pending_transactions(&sm->p_subn->p_osm->stats))
return;
+ OSM_LOG(sm->p_log, OSM_LOG_INFO, "[LFT]: "
+ "Done wait_for_pending_transactions()\n");
/* cleanup switch lft buffers */
cl_qmap_apply_func(&sm->p_subn->sw_guid_tbl, cleanup_switch,
sm->p_log);
diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c
index 78a7031..83867a0 100644
--- a/opensm/opensm/osm_ucast_mgr.c
+++ b/opensm/opensm/osm_ucast_mgr.c
@@ -901,8 +901,12 @@ static int ucast_mgr_build_lfts(osm_ucast_mgr_t *
p_mgr)
cl_qmap_apply_func(&p_mgr->p_subn->port_guid_tbl,
add_port_to_order_list, p_mgr);
+ OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "[LFT]: "
+ "ucast_mgr_process_tbl() - START\n");
cl_qmap_apply_func(&p_mgr->p_subn->sw_guid_tbl, ucast_mgr_process_tbl,
p_mgr);
+ OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "[LFT]: "
+ "ucast_mgr_process_tbl() - END\n");
cl_qlist_remove_all(&p_mgr->port_order_list);
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
