Re: Connect-IB not performing as well as ConnectX-3 with iSER
Sagi, Here is an example of the different types of tests. This was only on one kernel. The first two are to set a baseline. The lines starting with buffer is using fio with direct=0, the lines starting with direct is fio with direct=1. The lines starting with block is fio running against a raw block deice (technically 40 partitions on a single drive) with direct=0. I also reduced the tests to only test one path per port instead of four like before. # /root/run_path_tests.sh check-paths Test all iSER paths individually 4.5.0-rc5-5adabdd1-00023-g5adabdd buffer;sdc;10.218.128.17;3815778;953944;21984 buffer;sdd;10.219.128.17;3743744;935936;22407 buffer;sde;10.220.128.17;4915392;1228848;17066 direct;sdc;10.218.128.17;876644;219161;95690 direct;sdd;10.219.128.17;881684;220421;95143 direct;sde;10.220.128.17;892215;223053;94020 block;sdc;10.218.128.17;3890459;972614;21562 block;sdd;10.219.128.17;4127642;1031910;20323 block;sde;10.220.128.17;4939705;1234926;16982 # /root/run_path_tests.sh check-paths Test all iSER paths individually 4.5.0-rc5-5adabdd1-00023-g5adabdd buffer;sdc;10.218.128.17;3983572;995893;21058 buffer;sdd;10.219.128.17;3774231;943557;6 buffer;sde;10.220.128.17;4856204;1214051;17274 direct;sdc;10.218.128.17;875820;218955;95780 direct;sdd;10.219.128.17;884072;221018;94886 direct;sde;10.220.128.17;902486;225621;92950 block;sdc;10.218.128.17;3790433;947608;22131 block;sdd;10.219.128.17;3860025;965006;21732 block;sde;10.220.128.17;4946404;1236601;16959 For the following test, I set the IRQ on the initiator using mlx_tune -p HIGH_THROUGHPUT with irqbalance disabled. # /root/run_path_tests.sh check-paths Test all iSER paths individually 4.5.0-rc5-5adabdd1-00023-g5adabdd buffer;sdc;10.218.128.17;3742742;935685;22413 buffer;sdd;10.219.128.17;3786327;946581;22155 buffer;sde;10.220.128.17;5009619;1252404;16745 direct;sdc;10.218.128.17;871942;217985;96206 direct;sdd;10.219.128.17;883467;220866;94951 direct;sde;10.220.128.17;901138;225284;93089 block;sdc;10.218.128.17;3911319;977829;21447 block;sdd;10.219.128.17;3758168;939542;22321 block;sde;10.220.128.17;4968377;1242094;16884 For the following test, I also set the IRQs on the target using mlx_tune -p HIGH_THROUGHPUT and disabled irqbalance. # /root/run_path_tests.sh check-paths Test all iSER paths individually 4.5.0-rc5-5adabdd1-00023-g5adabdd buffer;sdc;10.218.128.17;3804357;951089;22050 buffer;sdd;10.219.128.17;3767113;941778;22268 buffer;sde;10.220.128.17;4966612;1241653;16890 direct;sdc;10.218.128.17;879742;219935;95353 direct;sdd;10.219.128.17;886641;221660;94611 direct;sde;10.220.128.17;886857;221714;94588 block;sdc;10.218.128.17;3760864;940216;22305 block;sdd;10.219.128.17;3763564;940891;22289 block;sde;10.220.128.17;4965436;1241359;16894 It seems that mlx_tune marginally helps, but not really providing anything groundbreaking. Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Jun 22, 2016 at 11:46 AM, Robert LeBlanc wrote: > Sagi, > > Yes you are understanding the data correctly and what I'm seeing. I > think you are also seeing the confusion that I've been running into > trying to figure this out as well. As far as your questions about SRP, > the performance data is from the initiator and the CPU info is from > the target (all fio threads on the initiator were low CPU > utilization). > > I spent a good day tweaking the IRQ assignments (spreading IRQs to all > cores, spreading to all cores on the NUMA node the card is attached > to, and spreading to all non-hyperthreaded cores on the NUMA node). > None of these provided any substantial gains/detriments (irqbalance > was not running). I don't know if there is IRQ steering going on, but > in some cases with irqbalance not running the IRQs would get pinned > back to the previous core(s) and I'd have to set them again. I did not > use the Mellanox scripts, I just did it by hand based on the > documents/scripts. I also offlined all cores on the second NUMA node > which didn't help either. I got more performance gains with nomerges > (1 or 2 provided about the same gain, 2 slightly more) and the queue. > It seems that something in 1aaa57f5 was going right as both cards > performed very well without needing any IRQ fudging. > > I understand that there are many moving parts to try and figure this > out, it could be anywhere in the IB drivers, LIO, and even the SCSI > sub systems, RAM disk implementation or file system. However since the > performance is bouncing between cards, it seems it is unlikely > something very common (except when both cards show a loss/gain), but > as you mentioned, there doesn't seem to be any rhyme or reason to the > shifts. > > I haven't been using the straight block device in these tests, before > when I did, after one thread read the data, if another read that same > block it then started reading it from cache invalidating the test. I > could only saturate the path/port
Re: Connect-IB not performing as well as ConnectX-3 with iSER
Sagi, Yes you are understanding the data correctly and what I'm seeing. I think you are also seeing the confusion that I've been running into trying to figure this out as well. As far as your questions about SRP, the performance data is from the initiator and the CPU info is from the target (all fio threads on the initiator were low CPU utilization). I spent a good day tweaking the IRQ assignments (spreading IRQs to all cores, spreading to all cores on the NUMA node the card is attached to, and spreading to all non-hyperthreaded cores on the NUMA node). None of these provided any substantial gains/detriments (irqbalance was not running). I don't know if there is IRQ steering going on, but in some cases with irqbalance not running the IRQs would get pinned back to the previous core(s) and I'd have to set them again. I did not use the Mellanox scripts, I just did it by hand based on the documents/scripts. I also offlined all cores on the second NUMA node which didn't help either. I got more performance gains with nomerges (1 or 2 provided about the same gain, 2 slightly more) and the queue. It seems that something in 1aaa57f5 was going right as both cards performed very well without needing any IRQ fudging. I understand that there are many moving parts to try and figure this out, it could be anywhere in the IB drivers, LIO, and even the SCSI sub systems, RAM disk implementation or file system. However since the performance is bouncing between cards, it seems it is unlikely something very common (except when both cards show a loss/gain), but as you mentioned, there doesn't seem to be any rhyme or reason to the shifts. I haven't been using the straight block device in these tests, before when I did, after one thread read the data, if another read that same block it then started reading it from cache invalidating the test. I could only saturate the path/port by highly threaded jobs, I may have to partition out the disk for block testing. When I ran the tests using direct I/O the performance was far lower and harder for me to know when I was reaching the theoretical max of the card/links/PCIe. I just may have my scripts run the three tests in succession. Thanks for looking at this. Please let me know what you think would be most helpful so that I'm making the best use of your and my time. Thanks, Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Jun 22, 2016 at 10:21 AM, Sagi Grimberg wrote: > Let me see if I get this correct: > >> 4.5.0_rc3_1aaa57f5_00399 >> >> sdc;10.218.128.17;4627942;1156985;18126 >> sdf;10.218.202.17;4590963;1147740;18272 >> sdk;10.218.203.17;4564980;1141245;18376 >> sdn;10.218.204.17;4571946;1142986;18348 >> sdd;10.219.128.17;4591717;1147929;18269 >> sdi;10.219.202.17;4505644;1126411;18618 >> sdg;10.219.203.17;4562001;1140500;18388 >> sdl;10.219.204.17;4583187;1145796;18303 >> sde;10.220.128.17;5511568;1377892;15220 >> sdh;10.220.202.17;551;137;15209 >> sdj;10.220.203.17;5609983;1402495;14953 >> sdm;10.220.204.17;5509035;1377258;15227 > > > In 1aaa57f5 you get on CIB ~115K IOPs per sd device > and on CX3 you get around 140K IOPs per sd device. > >> >> Mlx5_0;sde;3593013;898253;23347 100% CPU kworker/u69:2 >> Mlx5_0;sdd;3588555;897138;23376 100% CPU kworker/u69:2 >> Mlx4_0;sdc;3525662;881415;23793 100% CPU kworker/u68:0 > > > Is this on the host or the target? > >> 4.5.0_rc5_7861728d_1 >> sdc;10.218.128.17;3747591;936897;22384 >> sdf;10.218.202.17;3750607;937651;22366 >> sdh;10.218.203.17;3750439;937609;22367 >> sdn;10.218.204.17;3771008;942752;22245 >> sde;10.219.128.17;3867678;966919;21689 >> sdg;10.219.202.17;3781889;945472;22181 >> sdk;10.219.203.17;3791804;947951;22123 >> sdl;10.219.204.17;3795406;948851;22102 >> sdd;10.220.128.17;5039110;1259777;16647 >> sdi;10.220.202.17;4992921;1248230;16801 >> sdj;10.220.203.17;5015610;1253902;16725 >> Sdm;10.220.204.17;5087087;1271771;16490 > > > In 7861728d you get on CIB ~95K IOPs per sd device > and on CX3 you get around 125K IOPs per sd device. > > I don't see any difference in the code around iser/isert, > in fact, I don't see any commit in drivers/infiniband > > >> >> Mlx5_0;sde;2930722;732680;28623 ~98% CPU kworker/u69:0 >> Mlx5_0;sdd;2910891;727722;28818 ~98% CPU kworker/u69:0 >> Mlx4_0;sdc;3263668;815917;25703 ~98% CPU kworker/u68:0 > > > Again, host or target? > >> 4.5.0_rc5_f81bf458_00018 >> sdb;10.218.128.17;5023720;1255930;16698 >> sde;10.218.202.17;5016809;1254202;16721 >> sdj;10.218.203.17;5021915;1255478;16704 >> sdk;10.218.204.17;5021314;1255328;16706 >> sdc;10.219.128.17;4984318;1246079;16830 >> sdf;10.219.202.17;4986096;1246524;16824 >> sdh;10.219.203.17;5043958;1260989;16631 >> sdm;10.219.204.17;5032460;1258115;16669 >> sdd;10.220.128.17;3736740;934185;22449 >> sdg;10.220.202.17;3728767;932191;22497 >> sdi;10.220.203.17;3752117;938029;22357 >> Sdl;10.220.204.17;3763901;940975;22287 > > > In f81bf458 you get on CIB ~125K IOPs per sd device > and on CX3
Re: Connect-IB not performing as well as ConnectX-3 with iSER
Let me see if I get this correct: 4.5.0_rc3_1aaa57f5_00399 sdc;10.218.128.17;4627942;1156985;18126 sdf;10.218.202.17;4590963;1147740;18272 sdk;10.218.203.17;4564980;1141245;18376 sdn;10.218.204.17;4571946;1142986;18348 sdd;10.219.128.17;4591717;1147929;18269 sdi;10.219.202.17;4505644;1126411;18618 sdg;10.219.203.17;4562001;1140500;18388 sdl;10.219.204.17;4583187;1145796;18303 sde;10.220.128.17;5511568;1377892;15220 sdh;10.220.202.17;551;137;15209 sdj;10.220.203.17;5609983;1402495;14953 sdm;10.220.204.17;5509035;1377258;15227 In 1aaa57f5 you get on CIB ~115K IOPs per sd device and on CX3 you get around 140K IOPs per sd device. Mlx5_0;sde;3593013;898253;23347 100% CPU kworker/u69:2 Mlx5_0;sdd;3588555;897138;23376 100% CPU kworker/u69:2 Mlx4_0;sdc;3525662;881415;23793 100% CPU kworker/u68:0 Is this on the host or the target? 4.5.0_rc5_7861728d_1 sdc;10.218.128.17;3747591;936897;22384 sdf;10.218.202.17;3750607;937651;22366 sdh;10.218.203.17;3750439;937609;22367 sdn;10.218.204.17;3771008;942752;22245 sde;10.219.128.17;3867678;966919;21689 sdg;10.219.202.17;3781889;945472;22181 sdk;10.219.203.17;3791804;947951;22123 sdl;10.219.204.17;3795406;948851;22102 sdd;10.220.128.17;5039110;1259777;16647 sdi;10.220.202.17;4992921;1248230;16801 sdj;10.220.203.17;5015610;1253902;16725 Sdm;10.220.204.17;5087087;1271771;16490 In 7861728d you get on CIB ~95K IOPs per sd device and on CX3 you get around 125K IOPs per sd device. I don't see any difference in the code around iser/isert, in fact, I don't see any commit in drivers/infiniband Mlx5_0;sde;2930722;732680;28623 ~98% CPU kworker/u69:0 Mlx5_0;sdd;2910891;727722;28818 ~98% CPU kworker/u69:0 Mlx4_0;sdc;3263668;815917;25703 ~98% CPU kworker/u68:0 Again, host or target? 4.5.0_rc5_f81bf458_00018 sdb;10.218.128.17;5023720;1255930;16698 sde;10.218.202.17;5016809;1254202;16721 sdj;10.218.203.17;5021915;1255478;16704 sdk;10.218.204.17;5021314;1255328;16706 sdc;10.219.128.17;4984318;1246079;16830 sdf;10.219.202.17;4986096;1246524;16824 sdh;10.219.203.17;5043958;1260989;16631 sdm;10.219.204.17;5032460;1258115;16669 sdd;10.220.128.17;3736740;934185;22449 sdg;10.220.202.17;3728767;932191;22497 sdi;10.220.203.17;3752117;938029;22357 Sdl;10.220.204.17;3763901;940975;22287 In f81bf458 you get on CIB ~125K IOPs per sd device and on CX3 you get around 93K IOPs per sd device which is the other way around? CIB is better than CX3? The commits in this gap are: f81bf458208e iser-target: Separate flows for np listeners and connections cma events aea92980601f iser-target: Add new state ISER_CONN_BOUND to isert_conn b89a7c25462b iser-target: Fix identification of login rx descriptor type None of those should affect the data-path. Srpt keeps crashing couldn't test 4.5.0_rc5_5adabdd1_00023 Sdc;10.218.128.17;3726448;931612;22511 ~97% CPU kworker/u69:4 sdf;10.218.202.17;3750271;937567;22368 sdi;10.218.203.17;3749266;937316;22374 sdj;10.218.204.17;3798844;949711;22082 sde;10.219.128.17;3759852;939963;22311 ~97% CPU kworker/u69:4 sdg;10.219.202.17;3772534;943133;22236 sdl;10.219.203.17;3769483;942370;22254 sdn;10.219.204.17;3790604;947651;22130 sdd;10.220.128.17;5171130;1292782;16222 ~96% CPU kworker/u68:3 sdh;10.220.202.17;5105354;1276338;16431 sdk;10.220.203.17;4995300;1248825;16793 sdm;10.220.204.17;4959564;1239891;16914 In 5adabdd1 you get on CIB ~94K IOPs per sd device and on CX3 you get around 130K IOPs per sd device which means you flipped again (very strange). The commits in this gap are: 5adabdd122e4 iser-target: Split and properly type the login buffer ed1083b251f0 iser-target: Remove ISER_RECV_DATA_SEG_LEN 26c7b673db57 iser-target: Remove impossible condition from isert_wait_conn 69c48846f1c7 iser-target: Remove redundant wait in release_conn 6d1fba0c2cc7 iser-target: Rework connection termination Again, none are suspected to implicate the data-plane. Srpt crashes 4.5.0_rc5_07b63196_00027 sdb;10.218.128.17;3606142;901535;23262 sdg;10.218.202.17;3570988;892747;23491 sdf;10.218.203.17;3576011;894002;23458 sdk;10.218.204.17;3558113;889528;23576 sdc;10.219.128.17;3577384;894346;23449 sde;10.219.202.17;3575401;893850;23462 sdj;10.219.203.17;3567798;891949;23512 sdl;10.219.204.17;3584262;896065;23404 sdd;10.220.128.17;4430680;1107670;18933 sdh;10.220.202.17;4488286;1122071;18690 sdi;10.220.203.17;4487326;1121831;18694 sdm;10.220.204.17;4441236;1110309;1 In 5adabdd1 you get on CIB ~89K IOPs per sd device and on CX3 you get around 112K IOPs per sd device The commits in this gap are: e3416ab2d156 iser-target: Kill the ->isert_cmd back pointer in struct iser_tx_desc d1ca2ed7dcf8 iser-target: Kill struct isert_rdma_wr 9679cc51eb13 iser-target: Convert to new CQ API Which do effect the data-path, but nothing that can explain a specific CIB issue. Moreover, the perf drop happened before that. Srpt crashes 4.5.0_rc5_5e47f198_00036 sdb;10.218.128.17;3519597;879899;23834 sdi;10.218.202.17;3512229;878057;23884 sdh;10.218.203.17;3518563;879640;23
Re: Connect-IB not performing as well as ConnectX-3 with iSER
There is no need to be concerned about srpt crashing in the latest kernel. Srpt only crashed when I was testing kernels in that change set (7861728..5e47f19) that I identified the 10-15% performance drop in iSER between the 4.5 and 4.6 kernel. My tests from the 4.6 to 4.7rc3 didn't have a problem with srpt crashing. The format of the output is as follows: Kernel_tag_commit iSER tests with results in this format (last three fields are fields 7,8,9 from fio) i.e. sdc;10.218.128.17;3260244;815061;25730 SRP LIO tests i.e. mlx5_0;sdd;2986864;746716;28085 This is repeated for each kernel tested. On some tests I also documented the observed CPU utilization of some of the target processes. In some cases I was lazy and if the information was the same for both mlx5 targets, I didn't duplicate it. For iSER, there are four aliases on each adapter to provide four paths for each IB port (this is a remnant of some previous multipathing tests, and now only serves as providing additional data points to know how repeatable the tests are). 10.218.*.17 and 10.219.*.17 are generally on the mlx5 ports while 19.220.*.17 are on the mlx4 port (some tests had the adapters swapped, but none of these did and it is easy to identify them by the grouping). This test was performed against each path individually. I created an ext4 filesystem on the device (no partitions), then would mount the file system on one path, run the test, umount the path, mount the next path, run the test, etc so that there is no multipathing confusing the tests. I also am _NOT_ running the tests on all paths at the same time using fio. The fio command I'm using is: fio --rw=read --bs=4K --size=2G --numjobs=40 --name=worker.matt --group_reporting --minimal | cut -d';' -f7,8,9 I hope that clears up the confusion, if not, please ask for more clarification. On Jun 22, 2016 2:18 AM, "Bart Van Assche" wrote: > > On 06/21/2016 10:26 PM, Robert LeBlanc wrote: >> >> Srpt keeps crashing couldn't test > > > If this is reproducible with the latest rc kernel or with any of the stable > kernels please report this in a separate e-mail, together with the crash call > stack and information about how to reproduce this. > > Thanks, > > Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Connect-IB not performing as well as ConnectX-3 with iSER
- Original Message - > From: "Bart Van Assche" > To: "Robert LeBlanc" , "Sagi Grimberg" > > Cc: linux-r...@vger.kernel.org, linux-scsi@vger.kernel.org, "Max Gurtovoy" > > Sent: Wednesday, June 22, 2016 4:18:31 AM > Subject: Re: Connect-IB not performing as well as ConnectX-3 with iSER > > On 06/21/2016 10:26 PM, Robert LeBlanc wrote: > > Srpt keeps crashing couldn't test > > If this is reproducible with the latest rc kernel or with any of the > stable kernels please report this in a separate e-mail, together with > the crash call stack and information about how to reproduce this. > > Thanks, > > Bart. > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Robert I am exercising the ib_srpt configured vi a targetlio very heavily in 4.7.0-rc1. I have no crashes or issues. I also had 4.5 running ib_srpt with no crashes, although I had some other timeouts etc depending on the load. What sort of crashes are you talking about ? Does the system crash, ib_srpt dump stack ? Thanks Laurence -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Connect-IB not performing as well as ConnectX-3 with iSER
Sagi & Max, Here is the results of SRP using the same ramdisk backstore that I was using from iSER (as same as can be between reboots and restoring targetcli config). I also tested the commit before 9679cc51eb13 (5adabdd122e471fe978d49471624bab08b5373a7) which is included here. I'm not seeing a correlation between iSER and SRP that would lead me to believe that the changes are happening in both implementations. Does this provide enough information for you, or do you think TGT will be needed? I'm a little lost on which test belongs to what, can you specify that more clearly? 4.4 (afd2ff9) vanilla default config sdb;10.218.128.17;5150176;1287544;16288 sdd;10.218.202.17;5092337;1273084;16473 sdh;10.218.203.17;5129078;1282269;16355 sdk;10.218.204.17;5129078;1282269;16355 sdg;10.219.128.17;5155874;1288968;16270 sdf;10.219.202.17;5131588;1282897;16347 sdi;10.219.203.17;5165399;1291349;16240 sdl;10.219.204.17;5157459;1289364;16265 sdc;10.220.128.17;3684223;921055;22769 sde;10.220.202.17;3692169;923042;22720 sdj;10.220.203.17;3699170;924792;22677 Sdm;10.220.204.17;3697865;924466;22685 mlx5_0;sde;2968368;742092;28260 mlx4_0;sdd;3325645;831411;25224 mlx5_0;sdc;3023466;755866;27745 4.4.0_rc2_3e27c920 sdc;10.218.128.17;5291495;1322873;15853 sde;10.218.202.17;4966024;1241506;16892 sdh;10.218.203.17;4980471;1245117;16843 sdk;10.218.204.17;4966612;1241653;16890 sdd;10.219.128.17;5060084;1265021;16578 sdf;10.219.202.17;5065278;1266319;16561 sdi;10.219.203.17;5047600;1261900;16619 sdl;10.219.204.17;5036992;1259248;16654 sdn;10.220.128.17;3775081;943770;1 sdg;10.220.202.17;3758336;939584;22320 sdj;10.220.203.17;3792832;948208;22117 Sdm;10.220.204.17;3771516;942879;22242 Mlx4_0;sde;4648715;1162178;18045 ~73% cpu ib_srpt_compl Mlx5_0;sdd;3476566;869141;24129 ~80% cpu ib_srpt_compl mlx5_0;sdc;3492343;873085;24020 4.4.0_rc2_ab46db0a sdc;10.218.128.17;3792146;948036;22121 sdf;10.218.202.17;3738405;934601;22439 sdj;10.218.203.17;3764239;941059;22285 sdl;10.218.204.17;3785302;946325;22161 sdd;10.219.128.17;3762382;940595;22296 sdg;10.219.202.17;3765760;941440;22276 sdi;10.219.203.17;3873751;968437;21655 sdm;10.219.204.17;3769483;942370;22254 sde;10.220.128.17;5022517;1255629;16702 sdh;10.220.202.17;5018911;1254727;16714 sdk;10.220.203.17;5037295;1259323;16653 Sdn;10.220.204.17;5033064;1258266;16667 mlx4_0;sde;4635358;1158839;18097 mlx5_0;sdd;3459077;864769;24251 mlx5_0;sdc;3465650;866412;24205 4.5.0_rc3_1aaa57f5_00399 sdc;10.218.128.17;4627942;1156985;18126 sdf;10.218.202.17;4590963;1147740;18272 sdk;10.218.203.17;4564980;1141245;18376 sdn;10.218.204.17;4571946;1142986;18348 sdd;10.219.128.17;4591717;1147929;18269 sdi;10.219.202.17;4505644;1126411;18618 sdg;10.219.203.17;4562001;1140500;18388 sdl;10.219.204.17;4583187;1145796;18303 sde;10.220.128.17;5511568;1377892;15220 sdh;10.220.202.17;551;137;15209 sdj;10.220.203.17;5609983;1402495;14953 sdm;10.220.204.17;5509035;1377258;15227 Mlx5_0;sde;3593013;898253;23347 100% CPU kworker/u69:2 Mlx5_0;sdd;3588555;897138;23376 100% CPU kworker/u69:2 Mlx4_0;sdc;3525662;881415;23793 100% CPU kworker/u68:0 4.5.0_rc5_7861728d_1 sdc;10.218.128.17;3747591;936897;22384 sdf;10.218.202.17;3750607;937651;22366 sdh;10.218.203.17;3750439;937609;22367 sdn;10.218.204.17;3771008;942752;22245 sde;10.219.128.17;3867678;966919;21689 sdg;10.219.202.17;3781889;945472;22181 sdk;10.219.203.17;3791804;947951;22123 sdl;10.219.204.17;3795406;948851;22102 sdd;10.220.128.17;5039110;1259777;16647 sdi;10.220.202.17;4992921;1248230;16801 sdj;10.220.203.17;5015610;1253902;16725 Sdm;10.220.204.17;5087087;1271771;16490 Mlx5_0;sde;2930722;732680;28623 ~98% CPU kworker/u69:0 Mlx5_0;sdd;2910891;727722;28818 ~98% CPU kworker/u69:0 Mlx4_0;sdc;3263668;815917;25703 ~98% CPU kworker/u68:0 4.5.0_rc5_f81bf458_00018 sdb;10.218.128.17;5023720;1255930;16698 sde;10.218.202.17;5016809;1254202;16721 sdj;10.218.203.17;5021915;1255478;16704 sdk;10.218.204.17;5021314;1255328;16706 sdc;10.219.128.17;4984318;1246079;16830 sdf;10.219.202.17;4986096;1246524;16824 sdh;10.219.203.17;5043958;1260989;16631 sdm;10.219.204.17;5032460;1258115;16669 sdd;10.220.128.17;3736740;934185;22449 sdg;10.220.202.17;3728767;932191;22497 sdi;10.220.203.17;3752117;938029;22357 Sdl;10.220.204.17;3763901;940975;22287 Srpt keeps crashing couldn't test 4.5.0_rc5_5adabdd1_00023 Sdc;10.218.128.17;3726448;931612;22511 ~97% CPU kworker/u69:4 sdf;10.218.202.17;3750271;937567;22368 sdi;10.218.203.17;3749266;937316;22374 sdj;10.218.204.17;3798844;949711;22082 sde;10.219.128.17;3759852;939963;22311 ~97% CPU kworker/u69:4 sdg;10.219.202.17;3772534;943133;22236 sdl;10.219.203.17;3769483;942370;22254 sdn;10.219.204.17;3790604;947651;22130 sdd;10.220.128.17;5171130;1292782;16222 ~96% CPU kworker/u68:3 sdh;10.220.202.17;5105354;1276338;16431 sdk;10.220.203.17;4995300;1248825;16793 sdm;10.220.204.17;4959564;1239891;16914 Srpt crashes 4.5.0_rc5_07b63196_00027 sdb;10.218.128.17;3606142;901535;23262 sdg;10.218.202.17;3570988;892747;23491 sdf;10.218
Re: Connect-IB not performing as well as ConnectX-3 with iSER
On 06/21/2016 10:26 PM, Robert LeBlanc wrote: Srpt keeps crashing couldn't test If this is reproducible with the latest rc kernel or with any of the stable kernels please report this in a separate e-mail, together with the crash call stack and information about how to reproduce this. Thanks, Bart. -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Connect-IB not performing as well as ConnectX-3 with iSER
Sagi & Max, Here is the results of SRP using the same ramdisk backstore that I was using from iSER (as same as can be between reboots and restoring targetcli config). I also tested the commit before 9679cc51eb13 (5adabdd122e471fe978d49471624bab08b5373a7) which is included here. I'm not seeing a correlation between iSER and SRP that would lead me to believe that the changes are happening in both implementations. Does this provide enough information for you, or do you think TGT will be needed? 4.4 (afd2ff9) vanilla default config sdb;10.218.128.17;5150176;1287544;16288 sdd;10.218.202.17;5092337;1273084;16473 sdh;10.218.203.17;5129078;1282269;16355 sdk;10.218.204.17;5129078;1282269;16355 sdg;10.219.128.17;5155874;1288968;16270 sdf;10.219.202.17;5131588;1282897;16347 sdi;10.219.203.17;5165399;1291349;16240 sdl;10.219.204.17;5157459;1289364;16265 sdc;10.220.128.17;3684223;921055;22769 sde;10.220.202.17;3692169;923042;22720 sdj;10.220.203.17;3699170;924792;22677 Sdm;10.220.204.17;3697865;924466;22685 mlx5_0;sde;2968368;742092;28260 mlx4_0;sdd;3325645;831411;25224 mlx5_0;sdc;3023466;755866;27745 4.4.0_rc2_3e27c920 sdc;10.218.128.17;5291495;1322873;15853 sde;10.218.202.17;4966024;1241506;16892 sdh;10.218.203.17;4980471;1245117;16843 sdk;10.218.204.17;4966612;1241653;16890 sdd;10.219.128.17;5060084;1265021;16578 sdf;10.219.202.17;5065278;1266319;16561 sdi;10.219.203.17;5047600;1261900;16619 sdl;10.219.204.17;5036992;1259248;16654 sdn;10.220.128.17;3775081;943770;1 sdg;10.220.202.17;3758336;939584;22320 sdj;10.220.203.17;3792832;948208;22117 Sdm;10.220.204.17;3771516;942879;22242 Mlx4_0;sde;4648715;1162178;18045 ~73% cpu ib_srpt_compl Mlx5_0;sdd;3476566;869141;24129 ~80% cpu ib_srpt_compl mlx5_0;sdc;3492343;873085;24020 4.4.0_rc2_ab46db0a sdc;10.218.128.17;3792146;948036;22121 sdf;10.218.202.17;3738405;934601;22439 sdj;10.218.203.17;3764239;941059;22285 sdl;10.218.204.17;3785302;946325;22161 sdd;10.219.128.17;3762382;940595;22296 sdg;10.219.202.17;3765760;941440;22276 sdi;10.219.203.17;3873751;968437;21655 sdm;10.219.204.17;3769483;942370;22254 sde;10.220.128.17;5022517;1255629;16702 sdh;10.220.202.17;5018911;1254727;16714 sdk;10.220.203.17;5037295;1259323;16653 Sdn;10.220.204.17;5033064;1258266;16667 mlx4_0;sde;4635358;1158839;18097 mlx5_0;sdd;3459077;864769;24251 mlx5_0;sdc;3465650;866412;24205 4.5.0_rc3_1aaa57f5_00399 sdc;10.218.128.17;4627942;1156985;18126 sdf;10.218.202.17;4590963;1147740;18272 sdk;10.218.203.17;4564980;1141245;18376 sdn;10.218.204.17;4571946;1142986;18348 sdd;10.219.128.17;4591717;1147929;18269 sdi;10.219.202.17;4505644;1126411;18618 sdg;10.219.203.17;4562001;1140500;18388 sdl;10.219.204.17;4583187;1145796;18303 sde;10.220.128.17;5511568;1377892;15220 sdh;10.220.202.17;551;137;15209 sdj;10.220.203.17;5609983;1402495;14953 sdm;10.220.204.17;5509035;1377258;15227 Mlx5_0;sde;3593013;898253;23347 100% CPU kworker/u69:2 Mlx5_0;sdd;3588555;897138;23376 100% CPU kworker/u69:2 Mlx4_0;sdc;3525662;881415;23793 100% CPU kworker/u68:0 4.5.0_rc5_7861728d_1 sdc;10.218.128.17;3747591;936897;22384 sdf;10.218.202.17;3750607;937651;22366 sdh;10.218.203.17;3750439;937609;22367 sdn;10.218.204.17;3771008;942752;22245 sde;10.219.128.17;3867678;966919;21689 sdg;10.219.202.17;3781889;945472;22181 sdk;10.219.203.17;3791804;947951;22123 sdl;10.219.204.17;3795406;948851;22102 sdd;10.220.128.17;5039110;1259777;16647 sdi;10.220.202.17;4992921;1248230;16801 sdj;10.220.203.17;5015610;1253902;16725 Sdm;10.220.204.17;5087087;1271771;16490 Mlx5_0;sde;2930722;732680;28623 ~98% CPU kworker/u69:0 Mlx5_0;sdd;2910891;727722;28818 ~98% CPU kworker/u69:0 Mlx4_0;sdc;3263668;815917;25703 ~98% CPU kworker/u68:0 4.5.0_rc5_f81bf458_00018 sdb;10.218.128.17;5023720;1255930;16698 sde;10.218.202.17;5016809;1254202;16721 sdj;10.218.203.17;5021915;1255478;16704 sdk;10.218.204.17;5021314;1255328;16706 sdc;10.219.128.17;4984318;1246079;16830 sdf;10.219.202.17;4986096;1246524;16824 sdh;10.219.203.17;5043958;1260989;16631 sdm;10.219.204.17;5032460;1258115;16669 sdd;10.220.128.17;3736740;934185;22449 sdg;10.220.202.17;3728767;932191;22497 sdi;10.220.203.17;3752117;938029;22357 Sdl;10.220.204.17;3763901;940975;22287 Srpt keeps crashing couldn't test 4.5.0_rc5_5adabdd1_00023 Sdc;10.218.128.17;3726448;931612;22511 ~97% CPU kworker/u69:4 sdf;10.218.202.17;3750271;937567;22368 sdi;10.218.203.17;3749266;937316;22374 sdj;10.218.204.17;3798844;949711;22082 sde;10.219.128.17;3759852;939963;22311 ~97% CPU kworker/u69:4 sdg;10.219.202.17;3772534;943133;22236 sdl;10.219.203.17;3769483;942370;22254 sdn;10.219.204.17;3790604;947651;22130 sdd;10.220.128.17;5171130;1292782;16222 ~96% CPU kworker/u68:3 sdh;10.220.202.17;5105354;1276338;16431 sdk;10.220.203.17;4995300;1248825;16793 sdm;10.220.204.17;4959564;1239891;16914 Srpt crashes 4.5.0_rc5_07b63196_00027 sdb;10.218.128.17;3606142;901535;23262 sdg;10.218.202.17;3570988;892747;23491 sdf;10.218.203.17;3576011;894002;23458 sdk;10.218.204.17;3558113;889528;23576 sdc;10.219.128.17;357
Re: Connect-IB not performing as well as ConnectX-3 with iSER
Sagi, I'm working to implement SRP (I think I got it all working) to test some of the commits. I can try TGT afterwards and the commit you mention. I haven't been watching the CPU lately, but before when I was doing a lot of testing, there wasn't any one thread that was at 100%. There are several threads that have high utilization, but none 100% and there is plenty of CPU capacity available (32 cores). I can capture some of that data if it is helpful. I did test 4.7_rc3 on Friday, but it didn't change much, is that "new" enough? 4.7.0_rc3_5edb5649 sdc;10.218.128.17;3260244;815061;25730 sdg;10.218.202.17;3405988;851497;24629 sdh;10.218.203.17;3307419;826854;25363 sdm;10.218.204.17;3430502;857625;24453 sdi;10.219.128.17;3544282;886070;23668 sdj;10.219.202.17;3412083;853020;24585 sdk;10.219.203.17;3422385;855596;24511 sdl;10.219.204.17;3444164;861041;24356 sdb;10.220.128.17;4803646;1200911;17463 sdd;10.220.202.17;4832982;1208245;17357 sde;10.220.203.17;4809430;1202357;17442 sdf;10.220.204.17;4808878;1202219;17444 Thanks for the suggestions, I'll work to get some of the requested data back to you guys quickly. Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Jun 21, 2016 at 7:08 AM, Sagi Grimberg wrote: > Hey Robert, > >> I narrowed the performance degradation to this series >> 7861728..5e47f19, but while trying to bisect it, the changes were >> erratic between each commit that I could not figure out exactly which >> introduced the issue. If someone could give me some pointers on what >> to do, I can keep trying to dig through this. > > > This bisection brings suspects: > > e3416ab2d156 iser-target: Kill the ->isert_cmd back pointer in struct > iser_tx_desc > d1ca2ed7dcf8 iser-target: Kill struct isert_rdma_wr > 9679cc51eb13 iser-target: Convert to new CQ API > 5adabdd122e4 iser-target: Split and properly type the login buffer > ed1083b251f0 iser-target: Remove ISER_RECV_DATA_SEG_LEN > 26c7b673db57 iser-target: Remove impossible condition from isert_wait_conn > 69c48846f1c7 iser-target: Remove redundant wait in release_conn > 6d1fba0c2cc7 iser-target: Rework connection termination > f81bf458208e iser-target: Separate flows for np listeners and connections > cma events > aea92980601f iser-target: Add new state ISER_CONN_BOUND to isert_conn > b89a7c25462b iser-target: Fix identification of login rx descriptor type > > However I don't really see performance implications in these patches, > not to mention something that would affect on ConnectIB... > > Given that your bisection brings up target side patches, I have > a couple questions: > > 1. Are the CPU usage in the target side at 100%, or the initiator side > is the bottleneck? > > 2. Would it be possible to use another target implementation? TGT maybe? > > 3. Can you try testing right before 9679cc51eb13? This is a patch that > involves data-plane. > > 4. Can you try the latest upstream kernel? The iser target code uses > a generic data-transfer library and I'm interested in knowing what is > the status there. > > Cheers, > Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Connect-IB not performing as well as ConnectX-3 with iSER
Hey Robert, I narrowed the performance degradation to this series 7861728..5e47f19, but while trying to bisect it, the changes were erratic between each commit that I could not figure out exactly which introduced the issue. If someone could give me some pointers on what to do, I can keep trying to dig through this. This bisection brings suspects: e3416ab2d156 iser-target: Kill the ->isert_cmd back pointer in struct iser_tx_desc d1ca2ed7dcf8 iser-target: Kill struct isert_rdma_wr 9679cc51eb13 iser-target: Convert to new CQ API 5adabdd122e4 iser-target: Split and properly type the login buffer ed1083b251f0 iser-target: Remove ISER_RECV_DATA_SEG_LEN 26c7b673db57 iser-target: Remove impossible condition from isert_wait_conn 69c48846f1c7 iser-target: Remove redundant wait in release_conn 6d1fba0c2cc7 iser-target: Rework connection termination f81bf458208e iser-target: Separate flows for np listeners and connections cma events aea92980601f iser-target: Add new state ISER_CONN_BOUND to isert_conn b89a7c25462b iser-target: Fix identification of login rx descriptor type However I don't really see performance implications in these patches, not to mention something that would affect on ConnectIB... Given that your bisection brings up target side patches, I have a couple questions: 1. Are the CPU usage in the target side at 100%, or the initiator side is the bottleneck? 2. Would it be possible to use another target implementation? TGT maybe? 3. Can you try testing right before 9679cc51eb13? This is a patch that involves data-plane. 4. Can you try the latest upstream kernel? The iser target code uses a generic data-transfer library and I'm interested in knowing what is the status there. Cheers, Sagi. -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Connect-IB not performing as well as ConnectX-3 with iSER
Did you see this kind of regression in SRP ? or with some other target
(e.g TGT) ?
Trying to understand if it's a ULP issue or LLD...
On 6/20/2016 6:23 PM, Robert LeBlanc wrote:
Adding linux-scsi
This last week I tried to figure out where a 10-15% decrease in
performance showed up between 4.5 and 4.6 using iSER and ConnectX-3
and Connect-IB cards (10.{218,219}.*.17 are Connect-IB and 10.220.*.17
are ConnectX-3). To review, straight RDMA transfers between cards
showed line rate was being achieved, just iSER was not able to achieve
those same rates for some cards on different kernels.
4.5 vanilla default config
sdc;10.218.128.17;3800048;950012;22075
sdi;10.218.202.17;3757158;939289;22327
sdg;10.218.203.17;3774062;943515;7
sdn;10.218.204.17;3816299;954074;21981
sdd;10.219.128.17;3821863;955465;21949
sdf;10.219.202.17;3784106;946026;22168
sdj;10.219.203.17;3827094;956773;21919
sdm;10.219.204.17;3788208;947052;22144
sde;10.220.128.17;5054596;1263649;16596
sdh;10.220.202.17;5013811;1253452;16731
sdl;10.220.203.17;5052160;1263040;16604
sdk;10.220.204.17;4990248;1247562;16810
4.6 vanilla default config
sde;10.218.128.17;3431063;857765;24449
sdf;10.218.202.17;3360685;840171;24961
sdi;10.218.203.17;3355174;838793;25002
sdm;10.218.204.17;3360955;840238;24959
sdd;10.219.128.17;3337288;834322;25136
sdh;10.219.202.17;3327492;831873;25210
sdj;10.219.203.17;3380867;845216;24812
sdk;10.219.204.17;3418340;854585;24540
sdc;10.220.128.17;4668377;1167094;17969
sdg;10.220.202.17;4716675;1179168;17785
sdl;10.220.203.17;4675663;1168915;17941
sdn;10.220.204.17;4631519;1157879;18112
I narrowed the performance degradation to this series
7861728..5e47f19, but while trying to bisect it, the changes were
erratic between each commit that I could not figure out exactly which
introduced the issue. If someone could give me some pointers on what
to do, I can keep trying to dig through this.
4.5.0_rc5_7861728d_1
sdc;10.218.128.17;3747591;936897;22384
sdf;10.218.202.17;3750607;937651;22366
sdh;10.218.203.17;3750439;937609;22367
sdn;10.218.204.17;3771008;942752;22245
sde;10.219.128.17;3867678;966919;21689
sdg;10.219.202.17;3781889;945472;22181
sdk;10.219.203.17;3791804;947951;22123
sdl;10.219.204.17;3795406;948851;22102
sdd;10.220.128.17;5039110;1259777;16647
sdi;10.220.202.17;4992921;1248230;16801
sdj;10.220.203.17;5015610;1253902;16725
sdm;10.220.204.17;5087087;1271771;16490
4.5.0_rc5_f81bf458_00018
sdb;10.218.128.17;5023720;1255930;16698
sde;10.218.202.17;5016809;1254202;16721
sdj;10.218.203.17;5021915;1255478;16704
sdk;10.218.204.17;5021314;1255328;16706
sdc;10.219.128.17;4984318;1246079;16830
sdf;10.219.202.17;4986096;1246524;16824
sdh;10.219.203.17;5043958;1260989;16631
sdm;10.219.204.17;5032460;1258115;16669
sdd;10.220.128.17;3736740;934185;22449
sdg;10.220.202.17;3728767;932191;22497
sdi;10.220.203.17;3752117;938029;22357
sdl;10.220.204.17;3763901;940975;22287
4.5.0_rc5_07b63196_00027
sdb;10.218.128.17;3606142;901535;23262
sdg;10.218.202.17;3570988;892747;23491
sdf;10.218.203.17;3576011;894002;23458
sdk;10.218.204.17;3558113;889528;23576
sdc;10.219.128.17;3577384;894346;23449
sde;10.219.202.17;3575401;893850;23462
sdj;10.219.203.17;3567798;891949;23512
sdl;10.219.204.17;3584262;896065;23404
sdd;10.220.128.17;4430680;1107670;18933
sdh;10.220.202.17;4488286;1122071;18690
sdi;10.220.203.17;4487326;1121831;18694
sdm;10.220.204.17;4441236;1110309;1
4.5.0_rc5_5e47f198_00036
sdb;10.218.128.17;3519597;879899;23834
sdi;10.218.202.17;3512229;878057;23884
sdh;10.218.203.17;3518563;879640;23841
sdk;10.218.204.17;3582119;895529;23418
sdd;10.219.128.17;3550883;887720;23624
sdj;10.219.202.17;3558415;889603;23574
sde;10.219.203.17;3552086;888021;23616
sdl;10.219.204.17;3579521;894880;23435
sdc;10.220.128.17;4532912;1133228;18506
sdf;10.220.202.17;4558035;1139508;18404
sdg;10.220.203.17;4601035;1150258;18232
sdm;10.220.204.17;4548150;1137037;18444
While bisecting the kernel, I also stumbled across one that worked
really well for both adapters which I haven't seen in the release
kernels.
4.5.0_rc3_1aaa57f5_00399
sdc;10.218.128.17;4627942;1156985;18126
sdf;10.218.202.17;4590963;1147740;18272
sdk;10.218.203.17;4564980;1141245;18376
sdn;10.218.204.17;4571946;1142986;18348
sdd;10.219.128.17;4591717;1147929;18269
sdi;10.219.202.17;4505644;1126411;18618
sdg;10.219.203.17;4562001;1140500;18388
sdl;10.219.204.17;4583187;1145796;18303
sde;10.220.128.17;5511568;1377892;15220
sdh;10.220.202.17;551;137;15209
sdj;10.220.203.17;5609983;1402495;14953
sdm;10.220.204.17;5509035;1377258;15227
Here the ConnectX-3 card is performing perfectly while the Connect-IB
card still has some room for improvement.
I'd like to get to the bottom of why I'm not seeing the same
performance out of the newer kernels, but I just don't understand the
code. I've tried to do what I can in narrowing down where major
changes happened in the kernel to cause these changes in hopes that it
would help someone on the list. If there is anything I can do to help
out
Re: Connect-IB not performing as well as ConnectX-3 with iSER
I can test with SRP and report back what I find (haven't used SRP in
years so I'll need to brush up on it).
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Mon, Jun 20, 2016 at 3:27 PM, Max Gurtovoy wrote:
> Did you see this kind of regression in SRP ? or with some other target (e.g
> TGT) ?
> Trying to understand if it's a ULP issue or LLD...
>
>
> On 6/20/2016 6:23 PM, Robert LeBlanc wrote:
>>
>> Adding linux-scsi
>>
>> This last week I tried to figure out where a 10-15% decrease in
>> performance showed up between 4.5 and 4.6 using iSER and ConnectX-3
>> and Connect-IB cards (10.{218,219}.*.17 are Connect-IB and 10.220.*.17
>> are ConnectX-3). To review, straight RDMA transfers between cards
>> showed line rate was being achieved, just iSER was not able to achieve
>> those same rates for some cards on different kernels.
>>
>> 4.5 vanilla default config
>> sdc;10.218.128.17;3800048;950012;22075
>> sdi;10.218.202.17;3757158;939289;22327
>> sdg;10.218.203.17;3774062;943515;7
>> sdn;10.218.204.17;3816299;954074;21981
>> sdd;10.219.128.17;3821863;955465;21949
>> sdf;10.219.202.17;3784106;946026;22168
>> sdj;10.219.203.17;3827094;956773;21919
>> sdm;10.219.204.17;3788208;947052;22144
>> sde;10.220.128.17;5054596;1263649;16596
>> sdh;10.220.202.17;5013811;1253452;16731
>> sdl;10.220.203.17;5052160;1263040;16604
>> sdk;10.220.204.17;4990248;1247562;16810
>>
>> 4.6 vanilla default config
>> sde;10.218.128.17;3431063;857765;24449
>> sdf;10.218.202.17;3360685;840171;24961
>> sdi;10.218.203.17;3355174;838793;25002
>> sdm;10.218.204.17;3360955;840238;24959
>> sdd;10.219.128.17;3337288;834322;25136
>> sdh;10.219.202.17;3327492;831873;25210
>> sdj;10.219.203.17;3380867;845216;24812
>> sdk;10.219.204.17;3418340;854585;24540
>> sdc;10.220.128.17;4668377;1167094;17969
>> sdg;10.220.202.17;4716675;1179168;17785
>> sdl;10.220.203.17;4675663;1168915;17941
>> sdn;10.220.204.17;4631519;1157879;18112
>>
>> I narrowed the performance degradation to this series
>> 7861728..5e47f19, but while trying to bisect it, the changes were
>> erratic between each commit that I could not figure out exactly which
>> introduced the issue. If someone could give me some pointers on what
>> to do, I can keep trying to dig through this.
>>
>> 4.5.0_rc5_7861728d_1
>> sdc;10.218.128.17;3747591;936897;22384
>> sdf;10.218.202.17;3750607;937651;22366
>> sdh;10.218.203.17;3750439;937609;22367
>> sdn;10.218.204.17;3771008;942752;22245
>> sde;10.219.128.17;3867678;966919;21689
>> sdg;10.219.202.17;3781889;945472;22181
>> sdk;10.219.203.17;3791804;947951;22123
>> sdl;10.219.204.17;3795406;948851;22102
>> sdd;10.220.128.17;5039110;1259777;16647
>> sdi;10.220.202.17;4992921;1248230;16801
>> sdj;10.220.203.17;5015610;1253902;16725
>> sdm;10.220.204.17;5087087;1271771;16490
>>
>> 4.5.0_rc5_f81bf458_00018
>> sdb;10.218.128.17;5023720;1255930;16698
>> sde;10.218.202.17;5016809;1254202;16721
>> sdj;10.218.203.17;5021915;1255478;16704
>> sdk;10.218.204.17;5021314;1255328;16706
>> sdc;10.219.128.17;4984318;1246079;16830
>> sdf;10.219.202.17;4986096;1246524;16824
>> sdh;10.219.203.17;5043958;1260989;16631
>> sdm;10.219.204.17;5032460;1258115;16669
>> sdd;10.220.128.17;3736740;934185;22449
>> sdg;10.220.202.17;3728767;932191;22497
>> sdi;10.220.203.17;3752117;938029;22357
>> sdl;10.220.204.17;3763901;940975;22287
>>
>> 4.5.0_rc5_07b63196_00027
>> sdb;10.218.128.17;3606142;901535;23262
>> sdg;10.218.202.17;3570988;892747;23491
>> sdf;10.218.203.17;3576011;894002;23458
>> sdk;10.218.204.17;3558113;889528;23576
>> sdc;10.219.128.17;3577384;894346;23449
>> sde;10.219.202.17;3575401;893850;23462
>> sdj;10.219.203.17;3567798;891949;23512
>> sdl;10.219.204.17;3584262;896065;23404
>> sdd;10.220.128.17;4430680;1107670;18933
>> sdh;10.220.202.17;4488286;1122071;18690
>> sdi;10.220.203.17;4487326;1121831;18694
>> sdm;10.220.204.17;4441236;1110309;1
>>
>> 4.5.0_rc5_5e47f198_00036
>> sdb;10.218.128.17;3519597;879899;23834
>> sdi;10.218.202.17;3512229;878057;23884
>> sdh;10.218.203.17;3518563;879640;23841
>> sdk;10.218.204.17;3582119;895529;23418
>> sdd;10.219.128.17;3550883;887720;23624
>> sdj;10.219.202.17;3558415;889603;23574
>> sde;10.219.203.17;3552086;888021;23616
>> sdl;10.219.204.17;3579521;894880;23435
>> sdc;10.220.128.17;4532912;1133228;18506
>> sdf;10.220.202.17;4558035;1139508;18404
>> sdg;10.220.203.17;4601035;1150258;18232
>> sdm;10.220.204.17;4548150;1137037;18444
>>
>> While bisecting the kernel, I also stumbled across one that worked
>> really well for both adapters which I haven't seen in the release
>> kernels.
>>
>> 4.5.0_rc3_1aaa57f5_00399
>> sdc;10.218.128.17;4627942;1156985;18126
>> sdf;10.218.202.17;4590963;1147740;18272
>> sdk;10.218.203.17;4564980;1141245;18376
>> sdn;10.218.204.17;4571946;1142986;18348
>> sdd;10.219.128.17;4591717;1147929;18269
>> sdi;10.219.202.17;4505644;1126411;18618
>> sdg;10.219.203.17;4562001;1140500;18388
>> sdl;10.219.204.17;4583187;1145796;18303
>> sde;10.220
Re: Connect-IB not performing as well as ConnectX-3 with iSER
Adding linux-scsi
This last week I tried to figure out where a 10-15% decrease in
performance showed up between 4.5 and 4.6 using iSER and ConnectX-3
and Connect-IB cards (10.{218,219}.*.17 are Connect-IB and 10.220.*.17
are ConnectX-3). To review, straight RDMA transfers between cards
showed line rate was being achieved, just iSER was not able to achieve
those same rates for some cards on different kernels.
4.5 vanilla default config
sdc;10.218.128.17;3800048;950012;22075
sdi;10.218.202.17;3757158;939289;22327
sdg;10.218.203.17;3774062;943515;7
sdn;10.218.204.17;3816299;954074;21981
sdd;10.219.128.17;3821863;955465;21949
sdf;10.219.202.17;3784106;946026;22168
sdj;10.219.203.17;3827094;956773;21919
sdm;10.219.204.17;3788208;947052;22144
sde;10.220.128.17;5054596;1263649;16596
sdh;10.220.202.17;5013811;1253452;16731
sdl;10.220.203.17;5052160;1263040;16604
sdk;10.220.204.17;4990248;1247562;16810
4.6 vanilla default config
sde;10.218.128.17;3431063;857765;24449
sdf;10.218.202.17;3360685;840171;24961
sdi;10.218.203.17;3355174;838793;25002
sdm;10.218.204.17;3360955;840238;24959
sdd;10.219.128.17;3337288;834322;25136
sdh;10.219.202.17;3327492;831873;25210
sdj;10.219.203.17;3380867;845216;24812
sdk;10.219.204.17;3418340;854585;24540
sdc;10.220.128.17;4668377;1167094;17969
sdg;10.220.202.17;4716675;1179168;17785
sdl;10.220.203.17;4675663;1168915;17941
sdn;10.220.204.17;4631519;1157879;18112
I narrowed the performance degradation to this series
7861728..5e47f19, but while trying to bisect it, the changes were
erratic between each commit that I could not figure out exactly which
introduced the issue. If someone could give me some pointers on what
to do, I can keep trying to dig through this.
4.5.0_rc5_7861728d_1
sdc;10.218.128.17;3747591;936897;22384
sdf;10.218.202.17;3750607;937651;22366
sdh;10.218.203.17;3750439;937609;22367
sdn;10.218.204.17;3771008;942752;22245
sde;10.219.128.17;3867678;966919;21689
sdg;10.219.202.17;3781889;945472;22181
sdk;10.219.203.17;3791804;947951;22123
sdl;10.219.204.17;3795406;948851;22102
sdd;10.220.128.17;5039110;1259777;16647
sdi;10.220.202.17;4992921;1248230;16801
sdj;10.220.203.17;5015610;1253902;16725
sdm;10.220.204.17;5087087;1271771;16490
4.5.0_rc5_f81bf458_00018
sdb;10.218.128.17;5023720;1255930;16698
sde;10.218.202.17;5016809;1254202;16721
sdj;10.218.203.17;5021915;1255478;16704
sdk;10.218.204.17;5021314;1255328;16706
sdc;10.219.128.17;4984318;1246079;16830
sdf;10.219.202.17;4986096;1246524;16824
sdh;10.219.203.17;5043958;1260989;16631
sdm;10.219.204.17;5032460;1258115;16669
sdd;10.220.128.17;3736740;934185;22449
sdg;10.220.202.17;3728767;932191;22497
sdi;10.220.203.17;3752117;938029;22357
sdl;10.220.204.17;3763901;940975;22287
4.5.0_rc5_07b63196_00027
sdb;10.218.128.17;3606142;901535;23262
sdg;10.218.202.17;3570988;892747;23491
sdf;10.218.203.17;3576011;894002;23458
sdk;10.218.204.17;3558113;889528;23576
sdc;10.219.128.17;3577384;894346;23449
sde;10.219.202.17;3575401;893850;23462
sdj;10.219.203.17;3567798;891949;23512
sdl;10.219.204.17;3584262;896065;23404
sdd;10.220.128.17;4430680;1107670;18933
sdh;10.220.202.17;4488286;1122071;18690
sdi;10.220.203.17;4487326;1121831;18694
sdm;10.220.204.17;4441236;1110309;1
4.5.0_rc5_5e47f198_00036
sdb;10.218.128.17;3519597;879899;23834
sdi;10.218.202.17;3512229;878057;23884
sdh;10.218.203.17;3518563;879640;23841
sdk;10.218.204.17;3582119;895529;23418
sdd;10.219.128.17;3550883;887720;23624
sdj;10.219.202.17;3558415;889603;23574
sde;10.219.203.17;3552086;888021;23616
sdl;10.219.204.17;3579521;894880;23435
sdc;10.220.128.17;4532912;1133228;18506
sdf;10.220.202.17;4558035;1139508;18404
sdg;10.220.203.17;4601035;1150258;18232
sdm;10.220.204.17;4548150;1137037;18444
While bisecting the kernel, I also stumbled across one that worked
really well for both adapters which I haven't seen in the release
kernels.
4.5.0_rc3_1aaa57f5_00399
sdc;10.218.128.17;4627942;1156985;18126
sdf;10.218.202.17;4590963;1147740;18272
sdk;10.218.203.17;4564980;1141245;18376
sdn;10.218.204.17;4571946;1142986;18348
sdd;10.219.128.17;4591717;1147929;18269
sdi;10.219.202.17;4505644;1126411;18618
sdg;10.219.203.17;4562001;1140500;18388
sdl;10.219.204.17;4583187;1145796;18303
sde;10.220.128.17;5511568;1377892;15220
sdh;10.220.202.17;551;137;15209
sdj;10.220.203.17;5609983;1402495;14953
sdm;10.220.204.17;5509035;1377258;15227
Here the ConnectX-3 card is performing perfectly while the Connect-IB
card still has some room for improvement.
I'd like to get to the bottom of why I'm not seeing the same
performance out of the newer kernels, but I just don't understand the
code. I've tried to do what I can in narrowing down where major
changes happened in the kernel to cause these changes in hopes that it
would help someone on the list. If there is anything I can do to help
out, please let me know.
Thank you,
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Fri, Jun 10, 2016 at 3:36 PM, Robert LeBlanc
