On 09/12/2018 09:26 PM, 3kboy2...@gmail.com wrote: > Thank you for your reply, Mike. > Now my iscsi disk performance can be around 300MB/s in 4M sequence > write(TCMU+LIO) > It increase from 20MB/s to 300MB/s, after I can change max_data_area_mb > from 8 to 256 && hw_max_sectors from 128 to 8192. > To my cluster, after a lot of tests I found that I should keep > "max_data_area_mb>128 && hw_max_sectors>=4096" in order to get a good > performance. > Does my setting can cause some side effects?
It depends on the kernel. For the RHEL/Centos kernel you are using the kernel will preallocate max_data_area_mb of memory for each device. For upstream, we no longer preallocate, but once it is allocated we do not free the memory unless global_max_data_area_mb is hit or the device is removed. With a high hw_max_sectors latency will increase due to sending really large commands, so it depends on your workload and what you need. We used to set hw_max_sectors to the rbd object size (4MB by default), but in our testing we would see throughput go down around 512k - 1MB. > Are there any other parameters can improve the performance quite obvious? The normal networking ones like using jumbo frames, net.core.*/net.ipv4.*, etc. Check your nic's documentation for the best settings. There are some iscsi ones like the cmdsn/cmds_max I mentioned and then also the segment related ones like MaxRecvDataSegmentLength, MaxXmitDataSegmentLength, MaxBurstLength, FirstBurstLength and have ImmediateData on. > Why the default value of max_data_area_mb and hw_max_sectors is so > small, and bad performance? I do not know. It was just what the original author had used initially. > Could you talk something about this? > At least, max_data_area_mb>128 && hw_max_sectors>=4096, I can get a > better performance seems to be acceptable. > If my settings can give other users some help, I will be happy. > > 在 2018年9月12日星期三 UTC+8上午12:39:16，Mike Christie写道： > > On 09/11/2018 11:30 AM, Mike Christie wrote: > > Hey, > > > > Cc mchr...@redhat.com <javascript:>, or I will not see these > messages until I check > > the list maybe once a week. > > > > On 09/05/2018 10:36 PM, 3kbo...@gmail.com <javascript:> wrote: > >> What lio fabric driver are you using? iSCSI? What kernel > version > >> and > >> what version of tcmu-runner? > >> > >> io fabric driver : iscsi > >> > >> iscsid version: 2.0-873 > >> > >> OS version: CentOS Linux release 7.5.1804 > (Core) > >> > >> kernel version: 3.10.0-862.el7.x86_64 > >> > >> tcmu-runner version: 1.4.0-rc1 > >> > >> > > There is also a perf bug in that initiator if the node.session.cmds_max > is greater than the LIO default_cmdsn_depth and your IO test tries to > send cmds > node.session.cmds_max. > > I have known the bug before, because I had google a lot. > It increase from 320MB/s to 340MB/s (4M seq write), seems like a stable > promotion. > > Settings Before: 320MB/s > node.session.cmds_max = 2048 > default_cmdsn_depth = 64 > > Settings After: 340MB/s > node.session.cmds_max = 64 > default_cmdsn_depth = 64 > > So set the node.session.cmds_max and default_cmdsn_depth to the same > value. You can set the default_cmdsn_depth in saveconfig.json, and set > cmds_max in the iscsiadm node db (after you set it make sure you logout > and login the session again). > > Now I set set the node.session.cmds_max and default_cmdsn_depth both to > be 64. > > Thank you very much! > > -- > You received this message because you are subscribed to the Google > Groups "open-iscsi" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to open-iscsi+unsubscr...@googlegroups.com > <mailto:open-iscsi+unsubscr...@googlegroups.com>. > To post to this group, send email to open-iscsi@googlegroups.com > <mailto:open-iscsi@googlegroups.com>. > Visit this group at https://groups.google.com/group/open-iscsi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at https://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
