> Il giorno 19 dic 2018, alle ore 11:32, Ming Lei <[email protected]> ha
> scritto:
>
> On Wed, Dec 19, 2018 at 2:18 PM Paolo Valente <[email protected]>
> wrote:
>>
>>
>>
>>> Il giorno 19 dic 2018, alle ore 04:45, Ming Lei <[email protected]> ha
>>> scritto:
>>>
>>> On Wed, Dec 19, 2018 at 2:52 AM Jens Axboe <[email protected]> wrote:
>>>>
>>>> On 12/18/18 5:45 AM, Paolo Valente wrote:
>>>>> Hi Jens,
>>>>> sorry for the following silly question, but maybe you can solve very
>>>>> quickly a doubt for which I'd spend much more time investigating.
>>>>>
>>>>> While doing some tests with scsi_debug, I've just seen that (at least)
>>>>> with direct I/O, the maximum number of pending I/O requests (at least
>>>>> in the I/O schedulers) is equal, unexpectedly, to the queue depth of
>>>>> the drive and not to
>>>>> /sys/block/<dev>/queue/nr_requests
>>>>>
>>>>> For example, after:
>>>>>
>>>>> sudo modprobe scsi_debug max_queue=4
>>>>>
>>>>> and with fio executed as follows:
>>>>>
>>>>> job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
>>>>> 4096B-4096B, ioengine=libaio, iodepth=20
>>>>>
>>>>> I get this periodic trace, where four insertions are followed by four
>>>>> completions, and so on, till the end of the I/O. This trace is taken
>>>>> with none, but the result is the same with bfq.
>>>>>
>>>>> fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8
>>>>> [fio]
>>>>> fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8
>>>>> [fio]
>>>>> fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8
>>>>> [fio]
>>>>> fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8
>>>>> [fio]
>>>>> <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8
>>>>> [0]
>>>>> <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8
>>>>> [0]
>>>>> <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8
>>>>> [0]
>>>>> <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8
>>>>> [0]
>>>>> fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8
>>>>> [fio]
>>>>> fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8
>>>>> [fio]
>>>>> fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8
>>>>> [fio]
>>>>> fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8
>>>>> [fio]
>>>>> <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8
>>>>> [0]
>>>>> <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8
>>>>> [0]
>>>>> <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8
>>>>> [0]
>>>>> <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8
>>>>> [0]
>>>>>
>>>>> Shouldn't the total number of pending requests reach
>>>>> /sys/block/<dev>/queue/nr_requests ?
>>>>>
>>>>> The latter is of course equal to 8.
>>>>
>>>> With a scheduler, the depth is what the scheduler provides. You cannot
>>>> exceed the hardware queue depth in any situation. You just have 8
>>>> requests available for scheduling, with a max of 4 being inflight on
>>>> the device side.
>>>>
>>>> If both were 4, for instance, then you would have nothing to schedule
>>>> with, as all of them could reside on the hardware side. That's why
>>>> the scheduler defaults to twice the hardware queue depth.
>>>
>>> The default twice to hw queue depth might not be reasonable for multi LUN.
>>>
>>> Maybe it should be set twice of sdev->queue_depth for SCSI or
>>> hw queue depth/hctx->nr_active. But either way may become complicated
>>> because both can be adjusted runtime.
>>>
>>
>> Could you please explain why it is not working (if it is not working)
>> in my example, where there should be only one LUN?
>
> I didn't say it isn't working, and I mean it isn't perfect.
>
> The hardware queue depth is host-wide, that means it is shared by all LUNs.
> Of course, lots of LUNs may be attached to one single HBA. You can setup
> this setting via 'modprobe scsi_debug max_luns=16 max_queue=4' easily,
> then all 4 LUNs share the 4 tags.
>
Ok, so you are talking about the opposite problem, in a sense. What
I'm saying here is that tags had to be 8, as Jens pointed out, but
they are 4.
Thanks,
Paolo
> Thanks,
> Ming Lei
