On 09/18/2018 03:55 PM, Ming Lei wrote:
> On Tue, Sep 18, 2018 at 03:51:10PM +0800, jianchao.wang wrote:
>>
>>
>> On 09/18/2018 03:39 PM, Ming Lei wrote:
>>> On Tue, Sep 18, 2018 at 09:22:32AM +0800, jianchao.wang wrote:
>>>> Hi Ming
>>>>
>>>> On 09/17/2018 07:35 PM, Ming Lei wrote:
>>>>> On Mon, Sep 17, 2018 at 10:46:34AM +0800, jianchao.wang wrote:
>>>>>> Hi Ming
>>>>>>
>>>>>> On 09/14/2018 07:33 PM, Ming Lei wrote:
>>>>>>> On Fri, Sep 14, 2018 at 11:40 AM jianchao.wang
>>>>>>> <[email protected]> wrote:
>>>>>>>>
>>>>>>>> Hi Ming
>>>>>>>>
>>>>>>>> On 09/13/2018 08:15 PM, Ming Lei wrote:
>>>>>>>>> EXPORT_SYMBOL(__scsi_execute);
>>>>>>>>> @@ -3246,6 +3251,7 @@ static int scsi_internal_device_block(struct
>>>>>>>>> scsi_device *sdev)
>>>>>>>>> else
>>>>>>>>> scsi_wait_for_queuecommand(sdev);
>>>>>>>>> }
>>>>>>>>> + wait_event(sdev->admin_wq,
>>>>>>>>> !atomic_read(&sdev->nr_admin_pending));
>>>>>>>>> mutex_unlock(&sdev->state_mutex);
>>>>>>>>>
>>>>>>>>> return err;
>>>>>>>> ...
>>>>>>>>> diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
>>>>>>>>> index 3aee9464a7bf..8bcb7ecc0c06 100644
>>>>>>>>> --- a/drivers/scsi/scsi_sysfs.c
>>>>>>>>> +++ b/drivers/scsi/scsi_sysfs.c
>>>>>>>>> @@ -1393,6 +1393,7 @@ void __scsi_remove_device(struct scsi_device
>>>>>>>>> *sdev)
>>>>>>>>>
>>>>>>>>> blk_cleanup_queue(sdev->request_queue);
>>>>>>>>> cancel_work_sync(&sdev->requeue_work);
>>>>>>>>> + wait_event(sdev->admin_wq,
>>>>>>>>> !atomic_read(&sdev->nr_admin_pending))
>>>>>>>>
>>>>>>>> This nr_admin_pending could drain the ongoing
>>>>>>>> scsi_request_fn/scsi_queue_rq,
>>>>>>>> but I'm afraid it cannot stop new ones coming in, such as the ones
>>>>>>>> that have passed
>>>>>>>> the sdev state checking and have not crossed the
>>>>>>>> atomic_inc(&sdev->nr_admin_pending).
>>>>>>>>
>>>>>>>
>>>>>>> The counter of .nr_admin_pending is introduced for draining queued
>>>>>>> admin requests to this scsi device.
>>>>>>>
>>>>>>> Actually new requests have been prevented from entering scsi_queue_rq(),
>>>>>>> please see the two callers of wait_event(sdev->admin_wq,
>>>>>>> !atomic_read(&sdev->nr_admin_pending)).
>>>>>>>
>>>>>> For example
>>>>>>
>>>>>> _scsi_execute
>>>>>> ...
>>>>>> scsi_internal_device_block
>>>>>>
>>>>>> scsi_internal_device_block_nowait
>>>>>> blk_mq_quiesce_queue
>>>>>> wait_event(sdev->admin_wq,
>>>>>> !atomic_read(&sdev->nr_admin_pending))
>>>>>> &sdev->nr_admin_pending;
>>>>>>
>>>>>> blk_execute_rq（...）
>>>>>>
>>>>>> atomic_dec(&sdev->nr_admin_pending);
>>>>>> wake_up_all(&sdev->admin_wq);
>>>>>>
>>>>>> Or do you mean the scsi_queue_rq -> scsi_prep_state_check could gate out
>>>>>> of ?
>>>>>
>>>>> I got it, then this issue can be fixed simply by moving
>>>>> atomic_inc/dec(&sdev->nr_admin_pending)
>>>>> and related wake_up_all(&sdev->admin_wq) into scsi_admin_queue_rq().
>>>>>
>>>>
>>>> I don't think so. It is a similar scenario.
>>>>
>>>> I guess a state checking is needed after
>>>> atomic_inc(&sdev->nr_admin_pending), like:
>>>>
>>>> _scsi_execute
>>>> ...
>>>> scsi_internal_device_block
>>>>
>>>> scsi_internal_device_block_nowait
>>>> blk_mq_quiesce_queue
>>>> wait_event(sdev->admin_wq,
>>>> !atomic_read(&sdev->nr_admin_pending))
>>>> atomic_inc(&sdev->nr_admin_pending);
>>>> if state checking fails
>>>> goto done
>>>
>>> The check will be done in scsi_admin_queue_rq().
>>>
>>>>
>>>> blk_execute_rq（...）
>>>>
>>>> atomic_dec(&sdev->nr_admin_pending);
>>>> wake_up_all(&sdev->admin_wq);
>>>
>>> I guess you may misunderstand the purpose of .nr_admin_pending, which is
>>> for draining requests to .queue_rq(). So it is enough to just move the
>>> inc/dec of .nr_admin_pending into scsi_admin_queue_rq(), right?
>> Yes.
>
> Thanks for your confirmation.
>
>>
>> But I just think of how to assign with queue quiesce.
>> The existence of per-host adminq seems to break it.
>
> The per-host adminq won't be frozen or quiesced at all,
This is the place which makes me confused.
Before this patchset, when scsi_internal_device_block quiesces the
request_queue,
Both normal IO or admin requests cannot be processed any more.
Given the per-adminq, we could say every scsi device has two request_queues, or
there could be two request_queues sending requests to the a scsi_device:
- normal IO request_queue
- shared per-host adminq
We could quiesce the normal IO request_queue, but what's about the per-host
adminq ?
> could you explain
> your concern in a bit detail about 'assign with queue quiesce'?
>
The scsi_queue_rq->scsi_prep_state_check could stop requests entering further,
but
the scsi_queue_rq is still invoked.
Thanks
Jianchao
