from:"Daejun Park"

[PATCH v3] scsi: ufs: Add batched WB buffer flush

2021-04-20 Thread Daejun Park

Currently, WriteBooster (WB) buffer is always flushed during hibern8. However,
this is inefficient because data in the WB buffer can be invalid due to
spatial locality of IO workload.
If the WB buffer flush is flushed in a batched manner, the amount of data
migration and power consumption can be reduced because the overwritten data
of the WB buffer may be invalid due to spatial locality.

This patch supports batched flush of WB buffer. When batched flush is enabled,
fWriteBoosterBufferFlushDuringHibernate is set only when
b_rpm_dev_flush_capable is true during runtime suspend. When the device is
resumed, fWriteBoosterBufferFlushDuringHibernate is cleared to stop flush
during hibern8.

Changelog
  - Add reported-by tag by kernel test robot.
  - Fix warning reported by kernel test robot.

Reported-by: kernel test robot 
Co-developed-by: Keoseong Park 
Signed-off-by: Keoseong Park 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  9 +
 drivers/scsi/ufs/ufs-sysfs.c   | 47 ++
 drivers/scsi/ufs/ufshcd.c  | 14 +--
 drivers/scsi/ufs/ufshcd.h  |  2 +
 4 files changed, 68 insertions(+), 4 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..b67b8449e840 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,12 @@ Description: This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/wb_batched_flush
+Date:  April 2021
+Contact:   Daejun Park 
+Description:   This entry shows whether batch flushing of UFS WriteBooster
+   buffers is enabled. Writing 1 to this entry allows the device 
to flush
+   the WriteBooster buffer only when it needs to perform a buffer 
flush
+   during runtime suspend. Writing 0 to this entry allows the 
device to
+   flush the WriteBooster buffer during link hibernation.
diff --git a/drivers/scsi/ufs/ufs-sysfs.c b/drivers/scsi/ufs/ufs-sysfs.c
index d7c3cff9662f..b8fbe8676275 100644
--- a/drivers/scsi/ufs/ufs-sysfs.c
+++ b/drivers/scsi/ufs/ufs-sysfs.c
@@ -253,6 +253,51 @@ static ssize_t wb_on_store(struct device *dev, struct 
device_attribute *attr,
return res < 0 ? res : count;
 }
 
+
+static ssize_t wb_batched_flush_show(struct device *dev,
+struct device_attribute *attr, char *buf)
+{
+   struct ufs_hba *hba = dev_get_drvdata(dev);
+
+   return sysfs_emit(buf, "%d\n", hba->vps->wb_batched_flush);
+}
+
+static ssize_t wb_batched_flush_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+   struct ufs_hba *hba = dev_get_drvdata(dev);
+   unsigned int wb_batched_flush;
+   ssize_t res = 0;
+
+   if (!ufshcd_is_wb_allowed(hba)) {
+   dev_warn(dev, "To control WB through wb_batched_flush is not 
allowed!\n");
+   return -EOPNOTSUPP;
+   }
+
+   if (kstrtouint(buf, 0, _batched_flush))
+   return -EINVAL;
+
+   if (wb_batched_flush != 0 && wb_batched_flush != 1)
+   return -EINVAL;
+
+   down(>host_sem);
+   if (!ufshcd_is_user_access_allowed(hba)) {
+   res = -EBUSY;
+   goto out;
+   }
+
+   pm_runtime_get_sync(hba->dev);
+   res = ufshcd_wb_toggle_flush_during_h8(hba, !wb_batched_flush);
+   pm_runtime_put_sync(hba->dev);
+   if (!res)
+   hba->vps->wb_batched_flush = wb_batched_flush;
+
+out:
+   up(>host_sem);
+   return res < 0 ? res : count;
+}
+
 static DEVICE_ATTR_RW(rpm_lvl);
 static DEVICE_ATTR_RO(rpm_target_dev_state);
 static DEVICE_ATTR_RO(rpm_target_link_state);
@@ -261,6 +306,7 @@ static DEVICE_ATTR_RO(spm_target_dev_state);
 static DEVICE_ATTR_RO(spm_target_link_state);
 static DEVICE_ATTR_RW(auto_hibern8);
 static DEVICE_ATTR_RW(wb_on);
+static DEVICE_ATTR_RW(wb_batched_flush);
 
 static struct attribute *ufs_sysfs_ufshcd_attrs[] = {
_attr_rpm_lvl.attr,
@@ -271,6 +317,7 @@ static struct attribute *ufs_sysfs_ufshcd_attrs[] = {
_attr_spm_target_link_state.attr,
_attr_auto_hibern8.attr,
_attr_wb_on.attr,
+   _attr_wb_batched_flush.attr,
NULL
 };
 
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 0625da7a42ee..e11dc578a17c 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -244,7 +244,6 @@ static int ufshcd_setup_vreg(struct ufs_hba *hba,

[PATCH v2] scsi: ufs: Add batched WB buffer flush

2021-04-20 Thread Daejun Park

Currently, WriteBooster (WB) buffer is always flushed during hibern8. However,
this is inefficient because data in the WB buffer can be invalid due to
spatial locality of IO workload.
If the WB buffer flush is flushed in a batched manner, the amount of data
migration and power consumption can be reduced because the overwritten data
of the WB buffer may be invalid due to spatial locality.

This patch supports batched flush of WB buffer. When batched flush is enabled,
fWriteBoosterBufferFlushDuringHibernate is set only when
b_rpm_dev_flush_capable is true during runtime suspend. When the device is
resumed, fWriteBoosterBufferFlushDuringHibernate is cleared to stop flush
during hibern8.

Changelog
 Fix warning reported by kernel test robot.

Co-developed-by: Keoseong Park 
Signed-off-by: Keoseong Park 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  9 
 drivers/scsi/ufs/ufs-sysfs.c   | 50 ++
 drivers/scsi/ufs/ufshcd.c  | 14 --
 drivers/scsi/ufs/ufshcd.h  |  2 +
 4 files changed, 71 insertions(+), 4 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..b67b8449e840 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,12 @@ Description: This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/wb_batched_flush
+Date:  April 2021
+Contact:   Daejun Park 
+Description:   This entry shows whether batch flushing of UFS WriteBooster
+   buffers is enabled. Writing 1 to this entry allows the device 
to flush
+   the WriteBooster buffer only when it needs to perform a buffer 
flush
+   during runtime suspend. Writing 0 to this entry allows the 
device to
+   flush the WriteBooster buffer during link hibernation.
diff --git a/drivers/scsi/ufs/ufs-sysfs.c b/drivers/scsi/ufs/ufs-sysfs.c
index d7c3cff9662f..943ac12e59c6 100644
--- a/drivers/scsi/ufs/ufs-sysfs.c
+++ b/drivers/scsi/ufs/ufs-sysfs.c
@@ -253,6 +253,54 @@ static ssize_t wb_on_store(struct device *dev, struct 
device_attribute *attr,
return res < 0 ? res : count;
 }
 
+
+static ssize_t wb_batched_flush_show(struct device *dev,
+struct device_attribute *attr, char *buf)
+{
+   struct ufs_hba *hba = dev_get_drvdata(dev);
+
+   return sysfs_emit(buf, "%d\n", hba->vps->wb_batched_flush);
+}
+
+static ssize_t wb_batched_flush_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+   struct ufs_hba *hba = dev_get_drvdata(dev);
+   unsigned int wb_batched_flush;
+   ssize_t res;
+
+   if (!ufshcd_is_wb_allowed(hba)) {
+   dev_warn(dev, "To control WB through wb_batched_flush is not 
allowed!\n");
+   return -EOPNOTSUPP;
+   }
+
+   if (kstrtouint(buf, 0, _batched_flush))
+   return -EINVAL;
+
+   if (wb_batched_flush != 0 && wb_batched_flush != 1)
+   return -EINVAL;
+
+   down(>host_sem);
+   if (!ufshcd_is_user_access_allowed(hba)) {
+   res = -EBUSY;
+   goto out;
+   }
+
+   if (wb_batched_flush == hba->vps->wb_batched_flush)
+   goto out;
+
+   pm_runtime_get_sync(hba->dev);
+   res = ufshcd_wb_toggle_flush_during_h8(hba, !wb_batched_flush);
+   pm_runtime_put_sync(hba->dev);
+   if (!res)
+   hba->vps->wb_batched_flush = wb_batched_flush;
+
+out:
+   up(>host_sem);
+   return res < 0 ? res : count;
+}
+
 static DEVICE_ATTR_RW(rpm_lvl);
 static DEVICE_ATTR_RO(rpm_target_dev_state);
 static DEVICE_ATTR_RO(rpm_target_link_state);
@@ -261,6 +309,7 @@ static DEVICE_ATTR_RO(spm_target_dev_state);
 static DEVICE_ATTR_RO(spm_target_link_state);
 static DEVICE_ATTR_RW(auto_hibern8);
 static DEVICE_ATTR_RW(wb_on);
+static DEVICE_ATTR_RW(wb_batched_flush);
 
 static struct attribute *ufs_sysfs_ufshcd_attrs[] = {
_attr_rpm_lvl.attr,
@@ -271,6 +320,7 @@ static struct attribute *ufs_sysfs_ufshcd_attrs[] = {
_attr_spm_target_link_state.attr,
_attr_auto_hibern8.attr,
_attr_wb_on.attr,
+   _attr_wb_batched_flush.attr,
NULL
 };
 
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 0625da7a42ee..e11dc578a17c 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -244,7 +244,6 @@ static int ufshcd_setup_vreg(struct ufs_hba

[PATCH] scsi: ufs: Add batched WB buffer flush

2021-04-19 Thread Daejun Park

Currently, WriteBooster (WB) buffer is always flushed during hibern8. However,
this is inefficient because data in the WB buffer can be invalid due to
spatial locality of IO workload.
If the WB buffer flush is flushed in a batched manner, the amount of data
migration and power consumption can be reduced because the overwritten data
of the WB buffer may be invalid due to spatial locality.

This patch supports batched flush of WB buffer. When batched flush is enabled,
fWriteBoosterBufferFlushDuringHibernate is set only when
b_rpm_dev_flush_capable is true during runtime suspend. When the device is
resumed, fWriteBoosterBufferFlushDuringHibernate is cleared to stop flush
during hibern8.

Co-developed-by: Keoseong Park 
Signed-off-by: Keoseong Park 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  9 
 drivers/scsi/ufs/ufs-sysfs.c   | 50 ++
 drivers/scsi/ufs/ufshcd.c  | 14 --
 drivers/scsi/ufs/ufshcd.h  |  2 +
 4 files changed, 71 insertions(+), 4 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..b67b8449e840 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,12 @@ Description: This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/wb_batched_flush
+Date:  April 2021
+Contact:   Daejun Park 
+Description:   This entry shows whether batch flushing of UFS WriteBooster
+   buffers is enabled. Writing 1 to this entry allows the device 
to flush
+   the WriteBooster buffer only when it needs to perform a buffer 
flush
+   during runtime suspend. Writing 0 to this entry allows the 
device to
+   flush the WriteBooster buffer during link hibernation.
diff --git a/drivers/scsi/ufs/ufs-sysfs.c b/drivers/scsi/ufs/ufs-sysfs.c
index d7c3cff9662f..943ac12e59c6 100644
--- a/drivers/scsi/ufs/ufs-sysfs.c
+++ b/drivers/scsi/ufs/ufs-sysfs.c
@@ -253,6 +253,54 @@ static ssize_t wb_on_store(struct device *dev, struct 
device_attribute *attr,
return res < 0 ? res : count;
 }
 
+
+static ssize_t wb_batched_flush_show(struct device *dev,
+struct device_attribute *attr, char *buf)
+{
+   struct ufs_hba *hba = dev_get_drvdata(dev);
+
+   return sysfs_emit(buf, "%d\n", hba->vps->wb_batched_flush);
+}
+
+static ssize_t wb_batched_flush_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+   struct ufs_hba *hba = dev_get_drvdata(dev);
+   unsigned int wb_batched_flush;
+   ssize_t res;
+
+   if (!ufshcd_is_wb_allowed(hba)) {
+   dev_warn(dev, "To control WB through wb_batched_flush is not 
allowed!\n");
+   return -EOPNOTSUPP;
+   }
+
+   if (kstrtouint(buf, 0, _batched_flush))
+   return -EINVAL;
+
+   if (wb_batched_flush != 0 && wb_batched_flush != 1)
+   return -EINVAL;
+
+   down(>host_sem);
+   if (!ufshcd_is_user_access_allowed(hba)) {
+   res = -EBUSY;
+   goto out;
+   }
+
+   if (wb_batched_flush == hba->vps->wb_batched_flush)
+   goto out;
+
+   pm_runtime_get_sync(hba->dev);
+   res = ufshcd_wb_toggle_flush_during_h8(hba, !wb_batched_flush);
+   pm_runtime_put_sync(hba->dev);
+   if (!res)
+   hba->vps->wb_batched_flush = wb_batched_flush;
+
+out:
+   up(>host_sem);
+   return res < 0 ? res : count;
+}
+
 static DEVICE_ATTR_RW(rpm_lvl);
 static DEVICE_ATTR_RO(rpm_target_dev_state);
 static DEVICE_ATTR_RO(rpm_target_link_state);
@@ -261,6 +309,7 @@ static DEVICE_ATTR_RO(spm_target_dev_state);
 static DEVICE_ATTR_RO(spm_target_link_state);
 static DEVICE_ATTR_RW(auto_hibern8);
 static DEVICE_ATTR_RW(wb_on);
+static DEVICE_ATTR_RW(wb_batched_flush);
 
 static struct attribute *ufs_sysfs_ufshcd_attrs[] = {
_attr_rpm_lvl.attr,
@@ -271,6 +320,7 @@ static struct attribute *ufs_sysfs_ufshcd_attrs[] = {
_attr_spm_target_link_state.attr,
_attr_auto_hibern8.attr,
_attr_wb_on.attr,
+   _attr_wb_batched_flush.attr,
NULL
 };
 
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 0625da7a42ee..e11dc578a17c 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -244,7 +244,6 @@ static int ufshcd_setup_vreg(struct ufs_hba *hba, bool on

RE: [PATCH v17 1/2] scsi: ufs: Enable power management for wlun

2021-04-08 Thread Daejun Park

Hi Asutosh Das,

>During runtime-suspend of ufs host, the scsi devices are
>already suspended and so are the queues associated with them.
>But the ufs host sends SSU (START_STOP_UNIT) to wlun
>during its runtime-suspend.
>During the process blk_queue_enter checks if the queue is not in
>suspended state. If so, it waits for the queue to resume, and never
>comes out of it.
>The commit
>(d55d15a33: scsi: block: Do not accept any requests while suspended)
>adds the check if the queue is in suspended state in blk_queue_enter().
> 
>Call trace:
> __switch_to+0x174/0x2c4
> __schedule+0x478/0x764
> schedule+0x9c/0xe0
> blk_queue_enter+0x158/0x228
> blk_mq_alloc_request+0x40/0xa4
> blk_get_request+0x2c/0x70
> __scsi_execute+0x60/0x1c4
> ufshcd_set_dev_pwr_mode+0x124/0x1e4
> ufshcd_suspend+0x208/0x83c
> ufshcd_runtime_suspend+0x40/0x154
> ufshcd_pltfrm_runtime_suspend+0x14/0x20
> pm_generic_runtime_suspend+0x28/0x3c
> __rpm_callback+0x80/0x2a4
> rpm_suspend+0x308/0x614
> rpm_idle+0x158/0x228
> pm_runtime_work+0x84/0xac
> process_one_work+0x1f0/0x470
> worker_thread+0x26c/0x4c8
> kthread+0x13c/0x320
> ret_from_fork+0x10/0x18
> 
>Fix this by registering ufs device wlun as a scsi driver and
>registering it for block runtime-pm. Also make this as a
>supplier for all other luns. That way, this device wlun
>suspends after all the consumers and resumes after
>hba resumes.
> 
>Co-developed-by: Can Guo 
>Signed-off-by: Can Guo 
>Signed-off-by: Asutosh Das 
>---
> drivers/scsi/ufs/cdns-pltfrm.c |   2 +
> drivers/scsi/ufs/tc-dwc-g210-pci.c |   2 +
> drivers/scsi/ufs/ufs-debugfs.c |   6 +-
> drivers/scsi/ufs/ufs-debugfs.h |   2 +-
> drivers/scsi/ufs/ufs-exynos.c  |   2 +
> drivers/scsi/ufs/ufs-hisi.c|   2 +
> drivers/scsi/ufs/ufs-mediatek.c|  12 +-
> drivers/scsi/ufs/ufs-qcom.c|   2 +
> drivers/scsi/ufs/ufs_bsg.c |   6 +-
> drivers/scsi/ufs/ufshcd-pci.c  |  36 +--
> drivers/scsi/ufs/ufshcd.c  | 642 ++---
> drivers/scsi/ufs/ufshcd.h  |   6 +
> include/trace/events/ufs.h |  20 ++
> 13 files changed, 509 insertions(+), 231 deletions(-)

In this patch, you changed pm_runtime_{get, put}_sync to scsi_autopm_{get, 
put}_device.
But, scsi_autopm_get_device() calls pm_runtime_put_sync() in case of error
of pm_runtime_get_sync(). So, pm_runtime_put_sync() can be called twice if
scsi_autopm_get_device has error.

Thanks,
Daejun

RE: [PATCH v16 1/2] scsi: ufs: Enable power management for wlun

2021-04-07 Thread Daejun Park

Hi Asutosh Das,

>+static inline bool is_rpmb_wlun(struct scsi_device *sdev)
>+{
>+return (sdev->lun == 
>ufshcd_upiu_wlun_to_scsi_wlun(UFS_UPIU_RPMB_WLUN));
>+}
>+
>+static inline bool is_device_wlun(struct scsi_device *sdev)
>+{
>+return (sdev->lun ==
>+ufshcd_upiu_wlun_to_scsi_wlun(UFS_UPIU_UFS_DEVICE_WLUN));
>+}
>+
> static void ufshcd_init_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrb, int 
> i)
> {
> struct utp_transfer_cmd_desc *cmd_descp = hba->ucdl_base_addr;
>@@ -4099,11 +4113,11 @@ void ufshcd_auto_hibern8_update(struct ufs_hba *hba, 
>u32 ahit)
> spin_unlock_irqrestore(hba->host->host_lock, flags);
> 
> if (update && !pm_runtime_suspended(hba->dev)) {

Could it be changed hba->sdev_ufs_device->sdev_gendev instead of hba->dev?

Thanks,
Daejun

RE: Re: [PATCH 1/2] scsi: ufs: Introduce hba performance monitor sysfs nodes

2021-04-06 Thread Daejun Park

Hi Can Guo,
> 
>Hi Daejun,
> 
>On 2021-04-06 12:11, Daejun Park wrote:
>> Hi Can Guo,
>> 
>>> +static ssize_t monitor_enable_store(struct device *dev,
>>> +struct device_attribute *attr,
>>> +const char *buf, size_t count)
>>> +{
>>> +struct ufs_hba *hba = dev_get_drvdata(dev);
>>> +unsigned long value, flags;
>>> +
>>> +if (kstrtoul(buf, 0, ))
>>> +return -EINVAL;
>>> +
>>> +value = !!value;
>>> +spin_lock_irqsave(hba->host->host_lock, flags);
>>> +if (value == hba->monitor.enabled)
>>> +goto out_unlock;
>>> +
>>> +if (!value) {
>>> +memset(>monitor, 0, sizeof(hba->monitor));
>>> +} else {
>>> +hba->monitor.enabled = true;
>>> +hba->monitor.enabled_ts = ktime_get();
>> 
>> How about setting lat_max to and lat_min to KTIME_MAX and 0?
> 
>lat_min is already 0. What is the benefit of setting lat_max to 
>KTIME_MAX?
> 
>> I think lat_sum should be 0 at this point.
> 
>lat_sum is already 0 at this point, what is the problem?

Sorry. I misunderstood about resetting monitor values.

> 
>> 
>>> +}
>>> +
>>> +out_unlock:
>>> +spin_unlock_irqrestore(hba->host->host_lock, flags);
>>> +return count;
>>> +}
>> 
>> 
>>> +static void ufshcd_update_monitor(struct ufs_hba *hba, struct 
>>> ufshcd_lrb *lrbp)
>>> +{
>>> +int dir = ufshcd_monitor_opcode2dir(*lrbp->cmd->cmnd);
>>> +
>>> +if (dir >= 0 && hba->monitor.nr_queued[dir] > 0) {
>>> +struct request *req = lrbp->cmd->request;
>>> +struct ufs_hba_monitor *m = >monitor;
>>> +ktime_t now, inc, lat;
>>> +
>>> +now = ktime_get();
>> 
>> How about using lrbp->compl_time_stamp instead of getting new value?
> 
>I am expecting "now" keeps increasing and use it to update 
>m->busy_start_s,
>but if I use lrbp->compl_time_stamp to do that, below line ktime_sub() 
>may
>give me an unexpected value as lrbp->compl_time_stamp may be smaller 
>than
>m->busy_start_ts, because the actual requests are not completed by the 
>device
>in the exact same ordering as the bits set in hba->outstanding_tasks, 
>but driver
>is completing them from bit 0 to bit 31 in ascending order.

lrbp->compl_time_stamp is set just before calling ufshcd_update_monitor().
And I don't think it can be negative value, because ufshcd_send_command()
and __ufshcd_transfer_req_compl() are protected by host lock.

> 
>> 
>>> +inc = ktime_sub(now, m->busy_start_ts[dir]);
>>> +m->total_busy[dir] = ktime_add(m->total_busy[dir], 
>>> inc);
>>> +m->nr_sec_rw[dir] += blk_rq_sectors(req);
>>> +
>>> +/* Update latencies */
>>> +m->nr_req[dir]++;
>>> +lat = ktime_sub(now, lrbp->issue_time_stamp);
>>> +m->lat_sum[dir] += lat;
>>> +if (m->lat_max[dir] < lat || !m->lat_max[dir])
>>> +m->lat_max[dir] = lat;
>>> +if (m->lat_min[dir] > lat || !m->lat_min[dir])
>>> +m->lat_min[dir] = lat;
>> 
>> This if statement can be shorted, by setting lat_max / lat_min as 
>> default value.
> 
>I don't quite get it, can you show me the code sample?

I think " || !m->lat_max[dir]" can be removed.

if (m->lat_max[dir] < lat)
m->lat_max[dir] = lat;
if (m->lat_min[dir] > lat)
m->lat_min[dir] = lat;

Thanks,
Daejun

> 
>Thanks,
>Can Guo
> 
>> 
>>> +
>>> +m->nr_queued[dir]--;
>>> +/* Push forward the busy start of monitor */
>>> +m->busy_start_ts[dir] = now;
>>> +}
>>> +}
>> 
>> Thanks,
>> Daejun

RE: [PATCH 1/2] scsi: ufs: Introduce hba performance monitor sysfs nodes

2021-04-05 Thread Daejun Park

Hi Can Guo,

> +static ssize_t monitor_enable_store(struct device *dev,
> +struct device_attribute *attr,
> +const char *buf, size_t count)
> +{
> +struct ufs_hba *hba = dev_get_drvdata(dev);
> +unsigned long value, flags;
> +
> +if (kstrtoul(buf, 0, ))
> +return -EINVAL;
> +
> +value = !!value;
> +spin_lock_irqsave(hba->host->host_lock, flags);
> +if (value == hba->monitor.enabled)
> +goto out_unlock;
> +
> +if (!value) {
> +memset(>monitor, 0, sizeof(hba->monitor));
> +} else {
> +hba->monitor.enabled = true;
> +hba->monitor.enabled_ts = ktime_get();

How about setting lat_max to and lat_min to KTIME_MAX and 0?
I think lat_sum should be 0 at this point.

> +}
> +
> +out_unlock:
> +spin_unlock_irqrestore(hba->host->host_lock, flags);
> +return count;
> +}


> +static void ufshcd_update_monitor(struct ufs_hba *hba, struct ufshcd_lrb 
> *lrbp)
> +{
> +int dir = ufshcd_monitor_opcode2dir(*lrbp->cmd->cmnd);
> +
> +if (dir >= 0 && hba->monitor.nr_queued[dir] > 0) {
> +struct request *req = lrbp->cmd->request;
> +struct ufs_hba_monitor *m = >monitor;
> +ktime_t now, inc, lat;
> +
> +now = ktime_get();

How about using lrbp->compl_time_stamp instead of getting new value?

> +inc = ktime_sub(now, m->busy_start_ts[dir]);
> +m->total_busy[dir] = ktime_add(m->total_busy[dir], inc);
> +m->nr_sec_rw[dir] += blk_rq_sectors(req);
> +
> +/* Update latencies */
> +m->nr_req[dir]++;
> +lat = ktime_sub(now, lrbp->issue_time_stamp);
> +m->lat_sum[dir] += lat;
> +if (m->lat_max[dir] < lat || !m->lat_max[dir])
> +m->lat_max[dir] = lat;
> +if (m->lat_min[dir] > lat || !m->lat_min[dir])
> +m->lat_min[dir] = lat;

This if statement can be shorted, by setting lat_max / lat_min as default value.

> +
> +m->nr_queued[dir]--;
> +/* Push forward the busy start of monitor */
> +m->busy_start_ts[dir] = now;
> +}
> +}

Thanks,
Daejun

RE: [PATCH v7 00/11] Add Host control mode to HPB

2021-04-05 Thread Daejun Park

> The HPB spec defines 2 control modes - device control mode and host
> control mode. In oppose to device control mode, in which the host obey
> to whatever recommendation received from the device - In host control
> mode, the host uses its own algorithms to decide which regions should
> be activated or inactivated.
>  
> We kept the host managed heuristic simple and concise.
>  
> Aside from adding a by-spec functionality, host control mode entails
> some further potential benefits: makes the hpb logic transparent and
> readable, while allow tuning / scaling its various parameters, and
> utilize system-wide info to optimize HPB potential.
>  
> This series is based on Samsung's V32 device-control HPB2.0 driver
>  
> This version was tested on Galaxy S20, and Xiaomi Mi10 pro.
> Your meticulous review and testing is mostly welcome and appreciated.
>  
> Thanks,
> Avri
>  
>  
> Avri Altman (11):
>   scsi: ufshpb: Cache HPB Control mode on init
>   scsi: ufshpb: Add host control mode support to rsp_upiu
>   scsi: ufshpb: Transform set_dirty to iterate_rgn
>   scsi: ufshpb: Add reads counter
>   scsi: ufshpb: Make eviction depends on region's reads
>   scsi: ufshpb: Region inactivation in host mode
>   scsi: ufshpb: Add hpb dev reset response
>   scsi: ufshpb: Add "Cold" regions timer
>   scsi: ufshpb: Limit the number of inflight map requests
>   scsi: ufshpb: Add support for host control mode
>   scsi: ufshpb: Make host mode parameters configurable
>  
>  Documentation/ABI/testing/sysfs-driver-ufs |  84 ++-
>  drivers/scsi/ufs/ufshcd.h  |   2 +
>  drivers/scsi/ufs/ufshpb.c  | 568 -
>  drivers/scsi/ufs/ufshpb.h  |  44 ++
>  4 files changed, 663 insertions(+), 35 deletions(-)
>  

The patches in this series look good to me.

Reviewed-by: Daejun Park

RE: [PATCH v4 2/2] scsi: ufs: Fix wrong Task Tag used in task management request UPIUs

2021-04-01 Thread Daejun Park

Hi, Can Guo

> diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
...
>  
>   req->end_io_data = 
> - free_slot = req->tag;
>   WARN_ON_ONCE(free_slot < 0 || free_slot >= hba->nutmrs);
I think this line should be removed.

Thanks,
Daejun

[PATCH v32 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-30 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Reviewed-by: Can Guo 
Reviewed-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  47 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   4 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  25 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 623 +++--
 drivers/scsi/ufs/ufshpb.h  |  67 ++-
 7 files changed, 696 insertions(+), 80 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index 528bf89fc98b..419adf450b89 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,45 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/flags/wb_enable
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the status of HPB.
+
+   == 
+   0  HPB is not enabled.
+   1  HPB is enab

[PATCH v32 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-03-30 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Reviewed-by: Bean Huo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 257 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 258 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index f4443a7b252b..2145360d74da 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2645,6 +2645,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index c67acfc8c6bf..f789339f68d9 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,234 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+

[PATCH v32 2/4] scsi: ufs: L2P map management for HPB read

2021-03-30 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Reviewed-by: Bean Huo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1088 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1178 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 4eee7e31d08d..bfb84d2ba990 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -478,6 +478,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -488,6 +523,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 045a14843c14..f4443a7b252b 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5010,6 +5010,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9214,6 +9217,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 1a72f6541510..c67acfc8c6bf 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_activ

[PATCH v32 1/4] scsi: ufs: Introduce HPB feature

2021-03-30 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Reviewed-by: Bean Huo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 167 ++
 9 files changed, 977 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..528bf89fc98b 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control

[PATCH v32 0/4] scsi: ufs: Add Host Performance Booster Support

2021-03-30 Thread Daejun Park

n process
2) READ -> HPB READ using cached map information
3) L2P (logical to physical) map management

In the HPB probe and init process, the device information of the UFS is
queried. After checking supported features, the data structure for the HPB
is initialized according to the device information.

A read I/O in the active sub-region where the map is cached is changed to
HPB READ by the HPB.

The HPB manages the L2P map using information received from the
device. For active sub-region, the HPB caches through ufshpb_map
request. For the in-active region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

 Documentation/ABI/testing/sysfs-driver-ufs |  162 ++
 drivers/scsi/ufs/Kconfig   |9 +
 drivers/scsi/ufs/Makefile  |1 +
 drivers/scsi/ufs/ufs-sysfs.c   |   22 +
 drivers/scsi/ufs/ufs.h |   54 +-
 drivers/scsi/ufs/ufshcd.c  |   74 +-
 drivers/scsi/ufs/ufshcd.h  |   29 +
 drivers/scsi/ufs/ufshpb.c  | 2387 
 drivers/scsi/ufs/ufshpb.h  |  277 +++
 9 files changed, 3013 insertions(+), 2 deletions(-)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

-- 
2.25.1

RE: Re: [PATCH v31 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-25 Thread Daejun Park

> On 2021-03-25 08:47, Daejun Park wrote:
> >> On 2021-03-22 14:55, Daejun Park wrote:
> >> > This patch supports the HPB 2.0.
> >> >
> >> > The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
> >> > In the case of Read (<= 32KB) is supported as single HPB read.
> >> > In the case of Read (36KB ~ 512KB) is supported by as a combination of
> >> > write buffer command and HPB read command to deliver more PPN.
> >> > The write buffer commands may not be issued immediately due to busy
> >> > tags.
> >> > To use HPB read more aggressively, the driver can requeue the write
> >> > buffer
> >> > command. The requeue threshold is implemented as timeout and can be
> >> > modified with requeue_timeout_ms entry in sysfs.
> >> >
> >> > Signed-off-by: Daejun Park 
> >> > ---
> >> >  Documentation/ABI/testing/sysfs-driver-ufs |  47 +-
> >> >  drivers/scsi/ufs/ufs-sysfs.c   |   4 +
> >> >  drivers/scsi/ufs/ufs.h |   3 +-
> >> >  drivers/scsi/ufs/ufshcd.c  |  25 +-
> >> >  drivers/scsi/ufs/ufshcd.h  |   7 +
> >> >  drivers/scsi/ufs/ufshpb.c  | 626 +++--
> >> >  drivers/scsi/ufs/ufshpb.h  |  67 ++-
> >> >  7 files changed, 698 insertions(+), 81 deletions(-)
> >> >
> >> > diff --git a/Documentation/ABI/testing/sysfs-driver-ufs
> >> > b/Documentation/ABI/testing/sysfs-driver-ufs
> >> > index 528bf89fc98b..419adf450b89 100644
> >> > --- a/Documentation/ABI/testing/sysfs-driver-ufs
> >> > +++ b/Documentation/ABI/testing/sysfs-driver-ufs
> >> > @@ -1253,14 +1253,14 @@ Description:    This entry shows the number 
> >> > of
> >> > HPB pinned regions assigned to
> >> >
> >> >  The file is read only.
> >> >
> >> > -What:/sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
> >> > +What:
> >> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
> >> >  Date:March 2021
> >> >  Contact:Daejun Park 
> >> >  Description:This entry shows the number of reads that changed 
> >> > to HPB
> >> > read.
> >> >
> >> >  The file is read only.
> >> >
> >> > -What:/sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
> >> > +What:
> >> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
> >> >  Date:March 2021
> >> >  Contact:Daejun Park 
> >> >  Description:This entry shows the number of reads that cannot be
> >> > changed to
> >> > @@ -1268,7 +1268,7 @@ Description:This entry shows the number of
> >> > reads that cannot be changed to
> >> >
> >> >  The file is read only.
> >> >
> >> > -What:
> >> > /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
> >> > +What:
> >> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
> >> >  Date:March 2021
> >> >  Contact:Daejun Park 
> >> >  Description:This entry shows the number of response UPIUs that 
> >> > has
> >> > @@ -1276,7 +1276,7 @@ Description:This entry shows the number of
> >> > response UPIUs that has
> >> >
> >> >      The file is read only.
> >> >
> >> > -What:
> >> > /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
> >> > +What:
> >> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
> >> >  Date:March 2021
> >> >  Contact:Daejun Park 
> >> >  Description:This entry shows the number of active sub-regions
> >> > recommended by
> >> > @@ -1284,7 +1284,7 @@ Description:This entry shows the number of
> >> > active sub-regions recommended by
> >> >
> >> >  The file is read only.
> >> >
> >> > -What:
> >> > /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
> >> > +What:
> >> > /sys/class/sc

RE: Re: [PATCH v31 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-24 Thread Daejun Park

> On 2021-03-22 14:55, Daejun Park wrote:
> > This patch supports the HPB 2.0.
> > 
> > The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
> > In the case of Read (<= 32KB) is supported as single HPB read.
> > In the case of Read (36KB ~ 512KB) is supported by as a combination of
> > write buffer command and HPB read command to deliver more PPN.
> > The write buffer commands may not be issued immediately due to busy 
> > tags.
> > To use HPB read more aggressively, the driver can requeue the write 
> > buffer
> > command. The requeue threshold is implemented as timeout and can be
> > modified with requeue_timeout_ms entry in sysfs.
> > 
> > Signed-off-by: Daejun Park 
> > ---
> >  Documentation/ABI/testing/sysfs-driver-ufs |  47 +-
> >  drivers/scsi/ufs/ufs-sysfs.c   |   4 +
> >  drivers/scsi/ufs/ufs.h |   3 +-
> >  drivers/scsi/ufs/ufshcd.c  |  25 +-
> >  drivers/scsi/ufs/ufshcd.h  |   7 +
> >  drivers/scsi/ufs/ufshpb.c  | 626 +++--
> >  drivers/scsi/ufs/ufshpb.h  |  67 ++-
> >  7 files changed, 698 insertions(+), 81 deletions(-)
> > 
> > diff --git a/Documentation/ABI/testing/sysfs-driver-ufs
> > b/Documentation/ABI/testing/sysfs-driver-ufs
> > index 528bf89fc98b..419adf450b89 100644
> > --- a/Documentation/ABI/testing/sysfs-driver-ufs
> > +++ b/Documentation/ABI/testing/sysfs-driver-ufs
> > @@ -1253,14 +1253,14 @@ Description:This entry shows the number of
> > HPB pinned regions assigned to
> > 
> >  The file is read only.
> > 
> > -What:/sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
> > +What:/sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
> >  Date:March 2021
> >  Contact:Daejun Park 
> >  Description:This entry shows the number of reads that changed to 
> > HPB 
> > read.
> > 
> >      The file is read only.
> > 
> > -What:/sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
> > +What:
> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
> >  Date:March 2021
> >  Contact:Daejun Park 
> >  Description:This entry shows the number of reads that cannot be 
> > changed to
> > @@ -1268,7 +1268,7 @@ Description:This entry shows the number of
> > reads that cannot be changed to
> > 
> >  The file is read only.
> > 
> > -What:/sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
> > +What:
> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
> >  Date:March 2021
> >  Contact:Daejun Park 
> >  Description:This entry shows the number of response UPIUs that has
> > @@ -1276,7 +1276,7 @@ Description:This entry shows the number of
> > response UPIUs that has
> > 
> >  The file is read only.
> > 
> > -What:
> > /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
> > +What:
> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
> >  Date:March 2021
> >  Contact:Daejun Park 
> >  Description:This entry shows the number of active sub-regions 
> > recommended by
> > @@ -1284,7 +1284,7 @@ Description:This entry shows the number of
> > active sub-regions recommended by
> > 
> >  The file is read only.
> > 
> > -What:
> > /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
> > +What:
> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
> >  Date:March 2021
> >  Contact:Daejun Park 
> >  Description:This entry shows the number of inactive regions 
> > recommended by
> > @@ -1292,10 +1292,45 @@ Description:This entry shows the number of
> > inactive regions recommended by
> > 
> >  The file is read only.
> > 
> > -What:/sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
> > +What:
> > /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
> >  Date:March 2021
> >  Contact:Daejun Park 
> >  Description:This entry shows the number of read buffer commands for
> >  activating sub-regions recommended by response UPIUs.

RE: Re: [PATCH v31 2/4] scsi: ufs: L2P map management for HPB read

2021-03-23 Thread Daejun Park

>On 2021-03-23 14:19, Daejun Park wrote:
>>> On 2021-03-23 13:37, Daejun Park wrote:
>>>>> On 2021-03-23 12:22, Can Guo wrote:
>>>>>> On 2021-03-22 17:11, Bean Huo wrote:
>>>>>>> On Mon, 2021-03-22 at 15:54 +0900, Daejun Park wrote:
>>>>>>>> +   switch (rsp_field->hpb_op) {
>>>>>>>> 
>>>>>>>> +   case HPB_RSP_REQ_REGION_UPDATE:
>>>>>>>> 
>>>>>>>> +   if (data_seg_len != DEV_DATA_SEG_LEN)
>>>>>>>> 
>>>>>>>> +   dev_warn(>sdev_ufs_lu->sdev_dev,
>>>>>>>> 
>>>>>>>> +"%s: data seg length is not
>>>>>>>> same.\n",
>>>>>>>> 
>>>>>>>> +__func__);
>>>>>>>> 
>>>>>>>> +   ufshpb_rsp_req_region_update(hpb, rsp_field);
>>>>>>>> 
>>>>>>>> +   break;
>>>>>>>> 
>>>>>>>> +   case HPB_RSP_DEV_RESET:
>>>>>>>> 
>>>>>>>> +   dev_warn(>sdev_ufs_lu->sdev_dev,
>>>>>>>> 
>>>>>>>> +"UFS device lost HPB information during
>>>>>>>> PM.\n");
>>>>>>>> 
>>>>>>>> +   break;
>>>>>>> 
>>>>>>> Hi Deajun,
>>>>>>> This series looks good to me. Just here I have one question. You
>>>>>>> didn't
>>>>>>> handle HPB_RSP_DEV_RESET, just a warning.  Based on your SS UFS, 
>>>>>>> how
>>>>>>> to
>>>>>>> handle HPB_RSP_DEV_RESET from the host side? Do you think we shoud
>>>>>>> reset host side HPB entry as well or what else?
>>>>>>> 
>>>>>>> 
>>>>>>> Bean
>>>>>> 
>>>>>> Same question here - I am still collecting feedbacks from flash
>>>>>> vendors
>>>>>> about
>>>>>> what is recommanded host behavior on reception of HPB Op code 0x2,
>>>>>> since it
>>>>>> is not cleared defined in HPB2.0 specs.
>>>>>> 
>>>>>> Can Guo.
>>>>> 
>>>>> I think the question should be asked in the HPB2.0 patch, since in
>>>>> HPB1.0 device
>>>>> control mode, a HPB reset in device side does not impact anything in
>>>>> host side -
>>>>> host is not writing back any HPB entries to device anyways and HPB
>>>>> Read
>>>>> cmd with
>>>>> invalid HPB entries shall be treated as normal Read(10) cmd without
>>>>> any
>>>>> problems.
>>>> 
>>>> Yes, UFS device will process read command even the HPB entries are
>>>> valid or
>>>> not. So it is warning about read performance drop by dev reset.
>>> 
>>> Yeah, but still I am 100% sure about what should host do in case of
>>> HPB2.0
>>> when it receives HPB Op code 0x2, I am waiting for feedbacks.
>> 
>> I think the host has two choices when it receives 0x2.
>> One is nothing on host.
>> The other is discarding all HPB entries in the host.
>> 
>> In the JEDEC HPB spec, it as follows:
>> When the device is powered off by the host, the device may restore L2P 
>> map
>> data upon power up or build from the host’s HPB READ command.
>> 
>> If some UFS builds L2P map data from the host's HPB READ commands, we 
>> don't
>> have to discard HPB entries in the host.
>> 
>> So I thinks there is nothing to do when it receives 0x2.
> 
>But in HPB2.0, if we do nothing to active regions in host side, host can 
>write
>HPB entries (which host thinks valid, but actually invalid in device 
>side since
>reset happened) back to device through HPB Write Buffer cmds (BUFFER ID 
>= 0x2).
>My question is that are all UFSs OK with this?

Yes, it must be OK.

Please refer the following the HPB 2.0 spec:

If the HPB Entries sent by HPB WRITE BUFFER are removed by the device,
for example, because they are not consumed for a long enough period of time,
then the HPB READ command for the removed HPB entries shall be handled as a
normal READ command.

Thanks,
Daejun

>Thanks,
>Can Guo.
> 
>> 
>> Thanks,
>> Daejun
>> 
>>> Thanks,
>>> Can Guo.
>>> 
>>>> 
>>>> Thanks,
>>>> Daejun
>>>> 
>>>>> Please correct me if I am wrong.
>>>> 
>>>> 
>>>> 
>>>>> Thanks,
>>>>> Can Guo.
>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>> 
> 
> 
>

RE: Re: [PATCH v31 2/4] scsi: ufs: L2P map management for HPB read

2021-03-23 Thread Daejun Park

>On 2021-03-23 13:37, Daejun Park wrote:
>>> On 2021-03-23 12:22, Can Guo wrote:
>>>> On 2021-03-22 17:11, Bean Huo wrote:
>>>>> On Mon, 2021-03-22 at 15:54 +0900, Daejun Park wrote:
>>>>>> +   switch (rsp_field->hpb_op) {
>>>>>> 
>>>>>> +   case HPB_RSP_REQ_REGION_UPDATE:
>>>>>> 
>>>>>> +   if (data_seg_len != DEV_DATA_SEG_LEN)
>>>>>> 
>>>>>> +   dev_warn(>sdev_ufs_lu->sdev_dev,
>>>>>> 
>>>>>> +"%s: data seg length is not
>>>>>> same.\n",
>>>>>> 
>>>>>> +__func__);
>>>>>> 
>>>>>> +   ufshpb_rsp_req_region_update(hpb, rsp_field);
>>>>>> 
>>>>>> +   break;
>>>>>> 
>>>>>> +   case HPB_RSP_DEV_RESET:
>>>>>> 
>>>>>> +   dev_warn(>sdev_ufs_lu->sdev_dev,
>>>>>> 
>>>>>> +"UFS device lost HPB information during
>>>>>> PM.\n");
>>>>>> 
>>>>>> +   break;
>>>>> 
>>>>> Hi Deajun,
>>>>> This series looks good to me. Just here I have one question. You
>>>>> didn't
>>>>> handle HPB_RSP_DEV_RESET, just a warning.  Based on your SS UFS, how
>>>>> to
>>>>> handle HPB_RSP_DEV_RESET from the host side? Do you think we shoud
>>>>> reset host side HPB entry as well or what else?
>>>>> 
>>>>> 
>>>>> Bean
>>>> 
>>>> Same question here - I am still collecting feedbacks from flash 
>>>> vendors
>>>> about
>>>> what is recommanded host behavior on reception of HPB Op code 0x2,
>>>> since it
>>>> is not cleared defined in HPB2.0 specs.
>>>> 
>>>> Can Guo.
>>> 
>>> I think the question should be asked in the HPB2.0 patch, since in
>>> HPB1.0 device
>>> control mode, a HPB reset in device side does not impact anything in
>>> host side -
>>> host is not writing back any HPB entries to device anyways and HPB 
>>> Read
>>> cmd with
>>> invalid HPB entries shall be treated as normal Read(10) cmd without 
>>> any
>>> problems.
>> 
>> Yes, UFS device will process read command even the HPB entries are 
>> valid or
>> not. So it is warning about read performance drop by dev reset.
> 
>Yeah, but still I am 100% sure about what should host do in case of 
>HPB2.0
>when it receives HPB Op code 0x2, I am waiting for feedbacks.

I think the host has two choices when it receives 0x2.
One is nothing on host.
The other is discarding all HPB entries in the host. 

In the JEDEC HPB spec, it as follows:
When the device is powered off by the host, the device may restore L2P map
data upon power up or build from the host’s HPB READ command.

If some UFS builds L2P map data from the host's HPB READ commands, we don't
have to discard HPB entries in the host.

So I thinks there is nothing to do when it receives 0x2.

Thanks,
Daejun

>Thanks,
>Can Guo.
> 
>> 
>> Thanks,
>> Daejun
>> 
>>> Please correct me if I am wrong.
>> 
>> 
>> 
>>> Thanks,
>>> Can Guo.
>>> 
>>> 
>>> 
> 
> 
>

RE: Re: [PATCH v31 2/4] scsi: ufs: L2P map management for HPB read

2021-03-22 Thread Daejun Park

>On 2021-03-23 12:22, Can Guo wrote:
>> On 2021-03-22 17:11, Bean Huo wrote:
>>> On Mon, 2021-03-22 at 15:54 +0900, Daejun Park wrote:
>>>> +   switch (rsp_field->hpb_op) {
>>>> 
>>>> +   case HPB_RSP_REQ_REGION_UPDATE:
>>>> 
>>>> +   if (data_seg_len != DEV_DATA_SEG_LEN)
>>>> 
>>>> +   dev_warn(>sdev_ufs_lu->sdev_dev,
>>>> 
>>>> +"%s: data seg length is not
>>>> same.\n",
>>>> 
>>>> +__func__);
>>>> 
>>>> +   ufshpb_rsp_req_region_update(hpb, rsp_field);
>>>> 
>>>> +   break;
>>>> 
>>>> +   case HPB_RSP_DEV_RESET:
>>>> 
>>>> +   dev_warn(>sdev_ufs_lu->sdev_dev,
>>>> 
>>>> +"UFS device lost HPB information during
>>>> PM.\n");
>>>> 
>>>> +   break;
>>> 
>>> Hi Deajun,
>>> This series looks good to me. Just here I have one question. You 
>>> didn't
>>> handle HPB_RSP_DEV_RESET, just a warning.  Based on your SS UFS, how 
>>> to
>>> handle HPB_RSP_DEV_RESET from the host side? Do you think we shoud
>>> reset host side HPB entry as well or what else?
>>> 
>>> 
>>> Bean
>> 
>> Same question here - I am still collecting feedbacks from flash vendors 
>> about
>> what is recommanded host behavior on reception of HPB Op code 0x2, 
>> since it
>> is not cleared defined in HPB2.0 specs.
>> 
>> Can Guo.
> 
>I think the question should be asked in the HPB2.0 patch, since in 
>HPB1.0 device
>control mode, a HPB reset in device side does not impact anything in 
>host side -
>host is not writing back any HPB entries to device anyways and HPB Read 
>cmd with
>invalid HPB entries shall be treated as normal Read(10) cmd without any 
>problems.

Yes, UFS device will process read command even the HPB entries are valid or
not. So it is warning about read performance drop by dev reset.

Thanks,
Daejun

>Please correct me if I am wrong.



>Thanks,
>Can Guo.
> 
> 
>

RE: Re: [PATCH v31 2/4] scsi: ufs: L2P map management for HPB read

2021-03-22 Thread Daejun Park

Hi Bean,

>On Mon, 2021-03-22 at 15:54 +0900, Daejun Park wrote:
>> +   switch (rsp_field->hpb_op) {
>> 
>> +   case HPB_RSP_REQ_REGION_UPDATE:
>> 
>> +   if (data_seg_len != DEV_DATA_SEG_LEN)
>> 
>> +   dev_warn(>sdev_ufs_lu->sdev_dev,
>> 
>> +"%s: data seg length is not
>> same.\n",
>> 
>> +__func__);
>> 
>> +   ufshpb_rsp_req_region_update(hpb, rsp_field);
>> 
>> +   break;
>> 
>> +   case HPB_RSP_DEV_RESET:
>> 
>> +   dev_warn(>sdev_ufs_lu->sdev_dev,
>> 
>> +"UFS device lost HPB information during
>> PM.\n");
>> 
>> +   break;
> 
>Hi Deajun,
>This series looks good to me. Just here I have one question. You didn't

Thanks.

>handle HPB_RSP_DEV_RESET, just a warning.  Based on your SS UFS, how to
>handle HPB_RSP_DEV_RESET from the host side? Do you think we shoud
>reset host side HPB entry as well or what else?

In the JEDEC HPB spec, it as follows:

When the device is powered off by the host, the device may restore L2P map
data upon power up or build from the host’s HPB READ command.

So I think there is nothing to do, because UFS can build from host's
command. Moreover, in the case of the HPB read with invalid information by
dev reset, it just processed as normal read.

Thanks,
Daejun
> 
> 
>Bean
> 
> 
> 
>

[PATCH v31 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-22 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  47 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   4 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  25 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 626 +++--
 drivers/scsi/ufs/ufshpb.h  |  67 ++-
 7 files changed, 698 insertions(+), 81 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index 528bf89fc98b..419adf450b89 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,45 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/flags/wb_enable
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the status of HPB.
+
+   == 
+   0  HPB is not enabled.
+   1  HPB is enab

[PATCH v31 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-03-22 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 257 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 258 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 88280310bb64..a7cf9278965c 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2653,6 +2653,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index c67acfc8c6bf..f789339f68d9 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,234 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v31 2/4] scsi: ufs: L2P map management for HPB read

2021-03-22 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1088 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1178 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 4eee7e31d08d..bfb84d2ba990 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -478,6 +478,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -488,6 +523,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index ddeb5bb9fb88..88280310bb64 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5018,6 +5018,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9233,6 +9236,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 1a72f6541510..c67acfc8c6bf 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v31 1/4] scsi: ufs: Introduce HPB feature

2021-03-22 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 167 ++
 9 files changed, 977 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..528bf89fc98b 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v31 0/4] scsi: ufs: Add Host Performance Booster Support

2021-03-22 Thread Daejun Park

information
3) L2P (logical to physical) map management

In the HPB probe and init process, the device information of the UFS is
queried. After checking supported features, the data structure for the HPB
is initialized according to the device information.

A read I/O in the active sub-region where the map is cached is changed to
HPB READ by the HPB.

The HPB manages the L2P map using information received from the
device. For active sub-region, the HPB caches through ufshpb_map
request. For the in-active region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

 Documentation/ABI/testing/sysfs-driver-ufs |  162 ++
 drivers/scsi/ufs/Kconfig   |9 +
 drivers/scsi/ufs/Makefile  |1 +
 drivers/scsi/ufs/ufs-sysfs.c   |   22 +
 drivers/scsi/ufs/ufs.h |   54 +-
 drivers/scsi/ufs/ufshcd.c  |   74 +-
 drivers/scsi/ufs/ufshcd.h  |   29 +
 drivers/scsi/ufs/ufshpb.c  | 2388 
 drivers/scsi/ufs/ufshpb.h  |  277 +++
 9 files changed, 3014 insertions(+), 2 deletions(-)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

-- 
2.25.1

[PATCH v30 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-22 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  47 +-
 drivers/scsi/scsi_lib.c|   4 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   4 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  25 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 626 +++--
 drivers/scsi/ufs/ufshpb.h  |  67 ++-
 8 files changed, 701 insertions(+), 82 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index 528bf89fc98b..419adf450b89 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,45 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/flags/wb_enable
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the status of HPB.
+
+   == 
+   0  HPB is not enabled.
+   1  HPB is enab

[PATCH v30 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-03-22 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 257 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 258 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 88280310bb64..a7cf9278965c 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2653,6 +2653,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index c67acfc8c6bf..f789339f68d9 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,234 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v30 2/4] scsi: ufs: L2P map management for HPB read

2021-03-22 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1088 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1178 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 4eee7e31d08d..bfb84d2ba990 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -478,6 +478,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -488,6 +523,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index ddeb5bb9fb88..88280310bb64 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5018,6 +5018,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9233,6 +9236,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 1a72f6541510..c67acfc8c6bf 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v30 1/4] scsi: ufs: Introduce HPB feature

2021-03-22 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 167 ++
 9 files changed, 977 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..528bf89fc98b 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v30 0/4] scsi: ufs: Add Host Performance Booster Support

2021-03-22 Thread Daejun Park

nt

In the HPB probe and init process, the device information of the UFS is
queried. After checking supported features, the data structure for the HPB
is initialized according to the device information.

A read I/O in the active sub-region where the map is cached is changed to
HPB READ by the HPB.

The HPB manages the L2P map using information received from the
device. For active sub-region, the HPB caches through ufshpb_map
request. For the in-active region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

 Documentation/ABI/testing/sysfs-driver-ufs |  162 ++
 drivers/scsi/scsi_lib.c|4 +-
 drivers/scsi/ufs/Kconfig   |9 +
 drivers/scsi/ufs/Makefile  |1 +
 drivers/scsi/ufs/ufs-sysfs.c   |   22 +
 drivers/scsi/ufs/ufs.h |   54 +-
 drivers/scsi/ufs/ufshcd.c  |   74 +-
 drivers/scsi/ufs/ufshcd.h  |   29 +
 drivers/scsi/ufs/ufshpb.c  | 2388 
 drivers/scsi/ufs/ufshpb.h  |  277 +++
 10 files changed, 3017 insertions(+), 3 deletions(-)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

-- 
2.25.1

RE: RE: [PATCH v29 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-21 Thread Daejun Park

Hi Avri,

>> +static int ufshpb_execute_umap_req(struct ufshpb_lu *hpb,
>> +  struct ufshpb_req *umap_req,
>> +  struct ufshpb_region *rgn)
>> +{
>> +   struct request *req;
>> +   struct scsi_request *rq;
>> +
>> +   req = umap_req->req;
>> +   req->timeout = 0;
>> +   req->end_io_data = (void *)umap_req;
>> +   rq = scsi_req(req);
>> +   ufshpb_set_unmap_cmd(rq->cmd, rgn);
>> +   rq->cmd_len = HPB_WRITE_BUFFER_CMD_LENGTH;
>> +
>> +   blk_execute_rq_nowait(NULL, req, 1, ufshpb_umap_req_compl_fn);
>Typo? Forgot the struct request_queue *q?

The argument of q is removed after this patch.

https://lore.kernel.org/linux-scsi/1611550198-17142-1-git-send-email-guoqing.ji...@cloud.ionos.com/#r

Thanks,
Daejun

>> +
>> +   return 0;
>> +}
>> +
>>  static int ufshpb_execute_map_req(struct ufshpb_lu *hpb,
>>   struct ufshpb_req *map_req, bool last)
>>  {
>> @@ -533,12 +878,12 @@ static int ufshpb_execute_map_req(struct ufshpb_lu
>> *hpb,
>> 
>> q = hpb->sdev_ufs_lu->request_queue;
>> for (i = 0; i < hpb->pages_per_srgn; i++) {
>> -   ret = bio_add_pc_page(q, map_req->bio, 
>> map_req->mctx->m_page[i],
>> +   ret = bio_add_pc_page(q, map_req->bio, map_req->rb.mctx-
>> >m_page[i],
>>   PAGE_SIZE, 0);
>> if (ret != PAGE_SIZE) {
>> dev_err(>sdev_ufs_lu->sdev_dev,
>>"bio_add_pc_page fail %d - %d\n",
>> -  map_req->rgn_idx, map_req->srgn_idx);
>> +  map_req->rb.rgn_idx, 
>> map_req->rb.srgn_idx);
>> return ret;
>> }
>> }
>> @@ -554,8 +899,8 @@ static int ufshpb_execute_map_req(struct ufshpb_lu
>> *hpb,
>> if (unlikely(last))
>> mem_size = hpb->last_srgn_entries * HPB_ENTRY_SIZE;
>> 
>> -   ufshpb_set_read_buf_cmd(rq->cmd, map_req->rgn_idx,
>> -   map_req->srgn_idx, mem_size);
>> +   ufshpb_set_read_buf_cmd(rq->cmd, map_req->rb.rgn_idx,
>> +   map_req->rb.srgn_idx, mem_size);
>> rq->cmd_len = HPB_READ_BUFFER_CMD_LENGTH;
>> 
>> blk_execute_rq_nowait(NULL, req, 1, ufshpb_map_req_compl_fn);
>Ditto
> 
> 
>Thanks,
>Avri
> 
> 
>

RE: Re: [PATCH] scsi: ufs: Add selector to ufshcd_query_flag* APIs

2021-03-18 Thread Daejun Park

Hi Martin

> 
>Daejun,
> 
>> Unlike other query APIs in UFS, ufshcd_query_flag has a fixed selector
>> as 0. This patch allows ufshcd_query_flag API to choose selector value
>> by parameter.
> 
>I don't see any users of the added parameter. Am I missing something?

The JEDEC standard describes it as follows:
The selector field may be needed to further identify a particular element of a 
flag.

Other cannot used it because it is fixed as zero in the API.

Thanks,
Daejun

The selector field may be needed to further identify a particular element of a 
flag. Selector field is not
2218 used in this version of the standard and its value shall be zero.
> 
>-- 
>Martin K. PetersenOracle Linux Engineering
> 
> 
>

RE: Re: [PATCH v29 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-18 Thread Daejun Park

>On 2021-03-18 10:02, Daejun Park wrote:
>>> On 2021-03-17 09:42, Daejun Park wrote:
>>>>> On 2021-03-15 15:23, Can Guo wrote:
>>>>>> On 2021-03-15 15:07, Daejun Park wrote:
>>>>>>>>> This patch supports the HPB 2.0.
>>>>>>>>> 
>>>>>>>>> The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
>>>>>>>>> In the case of Read (<= 32KB) is supported as single HPB read.
>>>>>>>>> In the case of Read (36KB ~ 512KB) is supported by as a
>>>>>>>>> combination
>>>>>>>>> of
>>>>>>>>> write buffer command and HPB read command to deliver more PPN.
>>>>>>>>> The write buffer commands may not be issued immediately due to
>>>>>>>>> busy
>>>>>>>>> tags.
>>>>>>>>> To use HPB read more aggressively, the driver can requeue the
>>>>>>>>> write
>>>>>>>>> buffer
>>>>>>>>> command. The requeue threshold is implemented as timeout and can
>>>>>>>>> be
>>>>>>>>> modified with requeue_timeout_ms entry in sysfs.
>>>>>>>>> 
>>>>>>>>> Signed-off-by: Daejun Park 
>>>>>>>>> ---
>>>>>>>>> +static struct attribute *hpb_dev_param_attrs[] = {
>>>>>>>>> +_attr_requeue_timeout_ms.attr,
>>>>>>>>> +NULL,
>>>>>>>>> +};
>>>>>>>>> +
>>>>>>>>> +struct attribute_group ufs_sysfs_hpb_param_group = {
>>>>>>>>> +.name = "hpb_param_sysfs",
>>>>>>>>> +.attrs = hpb_dev_param_attrs,
>>>>>>>>> +};
>>>>>>>>> +
>>>>>>>>> +static int ufshpb_pre_req_mempool_init(struct ufshpb_lu *hpb)
>>>>>>>>> +{
>>>>>>>>> +struct ufshpb_req *pre_req = NULL;
>>>>>>>>> +int qd = hpb->sdev_ufs_lu->queue_depth / 2;
>>>>>>>>> +int i, j;
>>>>>>>>> +
>>>>>>>>> +INIT_LIST_HEAD(>lh_pre_req_free);
>>>>>>>>> +
>>>>>>>>> +hpb->pre_req = kcalloc(qd, sizeof(struct ufshpb_req),
>>>>>>>>> GFP_KERNEL);
>>>>>>>>> +hpb->throttle_pre_req = qd;
>>>>>>>>> +hpb->num_inflight_pre_req = 0;
>>>>>>>>> +
>>>>>>>>> +if (!hpb->pre_req)
>>>>>>>>> +goto release_mem;
>>>>>>>>> +
>>>>>>>>> +for (i = 0; i < qd; i++) {
>>>>>>>>> +pre_req = hpb->pre_req + i;
>>>>>>>>> +INIT_LIST_HEAD(_req->list_req);
>>>>>>>>> +pre_req->req = NULL;
>>>>>>>>> +pre_req->bio = NULL;
>>>>>>>> 
>>>>>>>> Why don't prepare bio as same as wb.m_page? Won't that save more
>>>>>>>> time
>>>>>>>> for ufshpb_issue_pre_req()?
>>>>>>> 
>>>>>>> It is pre_req pool. So although we prepare bio at this time, it 
>>>>>>> just
>>>>>>> only for first pre_req.
>>>>>> 
>>>>>> I meant removing the bio_alloc() in ufshpb_issue_pre_req() and
>>>>>> bio_put()
>>>>>> in ufshpb_pre_req_compl_fn(). bios, in pre_req's case, just hold a
>>>>>> page.
>>>>>> So, prepare 16 (if queue depth is 32) bios here, just use them 
>>>>>> along
>>>>>> with
>>>>>> wb.m_page and call bio_reset() in ufshpb_pre_req_compl_fn(). Shall 
>>>>>> it
>>>>>> work?
>>>>>> 
>>>>> 
>>>>> If it works, you can even have the bio_add_pc_page() called here.
>>>>> Later
>>>>> in
>>>>> ufshpb_execute_pre_req(), you don't need to call
>>>>> ufshpb_pre_req_add_bio_page(),
>>>>>

RE: Re: [PATCH v29 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-17 Thread Daejun Park

>On 2021-03-18 10:02, Daejun Park wrote:
>>> On 2021-03-17 09:42, Daejun Park wrote:
>>>>> On 2021-03-15 15:23, Can Guo wrote:
>>>>>> On 2021-03-15 15:07, Daejun Park wrote:
>>>>>>>>> This patch supports the HPB 2.0.
>>>>>>>>> 
>>>>>>>>> The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
>>>>>>>>> In the case of Read (<= 32KB) is supported as single HPB read.
>>>>>>>>> In the case of Read (36KB ~ 512KB) is supported by as a
>>>>>>>>> combination
>>>>>>>>> of
>>>>>>>>> write buffer command and HPB read command to deliver more PPN.
>>>>>>>>> The write buffer commands may not be issued immediately due to
>>>>>>>>> busy
>>>>>>>>> tags.
>>>>>>>>> To use HPB read more aggressively, the driver can requeue the
>>>>>>>>> write
>>>>>>>>> buffer
>>>>>>>>> command. The requeue threshold is implemented as timeout and can
>>>>>>>>> be
>>>>>>>>> modified with requeue_timeout_ms entry in sysfs.
>>>>>>>>> 
>>>>>>>>> Signed-off-by: Daejun Park 
>>>>>>>>> ---
>>>>>>>>> +static struct attribute *hpb_dev_param_attrs[] = {
>>>>>>>>> +_attr_requeue_timeout_ms.attr,
>>>>>>>>> +NULL,
>>>>>>>>> +};
>>>>>>>>> +
>>>>>>>>> +struct attribute_group ufs_sysfs_hpb_param_group = {
>>>>>>>>> +.name = "hpb_param_sysfs",
>>>>>>>>> +.attrs = hpb_dev_param_attrs,
>>>>>>>>> +};
>>>>>>>>> +
>>>>>>>>> +static int ufshpb_pre_req_mempool_init(struct ufshpb_lu *hpb)
>>>>>>>>> +{
>>>>>>>>> +struct ufshpb_req *pre_req = NULL;
>>>>>>>>> +int qd = hpb->sdev_ufs_lu->queue_depth / 2;
>>>>>>>>> +int i, j;
>>>>>>>>> +
>>>>>>>>> +INIT_LIST_HEAD(>lh_pre_req_free);
>>>>>>>>> +
>>>>>>>>> +hpb->pre_req = kcalloc(qd, sizeof(struct ufshpb_req),
>>>>>>>>> GFP_KERNEL);
>>>>>>>>> +hpb->throttle_pre_req = qd;
>>>>>>>>> +hpb->num_inflight_pre_req = 0;
>>>>>>>>> +
>>>>>>>>> +if (!hpb->pre_req)
>>>>>>>>> +goto release_mem;
>>>>>>>>> +
>>>>>>>>> +for (i = 0; i < qd; i++) {
>>>>>>>>> +pre_req = hpb->pre_req + i;
>>>>>>>>> +INIT_LIST_HEAD(_req->list_req);
>>>>>>>>> +pre_req->req = NULL;
>>>>>>>>> +pre_req->bio = NULL;
>>>>>>>> 
>>>>>>>> Why don't prepare bio as same as wb.m_page? Won't that save more
>>>>>>>> time
>>>>>>>> for ufshpb_issue_pre_req()?
>>>>>>> 
>>>>>>> It is pre_req pool. So although we prepare bio at this time, it 
>>>>>>> just
>>>>>>> only for first pre_req.
>>>>>> 
>>>>>> I meant removing the bio_alloc() in ufshpb_issue_pre_req() and
>>>>>> bio_put()
>>>>>> in ufshpb_pre_req_compl_fn(). bios, in pre_req's case, just hold a
>>>>>> page.
>>>>>> So, prepare 16 (if queue depth is 32) bios here, just use them 
>>>>>> along
>>>>>> with
>>>>>> wb.m_page and call bio_reset() in ufshpb_pre_req_compl_fn(). Shall 
>>>>>> it
>>>>>> work?
>>>>>> 
>>>>> 
>>>>> If it works, you can even have the bio_add_pc_page() called here.
>>>>> Later
>>>>> in
>>>>> ufshpb_execute_pre_req(), you don't need to call
>>>>> ufshpb_pre_req_add_bio_page(),
>>>>>

RE: Re: [PATCH v29 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-17 Thread Daejun Park

>On 2021-03-17 09:42, Daejun Park wrote:
>>> On 2021-03-15 15:23, Can Guo wrote:
>>>> On 2021-03-15 15:07, Daejun Park wrote:
>>>>>>> This patch supports the HPB 2.0.
>>>>>>> 
>>>>>>> The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
>>>>>>> In the case of Read (<= 32KB) is supported as single HPB read.
>>>>>>> In the case of Read (36KB ~ 512KB) is supported by as a 
>>>>>>> combination
>>>>>>> of
>>>>>>> write buffer command and HPB read command to deliver more PPN.
>>>>>>> The write buffer commands may not be issued immediately due to 
>>>>>>> busy
>>>>>>> tags.
>>>>>>> To use HPB read more aggressively, the driver can requeue the 
>>>>>>> write
>>>>>>> buffer
>>>>>>> command. The requeue threshold is implemented as timeout and can 
>>>>>>> be
>>>>>>> modified with requeue_timeout_ms entry in sysfs.
>>>>>>> 
>>>>>>> Signed-off-by: Daejun Park 
>>>>>>> ---
>>>>>>> +static struct attribute *hpb_dev_param_attrs[] = {
>>>>>>> +_attr_requeue_timeout_ms.attr,
>>>>>>> +NULL,
>>>>>>> +};
>>>>>>> +
>>>>>>> +struct attribute_group ufs_sysfs_hpb_param_group = {
>>>>>>> +.name = "hpb_param_sysfs",
>>>>>>> +.attrs = hpb_dev_param_attrs,
>>>>>>> +};
>>>>>>> +
>>>>>>> +static int ufshpb_pre_req_mempool_init(struct ufshpb_lu *hpb)
>>>>>>> +{
>>>>>>> +struct ufshpb_req *pre_req = NULL;
>>>>>>> +int qd = hpb->sdev_ufs_lu->queue_depth / 2;
>>>>>>> +int i, j;
>>>>>>> +
>>>>>>> +INIT_LIST_HEAD(>lh_pre_req_free);
>>>>>>> +
>>>>>>> +hpb->pre_req = kcalloc(qd, sizeof(struct ufshpb_req),
>>>>>>> GFP_KERNEL);
>>>>>>> +hpb->throttle_pre_req = qd;
>>>>>>> +hpb->num_inflight_pre_req = 0;
>>>>>>> +
>>>>>>> +if (!hpb->pre_req)
>>>>>>> +goto release_mem;
>>>>>>> +
>>>>>>> +for (i = 0; i < qd; i++) {
>>>>>>> +pre_req = hpb->pre_req + i;
>>>>>>> +INIT_LIST_HEAD(_req->list_req);
>>>>>>> +pre_req->req = NULL;
>>>>>>> +pre_req->bio = NULL;
>>>>>> 
>>>>>> Why don't prepare bio as same as wb.m_page? Won't that save more 
>>>>>> time
>>>>>> for ufshpb_issue_pre_req()?
>>>>> 
>>>>> It is pre_req pool. So although we prepare bio at this time, it just
>>>>> only for first pre_req.
>>>> 
>>>> I meant removing the bio_alloc() in ufshpb_issue_pre_req() and
>>>> bio_put()
>>>> in ufshpb_pre_req_compl_fn(). bios, in pre_req's case, just hold a
>>>> page.
>>>> So, prepare 16 (if queue depth is 32) bios here, just use them along
>>>> with
>>>> wb.m_page and call bio_reset() in ufshpb_pre_req_compl_fn(). Shall it
>>>> work?
>>>> 
>>> 
>>> If it works, you can even have the bio_add_pc_page() called here. 
>>> Later
>>> in
>>> ufshpb_execute_pre_req(), you don't need to call
>>> ufshpb_pre_req_add_bio_page(),
>>> just call ufshpb_prep_entry() once instead - it save many repeated 
>>> steps
>>> for a
>>> pre_req, and you don't even need to call bio_reset() in this case, 
>>> since
>>> for a
>>> bio, nothing changes after it is binded with a specific page...
>> 
>> Hi, Can Guo
>> 
>> I tried the idea that you suggested, but it doesn't work properly.
>> This optimization should be done next time for enhancement.
> 
>Can you elaborate please? Any error seen?
> 
>Per my understanding, in the case for pre_reqs, a bio is no different
>from a page. Here it can reserve 16 pages for later use, which can be
>done the same for bios.

I found some problem w

RE: Re: [PATCH v5 05/10] scsi: ufshpb: Region inactivation in host mode

2021-03-16 Thread Daejun Park

>> >> ---
>> >>  drivers/scsi/ufs/ufshpb.c | 14 ++
>> >>  drivers/scsi/ufs/ufshpb.h |  1 +
>> >>  2 files changed, 15 insertions(+)
>> >>
>> >> diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
>> >> index 6f4fd22eaf2f..0744feb4d484 100644
>> >> --- a/drivers/scsi/ufs/ufshpb.c
>> >> +++ b/drivers/scsi/ufs/ufshpb.c
>> >> @@ -907,6 +907,7 @@ static int ufshpb_execute_umap_req(struct
>> >> ufshpb_lu *hpb,
>> >>
>> >>  blk_execute_rq_nowait(q, NULL, req, 1, ufshpb_umap_req_compl_fn);
>> >>
>> >> +hpb->stats.umap_req_cnt++;
>> >>  return 0;
>> >>  }
>> >>
>> >> @@ -1103,6 +1104,12 @@ static int ufshpb_issue_umap_req(struct
>> >> ufshpb_lu *hpb,
>> >>  return -EAGAIN;
>> >>  }
>> >>
>> >> +static int ufshpb_issue_umap_single_req(struct ufshpb_lu *hpb,
>> >> +struct ufshpb_region *rgn)
>> >> +{
>> >> +return ufshpb_issue_umap_req(hpb, rgn);
>> >> +}
>> >> +
>> >>  static int ufshpb_issue_umap_all_req(struct ufshpb_lu *hpb)
>> >>  {
>> >>  return ufshpb_issue_umap_req(hpb, NULL);
>> >> @@ -1115,6 +1122,10 @@ static void __ufshpb_evict_region(struct
>> >> ufshpb_lu *hpb,
>> >>  struct ufshpb_subregion *srgn;
>> >>  int srgn_idx;
>> >>
>> >> +
>> >> +if (hpb->is_hcm && ufshpb_issue_umap_single_req(hpb, rgn))
>> >
>> > __ufshpb_evict_region() is called with rgn_state_lock held and IRQ
>> > disabled,
>> > when ufshpb_issue_umap_single_req() invokes blk_execute_rq_nowait(),
>> > below
>> > warning shall pop up every time, fix it?
>> >
>> > void blk_execute_rq_nowait(struct request_queue *q, struct gendisk
>> > *bd_disk,
>> >  struct request *rq, int at_head,
>> >  rq_end_io_fn *done)
>> > {
>> >   WARN_ON(irqs_disabled());
>> > ...
>> >
>> 
>> Moreover, since we are here with rgn_state_lock held and IRQ
>> disabled,
>> in ufshpb_get_req(), rq = kmem_cache_alloc(hpb->map_req_cache,
>> GFP_KERNEL)
>> has the GFP_KERNEL flag, scheduling while atomic???
> I think your comment applies to  ufshpb_issue_umap_all_req as well,
> Which is called from slave_configure/scsi_add_lun.
> 
> Since the host-mode series is utilizing the framework laid by the
> device-mode,
> Maybe you can add this comment to  Daejun's last version?
 
 Hi Avri, Can Guo
 
 I think ufshpb_issue_umap_single_req() can be moved to end of
 ufshpb_evict_region().
 Then we can avoid rgn_state_lock when it sends unmap command.
>>> 
>>> I am not the expert here, please you two fix it. I am just reporting
>>> what can be wrong. Anyways, ufshpb_issue_umap_single_req() should not
>>> be called with rgn_state_lock held - think about below (another 
>>> deadly)
>>> scenario.
>>> 
>>> lock(rgn_state_lock)
>>>   ufshpb_issue_umap_single_req()
>>> ufshpb_prep()
>>>lock(rgn_state_lock)   <-- recursive spin_lock
>>> 
>>> BTW, @Daejun shouldn't we stop passthrough cmds from stepping
>>> into ufshpb_prep()? In current code, you are trying to use below
>>> check to block cmds other than write/discard/read, but a passthrough
>>> cmd can not be blocked by the check.
>>> 
>>>  if (!ufshpb_is_write_or_discard_cmd(cmd) &&
>>>  !ufshpb_is_read_cmd(cmd) )
>>>  return 0;
>> 
>> I found this problem too. I fixed it and submit next patch.
> 
>You mean in V30, which has not been uploaded yet, right?

Yes, it is about v30.

Thanks,
Daejun

>Thanks,
>Can Guo.
> 
>> 
>> if (blk_rq_is_scsi(cmd->request) ||
>> (!ufshpb_is_write_or_discard_cmd(cmd) &&
>> !ufshpb_is_read_cmd(cmd)))
>> return 0;
>> 
>> 
>> Thanks,
>> Daejun
>> 
>>> Thanks,
>>> Can Guo.
>>> 
 
 Thanks,
 Daejun
 
 
> Thanks,
> Avri
> 
>> 
>> Can Guo.
>> 
>> > Thanks.
>> > Can Guo.
>> >
>> >> +return;
>> >> +
>> >>  lru_info = >lru_info;
>> >>
>> >>  dev_dbg(>sdev_ufs_lu->sdev_dev, "evict region %d\n",
>> >> rgn->rgn_idx);
>> >> @@ -1855,6 +1866,7 @@ ufshpb_sysfs_attr_show_func(rb_noti_cnt);
>> >>  ufshpb_sysfs_attr_show_func(rb_active_cnt);
>> >>  ufshpb_sysfs_attr_show_func(rb_inactive_cnt);
>> >>  ufshpb_sysfs_attr_show_func(map_req_cnt);
>> >> +ufshpb_sysfs_attr_show_func(umap_req_cnt);
>> >>
>> >>  static struct attribute *hpb_dev_stat_attrs[] = {
>> >>  _attr_hit_cnt.attr,
>> >> @@ -1863,6 +1875,7 @@ static struct attribute *hpb_dev_stat_attrs[] =
>> >> {
>> >>  _attr_rb_active_cnt.attr,
>> >>  _attr_rb_inactive_cnt.attr,
>> >>  _attr_map_req_cnt.attr,
>> >> +_attr_umap_req_cnt.attr,
>> >>  NULL,
>> >>  };
>> >>
>>

RE: Re: [PATCH v5 05/10] scsi: ufshpb: Region inactivation in host mode

2021-03-16 Thread Daejun Park

 >> ---
 >>  drivers/scsi/ufs/ufshpb.c | 14 ++
 >>  drivers/scsi/ufs/ufshpb.h |  1 +
 >>  2 files changed, 15 insertions(+)
 >>
 >> diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
 >> index 6f4fd22eaf2f..0744feb4d484 100644
 >> --- a/drivers/scsi/ufs/ufshpb.c
 >> +++ b/drivers/scsi/ufs/ufshpb.c
 >> @@ -907,6 +907,7 @@ static int ufshpb_execute_umap_req(struct
 >> ufshpb_lu *hpb,
 >>
 >>  blk_execute_rq_nowait(q, NULL, req, 1, ufshpb_umap_req_compl_fn);
 >>
 >> +hpb->stats.umap_req_cnt++;
 >>  return 0;
 >>  }
 >>
 >> @@ -1103,6 +1104,12 @@ static int ufshpb_issue_umap_req(struct
 >> ufshpb_lu *hpb,
 >>  return -EAGAIN;
 >>  }
 >>
 >> +static int ufshpb_issue_umap_single_req(struct ufshpb_lu *hpb,
 >> +struct ufshpb_region *rgn)
 >> +{
 >> +return ufshpb_issue_umap_req(hpb, rgn);
 >> +}
 >> +
 >>  static int ufshpb_issue_umap_all_req(struct ufshpb_lu *hpb)
 >>  {
 >>  return ufshpb_issue_umap_req(hpb, NULL);
 >> @@ -1115,6 +1122,10 @@ static void __ufshpb_evict_region(struct
 >> ufshpb_lu *hpb,
 >>  struct ufshpb_subregion *srgn;
 >>  int srgn_idx;
 >>
 >> +
 >> +if (hpb->is_hcm && ufshpb_issue_umap_single_req(hpb, rgn))
 >
 > __ufshpb_evict_region() is called with rgn_state_lock held and IRQ
 > disabled,
 > when ufshpb_issue_umap_single_req() invokes blk_execute_rq_nowait(),
 > below
 > warning shall pop up every time, fix it?
 >
 > void blk_execute_rq_nowait(struct request_queue *q, struct gendisk
 > *bd_disk,
 >  struct request *rq, int at_head,
 >  rq_end_io_fn *done)
 > {
 >   WARN_ON(irqs_disabled());
 > ...
 >
 
 Moreover, since we are here with rgn_state_lock held and IRQ 
 disabled,
 in ufshpb_get_req(), rq = kmem_cache_alloc(hpb->map_req_cache,
 GFP_KERNEL)
 has the GFP_KERNEL flag, scheduling while atomic???
>>> I think your comment applies to  ufshpb_issue_umap_all_req as well,
>>> Which is called from slave_configure/scsi_add_lun.
>>> 
>>> Since the host-mode series is utilizing the framework laid by the 
>>> device-mode,
>>> Maybe you can add this comment to  Daejun's last version?
>> 
>> Hi Avri, Can Guo
>> 
>> I think ufshpb_issue_umap_single_req() can be moved to end of
>> ufshpb_evict_region().
>> Then we can avoid rgn_state_lock when it sends unmap command.
> 
>I am not the expert here, please you two fix it. I am just reporting
>what can be wrong. Anyways, ufshpb_issue_umap_single_req() should not
>be called with rgn_state_lock held - think about below (another deadly)
>scenario.
> 
>lock(rgn_state_lock)
>   ufshpb_issue_umap_single_req()
> ufshpb_prep()
>lock(rgn_state_lock)   <-- recursive spin_lock
> 
>BTW, @Daejun shouldn't we stop passthrough cmds from stepping
>into ufshpb_prep()? In current code, you are trying to use below
>check to block cmds other than write/discard/read, but a passthrough
>cmd can not be blocked by the check.
> 
>  if (!ufshpb_is_write_or_discard_cmd(cmd) &&
>  !ufshpb_is_read_cmd(cmd) )
>  return 0;

I found this problem too. I fixed it and submit next patch.

if (blk_rq_is_scsi(cmd->request) ||
(!ufshpb_is_write_or_discard_cmd(cmd) &&
!ufshpb_is_read_cmd(cmd)))
return 0;


Thanks,
Daejun

>Thanks,
>Can Guo.
> 
>> 
>> Thanks,
>> Daejun
>> 
>> 
>>> Thanks,
>>> Avri
>>> 
 
 Can Guo.
 
 > Thanks.
 > Can Guo.
 >
 >> +return;
 >> +
 >>  lru_info = >lru_info;
 >>
 >>  dev_dbg(>sdev_ufs_lu->sdev_dev, "evict region %d\n",
 >> rgn->rgn_idx);
 >> @@ -1855,6 +1866,7 @@ ufshpb_sysfs_attr_show_func(rb_noti_cnt);
 >>  ufshpb_sysfs_attr_show_func(rb_active_cnt);
 >>  ufshpb_sysfs_attr_show_func(rb_inactive_cnt);
 >>  ufshpb_sysfs_attr_show_func(map_req_cnt);
 >> +ufshpb_sysfs_attr_show_func(umap_req_cnt);
 >>
 >>  static struct attribute *hpb_dev_stat_attrs[] = {
 >>  _attr_hit_cnt.attr,
 >> @@ -1863,6 +1875,7 @@ static struct attribute *hpb_dev_stat_attrs[] =
 >> {
 >>  _attr_rb_active_cnt.attr,
 >>  _attr_rb_inactive_cnt.attr,
 >>  _attr_map_req_cnt.attr,
 >> +_attr_umap_req_cnt.attr,
 >>  NULL,
 >>  };
 >>
 >> @@ -1978,6 +1991,7 @@ static void ufshpb_stat_init(struct ufshpb_lu
 >> *hpb)
 >>  hpb->stats.rb_active_cnt = 0;
 >>  hpb->stats.rb_inactive_cnt = 0;
 >>  hpb->stats.map_req_cnt = 0;
 >> +hpb->stats.umap_req_cnt = 0;
 >>  }
 >>
 >>  static void ufshpb_param_init(struct ufshpb_lu *hpb)
 >> diff --git a/drivers/scsi/ufs/ufshpb.h b/drivers/scsi/ufs/ufshpb.h
 >> index

[PATCH] scsi: ufs: Add selector to ufshcd_query_flag* APIs

2021-03-16 Thread Daejun Park

Unlike other query APIs in UFS, ufshcd_query_flag has a fixed selector
as 0. This patch allows ufshcd_query_flag API to choose selector value
by parameter.

Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs-sysfs.c |  2 +-
 drivers/scsi/ufs/ufshcd.c| 29 +
 drivers/scsi/ufs/ufshcd.h|  2 +-
 3 files changed, 19 insertions(+), 14 deletions(-)

diff --git a/drivers/scsi/ufs/ufs-sysfs.c b/drivers/scsi/ufs/ufs-sysfs.c
index acc54f530f2d..606b058a3394 100644
--- a/drivers/scsi/ufs/ufs-sysfs.c
+++ b/drivers/scsi/ufs/ufs-sysfs.c
@@ -746,7 +746,7 @@ static ssize_t _name##_show(struct device *dev, 
\
index = ufshcd_wb_get_query_index(hba); \
pm_runtime_get_sync(hba->dev);  \
ret = ufshcd_query_flag(hba, UPIU_QUERY_OPCODE_READ_FLAG,   \
-   QUERY_FLAG_IDN##_uname, index, );  \
+   QUERY_FLAG_IDN##_uname, index, , 0);   \
pm_runtime_put_sync(hba->dev);  \
if (ret) {  \
ret = -EINVAL;  \
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 8c0ff024231c..c2fd9c58d6b8 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2940,13 +2940,15 @@ static inline void ufshcd_init_query(struct ufs_hba 
*hba,
 }
 
 static int ufshcd_query_flag_retry(struct ufs_hba *hba,
-   enum query_opcode opcode, enum flag_idn idn, u8 index, bool *flag_res)
+   enum query_opcode opcode, enum flag_idn idn, u8 index, bool *flag_res,
+   u8 selector)
 {
int ret;
int retries;
 
for (retries = 0; retries < QUERY_REQ_RETRIES; retries++) {
-   ret = ufshcd_query_flag(hba, opcode, idn, index, flag_res);
+   ret = ufshcd_query_flag(hba, opcode, idn, index, flag_res,
+   selector);
if (ret)
dev_dbg(hba->dev,
"%s: failed with error %d, retries %d\n",
@@ -2969,15 +2971,17 @@ static int ufshcd_query_flag_retry(struct ufs_hba *hba,
  * @idn: flag idn to access
  * @index: flag index to access
  * @flag_res: the flag value after the query request completes
+ * @selector: selector field
  *
  * Returns 0 for success, non-zero in case of failure
  */
 int ufshcd_query_flag(struct ufs_hba *hba, enum query_opcode opcode,
-   enum flag_idn idn, u8 index, bool *flag_res)
+   enum flag_idn idn, u8 index, bool *flag_res,
+   u8 selector)
 {
struct ufs_query_req *request = NULL;
struct ufs_query_res *response = NULL;
-   int err, selector = 0;
+   int err;
int timeout = QUERY_REQ_TIMEOUT;
 
BUG_ON(!hba);
@@ -4331,7 +4335,7 @@ static int ufshcd_complete_dev_init(struct ufs_hba *hba)
ktime_t timeout;
 
err = ufshcd_query_flag_retry(hba, UPIU_QUERY_OPCODE_SET_FLAG,
-   QUERY_FLAG_IDN_FDEVICEINIT, 0, NULL);
+   QUERY_FLAG_IDN_FDEVICEINIT, 0, NULL, 0);
if (err) {
dev_err(hba->dev,
"%s setting fDeviceInit flag failed with error %d\n",
@@ -4343,7 +4347,8 @@ static int ufshcd_complete_dev_init(struct ufs_hba *hba)
timeout = ktime_add_ms(ktime_get(), FDEVICEINIT_COMPL_TIMEOUT);
do {
err = ufshcd_query_flag(hba, UPIU_QUERY_OPCODE_READ_FLAG,
-   QUERY_FLAG_IDN_FDEVICEINIT, 0, 
_res);
+   QUERY_FLAG_IDN_FDEVICEINIT, 0, 
_res,
+   0);
if (!flag_res)
break;
usleep_range(5000, 1);
@@ -5250,7 +5255,7 @@ static int ufshcd_enable_auto_bkops(struct ufs_hba *hba)
goto out;
 
err = ufshcd_query_flag_retry(hba, UPIU_QUERY_OPCODE_SET_FLAG,
-   QUERY_FLAG_IDN_BKOPS_EN, 0, NULL);
+   QUERY_FLAG_IDN_BKOPS_EN, 0, NULL, 0);
if (err) {
dev_err(hba->dev, "%s: failed to enable bkops %d\n",
__func__, err);
@@ -5300,7 +5305,7 @@ static int ufshcd_disable_auto_bkops(struct ufs_hba *hba)
}
 
err = ufshcd_query_flag_retry(hba, UPIU_QUERY_OPCODE_CLEAR_FLAG,
-   QUERY_FLAG_IDN_BKOPS_EN, 0, NULL);
+   QUERY_FLAG_IDN_BKOPS_EN, 0, NULL, 0);
if (err) {
dev_err(hba->dev, "%s: failed to disable bkops %d\n",
__func__, err);
@@ -5463,7 +5468,7 @@ int ufshcd_wb_ctrl(struct ufs_hba *hba, bool enable)
 
index = ufshcd_wb_get_query_

RE: RE: [PATCH v5 05/10] scsi: ufshpb: Region inactivation in host mode

2021-03-16 Thread Daejun Park

>> >> ---
>> >>  drivers/scsi/ufs/ufshpb.c | 14 ++
>> >>  drivers/scsi/ufs/ufshpb.h |  1 +
>> >>  2 files changed, 15 insertions(+)
>> >>
>> >> diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
>> >> index 6f4fd22eaf2f..0744feb4d484 100644
>> >> --- a/drivers/scsi/ufs/ufshpb.c
>> >> +++ b/drivers/scsi/ufs/ufshpb.c
>> >> @@ -907,6 +907,7 @@ static int ufshpb_execute_umap_req(struct
>> >> ufshpb_lu *hpb,
>> >>
>> >>  blk_execute_rq_nowait(q, NULL, req, 1, ufshpb_umap_req_compl_fn);
>> >>
>> >> +hpb->stats.umap_req_cnt++;
>> >>  return 0;
>> >>  }
>> >>
>> >> @@ -1103,6 +1104,12 @@ static int ufshpb_issue_umap_req(struct
>> >> ufshpb_lu *hpb,
>> >>  return -EAGAIN;
>> >>  }
>> >>
>> >> +static int ufshpb_issue_umap_single_req(struct ufshpb_lu *hpb,
>> >> +struct ufshpb_region *rgn)
>> >> +{
>> >> +return ufshpb_issue_umap_req(hpb, rgn);
>> >> +}
>> >> +
>> >>  static int ufshpb_issue_umap_all_req(struct ufshpb_lu *hpb)
>> >>  {
>> >>  return ufshpb_issue_umap_req(hpb, NULL);
>> >> @@ -1115,6 +1122,10 @@ static void __ufshpb_evict_region(struct
>> >> ufshpb_lu *hpb,
>> >>  struct ufshpb_subregion *srgn;
>> >>  int srgn_idx;
>> >>
>> >> +
>> >> +if (hpb->is_hcm && ufshpb_issue_umap_single_req(hpb, rgn))
>> >
>> > __ufshpb_evict_region() is called with rgn_state_lock held and IRQ
>> > disabled,
>> > when ufshpb_issue_umap_single_req() invokes blk_execute_rq_nowait(),
>> > below
>> > warning shall pop up every time, fix it?
>> >
>> > void blk_execute_rq_nowait(struct request_queue *q, struct gendisk
>> > *bd_disk,
>> >  struct request *rq, int at_head,
>> >  rq_end_io_fn *done)
>> > {
>> >   WARN_ON(irqs_disabled());
>> > ...
>> >
>> 
>> Moreover, since we are here with rgn_state_lock held and IRQ disabled,
>> in ufshpb_get_req(), rq = kmem_cache_alloc(hpb->map_req_cache,
>> GFP_KERNEL)
>> has the GFP_KERNEL flag, scheduling while atomic???
>I think your comment applies to  ufshpb_issue_umap_all_req as well,
>Which is called from slave_configure/scsi_add_lun.
> 
>Since the host-mode series is utilizing the framework laid by the device-mode,
>Maybe you can add this comment to  Daejun's last version?

Hi Avri, Can Guo

I think ufshpb_issue_umap_single_req() can be moved to end of 
ufshpb_evict_region().
Then we can avoid rgn_state_lock when it sends unmap command.

Thanks,
Daejun


>Thanks,
>Avri
> 
>> 
>> Can Guo.
>> 
>> > Thanks.
>> > Can Guo.
>> >
>> >> +return;
>> >> +
>> >>  lru_info = >lru_info;
>> >>
>> >>  dev_dbg(>sdev_ufs_lu->sdev_dev, "evict region %d\n",
>> >> rgn->rgn_idx);
>> >> @@ -1855,6 +1866,7 @@ ufshpb_sysfs_attr_show_func(rb_noti_cnt);
>> >>  ufshpb_sysfs_attr_show_func(rb_active_cnt);
>> >>  ufshpb_sysfs_attr_show_func(rb_inactive_cnt);
>> >>  ufshpb_sysfs_attr_show_func(map_req_cnt);
>> >> +ufshpb_sysfs_attr_show_func(umap_req_cnt);
>> >>
>> >>  static struct attribute *hpb_dev_stat_attrs[] = {
>> >>  _attr_hit_cnt.attr,
>> >> @@ -1863,6 +1875,7 @@ static struct attribute *hpb_dev_stat_attrs[] =
>> >> {
>> >>  _attr_rb_active_cnt.attr,
>> >>  _attr_rb_inactive_cnt.attr,
>> >>  _attr_map_req_cnt.attr,
>> >> +_attr_umap_req_cnt.attr,
>> >>  NULL,
>> >>  };
>> >>
>> >> @@ -1978,6 +1991,7 @@ static void ufshpb_stat_init(struct ufshpb_lu
>> >> *hpb)
>> >>  hpb->stats.rb_active_cnt = 0;
>> >>  hpb->stats.rb_inactive_cnt = 0;
>> >>  hpb->stats.map_req_cnt = 0;
>> >> +hpb->stats.umap_req_cnt = 0;
>> >>  }
>> >>
>> >>  static void ufshpb_param_init(struct ufshpb_lu *hpb)
>> >> diff --git a/drivers/scsi/ufs/ufshpb.h b/drivers/scsi/ufs/ufshpb.h
>> >> index bd4308010466..84598a317897 100644
>> >> --- a/drivers/scsi/ufs/ufshpb.h
>> >> +++ b/drivers/scsi/ufs/ufshpb.h
>> >> @@ -186,6 +186,7 @@ struct ufshpb_stats {
>> >>  u64 rb_inactive_cnt;
>> >>  u64 map_req_cnt;
>> >>  u64 pre_req_cnt;
>> >> +u64 umap_req_cnt;
>> >>  };
>> >>
>> >>  struct ufshpb_lu {
> 
> 
>

RE: Re: [PATCH v29 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-16 Thread Daejun Park

>On 2021-03-15 15:23, Can Guo wrote:
>> On 2021-03-15 15:07, Daejun Park wrote:
>>>>> This patch supports the HPB 2.0.
>>>>> 
>>>>> The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
>>>>> In the case of Read (<= 32KB) is supported as single HPB read.
>>>>> In the case of Read (36KB ~ 512KB) is supported by as a combination 
>>>>> of
>>>>> write buffer command and HPB read command to deliver more PPN.
>>>>> The write buffer commands may not be issued immediately due to busy
>>>>> tags.
>>>>> To use HPB read more aggressively, the driver can requeue the write
>>>>> buffer
>>>>> command. The requeue threshold is implemented as timeout and can be
>>>>> modified with requeue_timeout_ms entry in sysfs.
>>>>> 
>>>>> Signed-off-by: Daejun Park 
>>>>> ---
>>>>> +static struct attribute *hpb_dev_param_attrs[] = {
>>>>> +_attr_requeue_timeout_ms.attr,
>>>>> +NULL,
>>>>> +};
>>>>> +
>>>>> +struct attribute_group ufs_sysfs_hpb_param_group = {
>>>>> +.name = "hpb_param_sysfs",
>>>>> +.attrs = hpb_dev_param_attrs,
>>>>> +};
>>>>> +
>>>>> +static int ufshpb_pre_req_mempool_init(struct ufshpb_lu *hpb)
>>>>> +{
>>>>> +struct ufshpb_req *pre_req = NULL;
>>>>> +int qd = hpb->sdev_ufs_lu->queue_depth / 2;
>>>>> +int i, j;
>>>>> +
>>>>> +INIT_LIST_HEAD(>lh_pre_req_free);
>>>>> +
>>>>> +hpb->pre_req = kcalloc(qd, sizeof(struct ufshpb_req), 
>>>>> GFP_KERNEL);
>>>>> +hpb->throttle_pre_req = qd;
>>>>> +hpb->num_inflight_pre_req = 0;
>>>>> +
>>>>> +if (!hpb->pre_req)
>>>>> +goto release_mem;
>>>>> +
>>>>> +for (i = 0; i < qd; i++) {
>>>>> +pre_req = hpb->pre_req + i;
>>>>> +INIT_LIST_HEAD(_req->list_req);
>>>>> +pre_req->req = NULL;
>>>>> +pre_req->bio = NULL;
>>>> 
>>>> Why don't prepare bio as same as wb.m_page? Won't that save more time
>>>> for ufshpb_issue_pre_req()?
>>> 
>>> It is pre_req pool. So although we prepare bio at this time, it just
>>> only for first pre_req.
>> 
>> I meant removing the bio_alloc() in ufshpb_issue_pre_req() and 
>> bio_put()
>> in ufshpb_pre_req_compl_fn(). bios, in pre_req's case, just hold a 
>> page.
>> So, prepare 16 (if queue depth is 32) bios here, just use them along 
>> with
>> wb.m_page and call bio_reset() in ufshpb_pre_req_compl_fn(). Shall it 
>> work?
>> 
> 
>If it works, you can even have the bio_add_pc_page() called here. Later 
>in
>ufshpb_execute_pre_req(), you don't need to call 
>ufshpb_pre_req_add_bio_page(),
>just call ufshpb_prep_entry() once instead - it save many repeated steps 
>for a
>pre_req, and you don't even need to call bio_reset() in this case, since 
>for a
>bio, nothing changes after it is binded with a specific page...

Hi, Can Guo

I tried the idea that you suggested, but it doesn't work properly.
This optimization should be done next time for enhancement.

Thanks
Daejun

>Can Guo.
> 
>> Thanks,
>> Can Guo.
>> 
>>> After use it, it should be prepared bio at issue phase.
>>> 
>>> Thanks,
>>> Daejun
>>> 
>>>> 
>>>> Thanks,
>>>> Can Guo.
>>>> 
>>>>> +
>>>>> +pre_req->wb.m_page = alloc_page(GFP_KERNEL | 
>>>>> __GFP_ZERO);
>>>>> +if (!pre_req->wb.m_page) {
>>>>> +for (j = 0; j < i; j++)
>>>>> +
>>>>> __free_page(hpb->pre_req[j].wb.m_page);
>>>>> +
>>>>> +goto release_mem;
>>>>> +}
>>>>> +list_add_tail(_req->list_req, 
>>>>> >lh_pre_req_free);
>>>>> +}
>>>>> +
>>>>> +return 0;
>>>>> +release_mem:
>>>>> +kfree(hpb->pre_req);
>>>>> +return -ENOMEM;
>>>>> +}
>>>>> +
>>>> 
>>>> 
>>>> 
> 
> 
>

RE: Re: [PATCH v29 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-15 Thread Daejun Park

>> This patch supports the HPB 2.0.
>> 
>> The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
>> In the case of Read (<= 32KB) is supported as single HPB read.
>> In the case of Read (36KB ~ 512KB) is supported by as a combination of
>> write buffer command and HPB read command to deliver more PPN.
>> The write buffer commands may not be issued immediately due to busy 
>> tags.
>> To use HPB read more aggressively, the driver can requeue the write 
>> buffer
>> command. The requeue threshold is implemented as timeout and can be
>> modified with requeue_timeout_ms entry in sysfs.
>> 
>> Signed-off-by: Daejun Park 
>> ---
>> +static struct attribute *hpb_dev_param_attrs[] = {
>> +_attr_requeue_timeout_ms.attr,
>> +NULL,
>> +};
>> +
>> +struct attribute_group ufs_sysfs_hpb_param_group = {
>> +.name = "hpb_param_sysfs",
>> +.attrs = hpb_dev_param_attrs,
>> +};
>> +
>> +static int ufshpb_pre_req_mempool_init(struct ufshpb_lu *hpb)
>> +{
>> +struct ufshpb_req *pre_req = NULL;
>> +int qd = hpb->sdev_ufs_lu->queue_depth / 2;
>> +int i, j;
>> +
>> +INIT_LIST_HEAD(>lh_pre_req_free);
>> +
>> +hpb->pre_req = kcalloc(qd, sizeof(struct ufshpb_req), GFP_KERNEL);
>> +hpb->throttle_pre_req = qd;
>> +hpb->num_inflight_pre_req = 0;
>> +
>> +if (!hpb->pre_req)
>> +goto release_mem;
>> +
>> +for (i = 0; i < qd; i++) {
>> +pre_req = hpb->pre_req + i;
>> +INIT_LIST_HEAD(_req->list_req);
>> +pre_req->req = NULL;
>> +pre_req->bio = NULL;
> 
>Why don't prepare bio as same as wb.m_page? Won't that save more time
>for ufshpb_issue_pre_req()?

It is pre_req pool. So although we prepare bio at this time, it just only for 
first pre_req.
After use it, it should be prepared bio at issue phase.

Thanks,
Daejun

> 
>Thanks,
>Can Guo.
> 
>> +
>> +pre_req->wb.m_page = alloc_page(GFP_KERNEL | __GFP_ZERO);
>> +if (!pre_req->wb.m_page) {
>> +for (j = 0; j < i; j++)
>> +__free_page(hpb->pre_req[j].wb.m_page);
>> +
>> +goto release_mem;
>> +}
>> +list_add_tail(_req->list_req, >lh_pre_req_free);
>> +}
>> +
>> +return 0;
>> +release_mem:
>> +kfree(hpb->pre_req);
>> +return -ENOMEM;
>> +}
>> +
> 
> 
>

RE: Re: [PATCH v26 2/4] scsi: ufs: L2P map management for HPB read

2021-03-14 Thread Daejun Park

>>>>> > This is a patch for managing L2P map in HPB module.
>>>>> >
>>>>> > The HPB divides logical addresses into several regions. A region
>>>>> > consists
>>>>> > of several sub-regions. The sub-region is a basic unit where L2P
>>>>> > mapping is
>>>>> > managed. The driver loads L2P mapping data of each sub-region. The
>>>>> > loaded
>>>>> > sub-region is called active-state. The HPB driver unloads L2P mapping
>>>>> > data
>>>>> > as region unit. The unloaded region is called inactive-state.
>>>>> >
>>>>> > Sub-region/region candidates to be loaded and unloaded are delivered
>>>>> > from
>>>>> > the UFS device. The UFS device delivers the recommended active
>>>>> > sub-region
>>>>> > and inactivate region to the driver using sensedata.
>>>>> > The HPB module performs L2P mapping management on the host through the
>>>>> > delivered information.
>>>>> >
>>>>> > A pinned region is a pre-set regions on the UFS device that is always
>>>>> > activate-state.
>>>>> >
>>>>> > The data structure for map data request and L2P map uses mempool API,
>>>>> > minimizing allocation overhead while avoiding static allocation.
>>>>> >
>>>>> > The mininum size of the memory pool used in the HPB is implemented
>>>>> > as a module parameter, so that it can be configurable by the user.
>>>>> >
>>>>> > To gurantee a minimum memory pool size of 4MB:
>>>>> > ufshpb_host_map_kbytes=4096
>>>>> >
>>>>> > The map_work manages active/inactive by 2 "to-do" lists.
>>>>> > Each hpb lun maintains 2 "to-do" lists:
>>>>> >   hpb->lh_inact_rgn - regions to be inactivated, and
>>>>> >   hpb->lh_act_srgn - subregions to be activated
>>>>> > Those lists are maintained on IO completion.
>>>>> >
>>>>> > Reviewed-by: Bart Van Assche 
>>>>> > Reviewed-by: Can Guo 
>>>>> > Acked-by: Avri Altman 
>>>>> > Tested-by: Bean Huo 
>>>>> > Signed-off-by: Daejun Park 
>>>>> > ---
>>>>> >  drivers/scsi/ufs/ufs.h|   36 ++
>>>>> >  drivers/scsi/ufs/ufshcd.c |4 +
>>>>> >  drivers/scsi/ufs/ufshpb.c | 1091 -
>>>>> >  drivers/scsi/ufs/ufshpb.h |   65 +++
>>>>> >  4 files changed, 1181 insertions(+), 15 deletions(-)
>>>>> >
>>>>> > diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
>>>>> > index 65563635e20e..957763db1006 100644
>>>>> > --- a/drivers/scsi/ufs/ufs.h
>>>>> > +++ b/drivers/scsi/ufs/ufs.h
>>>>> > @@ -472,6 +472,41 @@ struct utp_cmd_rsp {
>>>>> >  u8 sense_data[UFS_SENSE_SIZE];
>>>>> >  };
>>>>> >
>>>>> > +struct ufshpb_active_field {
>>>>> > +__be16 active_rgn;
>>>>> > +__be16 active_srgn;
>>>>> > +};
>>>>> > +#define HPB_ACT_FIELD_SIZE 4
>>>>> > +
>>>>> > +/**
>>>>> > + * struct utp_hpb_rsp - Response UPIU structure
>>>>> > + * @residual_transfer_count: Residual transfer count DW-3
>>>>> > + * @reserved1: Reserved double words DW-4 to DW-7
>>>>> > + * @sense_data_len: Sense data length DW-8 U16
>>>>> > + * @desc_type: Descriptor type of sense data
>>>>> > + * @additional_len: Additional length of sense data
>>>>> > + * @hpb_op: HPB operation type
>>>>> > + * @lun: LUN of response UPIU
>>>>> > + * @active_rgn_cnt: Active region count
>>>>> > + * @inactive_rgn_cnt: Inactive region count
>>>>> > + * @hpb_active_field: Recommended to read HPB region and subregion
>>>>> > + * @hpb_inactive_field: To be inactivated HPB region and subregion
>>>>> > + */
>>>>> > +struct utp_hpb_rsp {
>>>>> > +__be32 residual_transfer_count;
>>>>> > +__be32 reserved1[4];
>>>>> > +__be16 s

[PATCH v29 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-14 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  47 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   4 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  25 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 627 +++--
 drivers/scsi/ufs/ufshpb.h  |  66 ++-
 7 files changed, 698 insertions(+), 81 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index 528bf89fc98b..419adf450b89 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,45 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/flags/wb_enable
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the status of HPB.
+
+   == 
+   0  HPB is not enabled.
+   1  HPB is enab

[PATCH v29 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-03-14 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 256 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 257 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index e10984fd8d47..88dd0f34fa09 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2656,6 +2656,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 489c8b1ac580..201dc24d55b3 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,233 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v29 2/4] scsi: ufs: L2P map management for HPB read

2021-03-14 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1094 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1184 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 4eee7e31d08d..bfb84d2ba990 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -478,6 +478,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -488,6 +523,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 70a567ea7d5a..e10984fd8d47 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5021,6 +5021,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9241,6 +9244,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 1a72f6541510..489c8b1ac580 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v29 1/4] scsi: ufs: Introduce HPB feature

2021-03-14 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 167 ++
 9 files changed, 977 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..528bf89fc98b 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v29 0/4] scsi: ufs: Add Host Performance Booster Support

2021-03-14 Thread Daejun Park

, the data structure for the HPB
is initialized according to the device information.

A read I/O in the active sub-region where the map is cached is changed to
HPB READ by the HPB.

The HPB manages the L2P map using information received from the
device. For active sub-region, the HPB caches through ufshpb_map
request. For the in-active region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

 Documentation/ABI/testing/sysfs-driver-ufs |  162 ++
 drivers/scsi/ufs/Kconfig   |9 +
 drivers/scsi/ufs/Makefile  |1 +
 drivers/scsi/ufs/ufs-sysfs.c   |   22 +
 drivers/scsi/ufs/ufs.h |   54 +-
 drivers/scsi/ufs/ufshcd.c  |   74 +-
 drivers/scsi/ufs/ufshcd.h  |   29 +
 drivers/scsi/ufs/ufshpb.c  | 2394 
 drivers/scsi/ufs/ufshpb.h  |  276 +++
 9 files changed, 3019 insertions(+), 2 deletions(-)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

-- 
2.25.1

[PATCH v28 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-12 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  47 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   4 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  25 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 629 +++--
 drivers/scsi/ufs/ufshpb.h  |  66 ++-
 7 files changed, 700 insertions(+), 81 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index 528bf89fc98b..419adf450b89 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,45 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/flags/wb_enable
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the status of HPB.
+
+   == 
+   0  HPB is not enabled.
+   1  HPB is enab

[PATCH v28 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-03-12 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 256 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 257 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index e10984fd8d47..88dd0f34fa09 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2656,6 +2656,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 489c8b1ac580..201dc24d55b3 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,233 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v28 2/4] scsi: ufs: L2P map management for HPB read

2021-03-12 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1094 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1184 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 4eee7e31d08d..bfb84d2ba990 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -478,6 +478,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -488,6 +523,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 70a567ea7d5a..e10984fd8d47 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5021,6 +5021,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9241,6 +9244,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 1a72f6541510..489c8b1ac580 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v28 1/4] scsi: ufs: Introduce HPB feature

2021-03-12 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 167 ++
 9 files changed, 977 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..528bf89fc98b 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v28 0/4] scsi: ufs: Add Host Performance Booster Support

2021-03-12 Thread Daejun Park

ormation.

A read I/O in the active sub-region where the map is cached is changed to
HPB READ by the HPB.

The HPB manages the L2P map using information received from the
device. For active sub-region, the HPB caches through ufshpb_map
request. For the in-active region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong


Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

 Documentation/ABI/testing/sysfs-driver-ufs |  162 ++
 drivers/scsi/ufs/Kconfig   |9 +
 drivers/scsi/ufs/Makefile  |1 +
 drivers/scsi/ufs/ufs-sysfs.c   |   22 +
 drivers/scsi/ufs/ufs.h |   54 +-
 drivers/scsi/ufs/ufshcd.c  |   74 +-
 drivers/scsi/ufs/ufshcd.h  |   29 +
 drivers/scsi/ufs/ufshpb.c  | 2396 
 drivers/scsi/ufs/ufshpb.h  |  276 +++
 9 files changed, 3021 insertions(+), 2 deletions(-)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

-- 
2.25.1

[PATCH v27 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-12 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  47 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   4 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  25 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 627 +++--
 drivers/scsi/ufs/ufshpb.h  |  66 ++-
 7 files changed, 699 insertions(+), 80 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index 528bf89fc98b..419adf450b89 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,45 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  March 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/flags/wb_enable
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the status of HPB.
+
+   == 
+   0  HPB is not enabled.
+   1  HPB is enab

[PATCH v27 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-03-12 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 256 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 257 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index e10984fd8d47..88dd0f34fa09 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2656,6 +2656,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 489c8b1ac580..201dc24d55b3 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,233 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v27 2/4] scsi: ufs: L2P map management for HPB read

2021-03-12 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1094 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1184 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 4eee7e31d08d..bfb84d2ba990 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -478,6 +478,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -488,6 +523,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 70a567ea7d5a..e10984fd8d47 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5021,6 +5021,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9241,6 +9244,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 1a72f6541510..489c8b1ac580 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v27 1/4] scsi: ufs: Introduce HPB feature

2021-03-12 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 167 ++
 9 files changed, 977 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..528bf89fc98b 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  March 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v27 0/4] scsi: ufs: Add Host Performance Booster Support

2021-03-12 Thread Daejun Park

s cached is changed to
HPB READ by the HPB.

The HPB manages the L2P map using information received from the
device. For active sub-region, the HPB caches through ufshpb_map
request. For the in-active region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

 Documentation/ABI/testing/sysfs-driver-ufs |  162 ++
 drivers/scsi/ufs/Kconfig   |9 +
 drivers/scsi/ufs/Makefile  |1 +
 drivers/scsi/ufs/ufs-sysfs.c   |   22 +
 drivers/scsi/ufs/ufs.h |   54 +-
 drivers/scsi/ufs/ufshcd.c  |   74 +-
 drivers/scsi/ufs/ufshcd.h  |   29 +
 drivers/scsi/ufs/ufshpb.c  | 2396 
 drivers/scsi/ufs/ufshpb.h  |  276 +++
 9 files changed, 3021 insertions(+), 2 deletions(-)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

-- 
2.25.1

RE: Re: [PATCH v26 2/4] scsi: ufs: L2P map management for HPB read

2021-03-11 Thread Daejun Park

>>> > This is a patch for managing L2P map in HPB module.
>>> >
>>> > The HPB divides logical addresses into several regions. A region
>>> > consists
>>> > of several sub-regions. The sub-region is a basic unit where L2P
>>> > mapping is
>>> > managed. The driver loads L2P mapping data of each sub-region. The
>>> > loaded
>>> > sub-region is called active-state. The HPB driver unloads L2P mapping
>>> > data
>>> > as region unit. The unloaded region is called inactive-state.
>>> >
>>> > Sub-region/region candidates to be loaded and unloaded are delivered
>>> > from
>>> > the UFS device. The UFS device delivers the recommended active
>>> > sub-region
>>> > and inactivate region to the driver using sensedata.
>>> > The HPB module performs L2P mapping management on the host through the
>>> > delivered information.
>>> >
>>> > A pinned region is a pre-set regions on the UFS device that is always
>>> > activate-state.
>>> >
>>> > The data structure for map data request and L2P map uses mempool API,
>>> > minimizing allocation overhead while avoiding static allocation.
>>> >
>>> > The mininum size of the memory pool used in the HPB is implemented
>>> > as a module parameter, so that it can be configurable by the user.
>>> >
>>> > To gurantee a minimum memory pool size of 4MB:
>>> > ufshpb_host_map_kbytes=4096
>>> >
>>> > The map_work manages active/inactive by 2 "to-do" lists.
>>> > Each hpb lun maintains 2 "to-do" lists:
>>> >   hpb->lh_inact_rgn - regions to be inactivated, and
>>> >   hpb->lh_act_srgn - subregions to be activated
>>> > Those lists are maintained on IO completion.
>>> >
>>> > Reviewed-by: Bart Van Assche 
>>> > Reviewed-by: Can Guo 
>>> > Acked-by: Avri Altman 
>>> > Tested-by: Bean Huo 
>>> > Signed-off-by: Daejun Park 
>>> > ---
>>> >  drivers/scsi/ufs/ufs.h|   36 ++
>>> >  drivers/scsi/ufs/ufshcd.c |4 +
>>> >  drivers/scsi/ufs/ufshpb.c | 1091 -
>>> >  drivers/scsi/ufs/ufshpb.h |   65 +++
>>> >  4 files changed, 1181 insertions(+), 15 deletions(-)
>>> >
>>> > diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
>>> > index 65563635e20e..957763db1006 100644
>>> > --- a/drivers/scsi/ufs/ufs.h
>>> > +++ b/drivers/scsi/ufs/ufs.h
>>> > @@ -472,6 +472,41 @@ struct utp_cmd_rsp {
>>> >  u8 sense_data[UFS_SENSE_SIZE];
>>> >  };
>>> >
>>> > +struct ufshpb_active_field {
>>> > +__be16 active_rgn;
>>> > +__be16 active_srgn;
>>> > +};
>>> > +#define HPB_ACT_FIELD_SIZE 4
>>> > +
>>> > +/**
>>> > + * struct utp_hpb_rsp - Response UPIU structure
>>> > + * @residual_transfer_count: Residual transfer count DW-3
>>> > + * @reserved1: Reserved double words DW-4 to DW-7
>>> > + * @sense_data_len: Sense data length DW-8 U16
>>> > + * @desc_type: Descriptor type of sense data
>>> > + * @additional_len: Additional length of sense data
>>> > + * @hpb_op: HPB operation type
>>> > + * @lun: LUN of response UPIU
>>> > + * @active_rgn_cnt: Active region count
>>> > + * @inactive_rgn_cnt: Inactive region count
>>> > + * @hpb_active_field: Recommended to read HPB region and subregion
>>> > + * @hpb_inactive_field: To be inactivated HPB region and subregion
>>> > + */
>>> > +struct utp_hpb_rsp {
>>> > +__be32 residual_transfer_count;
>>> > +__be32 reserved1[4];
>>> > +__be16 sense_data_len;
>>> > +u8 desc_type;
>>> > +u8 additional_len;
>>> > +u8 hpb_op;
>>> > +u8 lun;
>>> > +u8 active_rgn_cnt;
>>> > +u8 inactive_rgn_cnt;
>>> > +struct ufshpb_active_field hpb_active_field[2];
>>> > +__be16 hpb_inactive_field[2];
>>> > +};
>>> > +#define UTP_HPB_RSP_SIZE 40
>>> > +
>>> >  /**
>>> >   * struct utp_upiu_rsp - general upiu response structure
>>> >   * @header: UPIU header structure DW-0 to DW-2
>>> >

RE: Re: [PATCH v26 2/4] scsi: ufs: L2P map management for HPB read

2021-03-11 Thread Daejun Park

> > This is a patch for managing L2P map in HPB module.
> > 
> > The HPB divides logical addresses into several regions. A region 
> > consists
> > of several sub-regions. The sub-region is a basic unit where L2P 
> > mapping is
> > managed. The driver loads L2P mapping data of each sub-region. The 
> > loaded
> > sub-region is called active-state. The HPB driver unloads L2P mapping 
> > data
> > as region unit. The unloaded region is called inactive-state.
> > 
> > Sub-region/region candidates to be loaded and unloaded are delivered 
> > from
> > the UFS device. The UFS device delivers the recommended active 
> > sub-region
> > and inactivate region to the driver using sensedata.
> > The HPB module performs L2P mapping management on the host through the
> > delivered information.
> > 
> > A pinned region is a pre-set regions on the UFS device that is always
> > activate-state.
> > 
> > The data structure for map data request and L2P map uses mempool API,
> > minimizing allocation overhead while avoiding static allocation.
> > 
> > The mininum size of the memory pool used in the HPB is implemented
> > as a module parameter, so that it can be configurable by the user.
> > 
> > To gurantee a minimum memory pool size of 4MB: 
> > ufshpb_host_map_kbytes=4096
> > 
> > The map_work manages active/inactive by 2 "to-do" lists.
> > Each hpb lun maintains 2 "to-do" lists:
> >   hpb->lh_inact_rgn - regions to be inactivated, and
> >   hpb->lh_act_srgn - subregions to be activated
> > Those lists are maintained on IO completion.
> > 
> > Reviewed-by: Bart Van Assche 
> > Reviewed-by: Can Guo 
> > Acked-by: Avri Altman 
> > Tested-by: Bean Huo 
> > Signed-off-by: Daejun Park 
> > ---
> >  drivers/scsi/ufs/ufs.h|   36 ++
> >  drivers/scsi/ufs/ufshcd.c |4 +
> >  drivers/scsi/ufs/ufshpb.c | 1091 -
> >  drivers/scsi/ufs/ufshpb.h |   65 +++
> >  4 files changed, 1181 insertions(+), 15 deletions(-)
> > 
> > diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
> > index 65563635e20e..957763db1006 100644
> > --- a/drivers/scsi/ufs/ufs.h
> > +++ b/drivers/scsi/ufs/ufs.h
> > @@ -472,6 +472,41 @@ struct utp_cmd_rsp {
> >  u8 sense_data[UFS_SENSE_SIZE];
> >  };
> > 
> > +struct ufshpb_active_field {
> > +__be16 active_rgn;
> > +__be16 active_srgn;
> > +};
> > +#define HPB_ACT_FIELD_SIZE 4
> > +
> > +/**
> > + * struct utp_hpb_rsp - Response UPIU structure
> > + * @residual_transfer_count: Residual transfer count DW-3
> > + * @reserved1: Reserved double words DW-4 to DW-7
> > + * @sense_data_len: Sense data length DW-8 U16
> > + * @desc_type: Descriptor type of sense data
> > + * @additional_len: Additional length of sense data
> > + * @hpb_op: HPB operation type
> > + * @lun: LUN of response UPIU
> > + * @active_rgn_cnt: Active region count
> > + * @inactive_rgn_cnt: Inactive region count
> > + * @hpb_active_field: Recommended to read HPB region and subregion
> > + * @hpb_inactive_field: To be inactivated HPB region and subregion
> > + */
> > +struct utp_hpb_rsp {
> > +__be32 residual_transfer_count;
> > +__be32 reserved1[4];
> > +__be16 sense_data_len;
> > +u8 desc_type;
> > +u8 additional_len;
> > +u8 hpb_op;
> > +u8 lun;
> > +u8 active_rgn_cnt;
> > +u8 inactive_rgn_cnt;
> > +struct ufshpb_active_field hpb_active_field[2];
> > +__be16 hpb_inactive_field[2];
> > +};
> > +#define UTP_HPB_RSP_SIZE 40
> > +
> >  /**
> >   * struct utp_upiu_rsp - general upiu response structure
> >   * @header: UPIU header structure DW-0 to DW-2
> > @@ -482,6 +517,7 @@ struct utp_upiu_rsp {
> >  struct utp_upiu_header header;
> >  union {
> >  struct utp_cmd_rsp sr;
> > +struct utp_hpb_rsp hr;
> >  struct utp_upiu_query qr;
> >  };
> >  };
> > diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
> > index 49b3d5d24fa6..5852ff44c3cc 100644
> > --- a/drivers/scsi/ufs/ufshcd.c
> > +++ b/drivers/scsi/ufs/ufshcd.c
> > @@ -5021,6 +5021,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba,
> > struct ufshcd_lrb *lrbp)
> >   */
> >  pm_runtime_get_noresume(hba

RE: Re: [PATCH v26 2/4] scsi: ufs: L2P map management for HPB read

2021-03-11 Thread Daejun Park

> > The HPB divides logical addresses into several regions. A region 
> > consists
> > of several sub-regions. The sub-region is a basic unit where L2P 
> > mapping is
> > managed. The driver loads L2P mapping data of each sub-region. The 
> > loaded
> > sub-region is called active-state. The HPB driver unloads L2P mapping 
> > data
> > as region unit. The unloaded region is called inactive-state.
> > 
> > Sub-region/region candidates to be loaded and unloaded are delivered 
> > from
> > the UFS device. The UFS device delivers the recommended active 
> > sub-region
> > and inactivate region to the driver using sensedata.
> > The HPB module performs L2P mapping management on the host through the
> > delivered information.
> > 
> > A pinned region is a pre-set regions on the UFS device that is always
> > activate-state.
> > 
> > The data structure for map data request and L2P map uses mempool API,
> > minimizing allocation overhead while avoiding static allocation.
> > 
> > The mininum size of the memory pool used in the HPB is implemented
> > as a module parameter, so that it can be configurable by the user.
> > 
> > To gurantee a minimum memory pool size of 4MB: 
> > ufshpb_host_map_kbytes=4096
> > 
> > The map_work manages active/inactive by 2 "to-do" lists.
> > Each hpb lun maintains 2 "to-do" lists:
> >   hpb->lh_inact_rgn - regions to be inactivated, and
> >   hpb->lh_act_srgn - subregions to be activated
> > Those lists are maintained on IO completion.
> > 
> > Reviewed-by: Bart Van Assche 
> > Reviewed-by: Can Guo 
> > Acked-by: Avri Altman 
> > Tested-by: Bean Huo 
> > Signed-off-by: Daejun Park 
> > ---
> >  drivers/scsi/ufs/ufs.h|   36 ++
> >  drivers/scsi/ufs/ufshcd.c |4 +
> >  drivers/scsi/ufs/ufshpb.c | 1091 -
> >  drivers/scsi/ufs/ufshpb.h |   65 +++
> >  4 files changed, 1181 insertions(+), 15 deletions(-)
> > 
> > diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
> > index 65563635e20e..957763db1006 100644
> > --- a/drivers/scsi/ufs/ufs.h
> > +++ b/drivers/scsi/ufs/ufs.h
> > @@ -472,6 +472,41 @@ struct utp_cmd_rsp {
> >  u8 sense_data[UFS_SENSE_SIZE];
> >  };
> > ...
> > +/*
> > + * This function will parse recommended active subregion information 
> > in sense
> > + * data field of response UPIU with SAM_STAT_GOOD state.
> > + */
> > +void ufshpb_rsp_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
> > +{
> > +struct ufshpb_lu *hpb = ufshpb_get_hpb_data(lrbp->cmd->device);
> > +struct utp_hpb_rsp *rsp_field = >ucd_rsp_ptr->hr;
> > +int data_seg_len;
> > +
> > +if (unlikely(lrbp->lun != rsp_field->lun)) {
> > +struct scsi_device *sdev;
> > +bool found = false;
> > +
> > +__shost_for_each_device(sdev, hba->host) {
> > +hpb = ufshpb_get_hpb_data(sdev);
> > +
> > +if (!hpb)
> > +continue;
> > +
> > +if (rsp_field->lun == hpb->lun) {
> > +found = true;
> > +break;
> > +}
> > +}
> > +
> > +if (!found)
> > +return;
> > +}
> > +
> > +if (!hpb)
> > +return;
> > +
> > +if ((ufshpb_get_state(hpb) != HPB_PRESENT) &&
> > +(ufshpb_get_state(hpb) != HPB_SUSPEND)) {
> > +dev_notice(>sdev_ufs_lu->sdev_dev,
> > +   "%s: ufshpb state is not PRESENT/SUSPEND\n",
> > +   __func__);
>  
> Please mute these prints before hpb is fully initilized, otherwise
> there can be tons of these prints during bootup. Say set a flag in
> ufshpb_hpb_lu_prepared() and check for that flag - just a rough idea.

OK, I will change it.

Thanks,
Daejun

RE: Re: [PATCH v26 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-04 Thread Daejun Park

Hi Bean,

> > +
> > +static inline int ufshpb_get_read_id(struct ufshpb_lu *hpb)
> > +{
> > +   if (++hpb->cur_read_id >= MAX_HPB_READ_ID)
> > +   hpb->cur_read_id = 0;
> > +   return hpb->cur_read_id;
> > +}
> > +
> > +static int ufshpb_execute_pre_req(struct ufshpb_lu *hpb, struct
> > scsi_cmnd *cmd,
> > + struct ufshpb_req *pre_req, int
> > read_id)
> > +{
> > +   struct scsi_device *sdev = cmd->device;
> > +   struct request_queue *q = sdev->request_queue;
> > +   struct request *req;
> > +   struct scsi_request *rq;
> > +   struct bio *bio = pre_req->bio;
> > +
> > +   pre_req->hpb = hpb;
> > +   pre_req->wb.lpn = sectors_to_logical(cmd->device,
> > +blk_rq_pos(cmd-
> > >request));
> > +   pre_req->wb.len = sectors_to_logical(cmd->device,
> > +blk_rq_sectors(cmd-
> > >request));
> > +   if (ufshpb_pre_req_add_bio_page(hpb, q, pre_req))
> > +   return -ENOMEM;
> > +
> > +   req = pre_req->req;
> > +
> > +   /* 1. request setup */
> > +   blk_rq_append_bio(req, );
> > +   req->rq_disk = NULL;
> > +   req->end_io_data = (void *)pre_req;
> > +   req->end_io = ufshpb_pre_req_compl_fn;
> > +
> > +   /* 2. scsi_request setup */
> > +   rq = scsi_req(req);
> > +   rq->retries = 1;
> > +
> > +   ufshpb_set_write_buf_cmd(rq->cmd, pre_req->wb.lpn, pre_req-
> > >wb.len,
> > +read_id);
> > +   rq->cmd_len = scsi_command_size(rq->cmd);
> > +
> > +   if (blk_insert_cloned_request(q, req) != BLK_STS_OK)
> > +   return -EAGAIN;
> > +
> > +   hpb->stats.pre_req_cnt++;
> > +
> > +   return 0;
> > +}
> > +
> > +static int ufshpb_issue_pre_req(struct ufshpb_lu *hpb, struct
> > scsi_cmnd *cmd,
> > +   int *read_id)
> > +{
> > +   struct ufshpb_req *pre_req;
> > +   struct request *req = NULL;
> > +   struct bio *bio = NULL;
> > +   unsigned long flags;
> > +   int _read_id;
> > +   int ret = 0;
> > +
> > +   req = blk_get_request(cmd->device->request_queue,
> > + REQ_OP_SCSI_OUT | REQ_SYNC,
> > BLK_MQ_REQ_NOWAIT);
> > +   if (IS_ERR(req))
> > +   return -EAGAIN;
> > +
> > +   bio = bio_alloc(GFP_ATOMIC, 1);
> > +   if (!bio) {
> > +   blk_put_request(req);
> > +   return -EAGAIN;
> > +   }
> > +
> > +   spin_lock_irqsave(>rgn_state_lock, flags);
> > +   pre_req = ufshpb_get_pre_req(hpb);
> > +   if (!pre_req) {
> > +   ret = -EAGAIN;
> > +   goto unlock_out;
> > +   }
> > +   _read_id = ufshpb_get_read_id(hpb);
> > +   spin_unlock_irqrestore(>rgn_state_lock, flags);
> > +
> > +   pre_req->req = req;
> > +   pre_req->bio = bio;
> > +
> > +   ret = ufshpb_execute_pre_req(hpb, cmd, pre_req, _read_id);
> > +   if (ret)
> > +   goto free_pre_req;
> > +
> > +   *read_id = _read_id;
> > +
> > +   return ret;
> > +free_pre_req:
> > +   spin_lock_irqsave(>rgn_state_lock, flags);
> > +   ufshpb_put_pre_req(hpb, pre_req);
> > +unlock_out:
> > +   spin_unlock_irqrestore(>rgn_state_lock, flags);
> > +   bio_put(bio);
> > +   blk_put_request(req);
> > +   return ret;
> > +}
> > +
> >  /*
> >   * This function will set up HPB read command using host-side L2P
> > map data.
> > - * In HPB v1.0, maximum size of HPB read command is 4KB.
> >   */
> > -void ufshpb_prep(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
> > +int ufshpb_prep(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
> >  {
> > struct ufshpb_lu *hpb;
> > struct ufshpb_region *rgn;
> > @@ -291,26 +560,27 @@ void ufshpb_prep(struct ufs_hba *hba, struct
> > ufshcd_lrb *lrbp)
> > u64 ppn;
> > unsigned long flags;
> > int transfer_len, rgn_idx, srgn_idx, srgn_offset;
> > +   int read_id = 0;
> > int err = 0;
> >  
> > hpb = ufshpb_get_hpb_data(cmd->device);
> > if (!hpb)
> > -   return;
> > +   return -ENODEV;
> >  
> > if (ufshpb_get_state(hpb) != HPB_PRESENT) {
> > dev_notice(>sdev_ufs_lu->sdev_dev,
> >"%s: ufshpb state is not PRESENT",
> > __func__);
> > -   return;
> > +   return -ENODEV;
> > }
> >  
> > if (!ufshpb_is_write_or_discard_cmd(cmd) &&
> > !ufshpb_is_read_cmd(cmd))
> > -   return;
> > +   return 0;
> >  
> > transfer_len = sectors_to_logical(cmd->device,
> >   blk_rq_sectors(cmd-
> > >request));
> > if (unlikely(!transfer_len))
> > -   return;
> > +   return 0;
> >  
> > lpn = sectors_to_logical(cmd->device, blk_rq_pos(cmd-
> > >request));
> >

RE: Re: [PATCH v26 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-04 Thread Daejun Park

Hi, Can Guo

> > diff --git a/drivers/scsi/ufs/ufs-sysfs.c 
> > b/drivers/scsi/ufs/ufs-sysfs.c
> > index 2546e7a1ac4f..00fb519406cf 100644
> > --- a/drivers/scsi/ufs/ufs-sysfs.c
> > +++ b/drivers/scsi/ufs/ufs-sysfs.c
> > @@ -841,6 +841,7 @@ out:
> > \
> >  static DEVICE_ATTR_RO(_name)
> > 
> >  UFS_ATTRIBUTE(boot_lun_enabled, _BOOT_LU_EN);
> > +UFS_ATTRIBUTE(max_data_size_hpb_single_cmd, _MAX_HPB_SINGLE_CMD);
> >  UFS_ATTRIBUTE(current_power_mode, _POWER_MODE);
> >  UFS_ATTRIBUTE(active_icc_level, _ACTIVE_ICC_LVL);
> >  UFS_ATTRIBUTE(ooo_data_enabled, _OOO_DATA_EN);
> > @@ -864,6 +865,7 @@ UFS_ATTRIBUTE(wb_cur_buf, _CURR_WB_BUFF_SIZE);
> > 
> >  static struct attribute *ufs_sysfs_attributes[] = {
> >  _attr_boot_lun_enabled.attr,
> > +_attr_max_data_size_hpb_single_cmd.attr,
> >  _attr_current_power_mode.attr,
> >  _attr_active_icc_level.attr,
> >  _attr_ooo_data_enabled.attr,
> > diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
> > index 957763db1006..e0b748777a1b 100644
> > --- a/drivers/scsi/ufs/ufs.h
> > +++ b/drivers/scsi/ufs/ufs.h
> > @@ -123,12 +123,13 @@ enum flag_idn {
> >  QUERY_FLAG_IDN_WB_BUFF_FLUSH_EN = 0x0F,
> >  QUERY_FLAG_IDN_WB_BUFF_FLUSH_DURING_HIBERN8 = 0x10,
> >  QUERY_FLAG_IDN_HPB_RESET= 0x11,
> > +QUERY_FLAG_IDN_HPB_EN= 0x12,
>  
> Also add this flag to sysfs?

Sure, I will do.

Thanks,
Daejun

RE: Re: [PATCH v26 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-04 Thread Daejun Park

Hi Bean,

> > +
> > +static inline void ufshpb_put_pre_req(struct ufshpb_lu *hpb,
> > + struct ufshpb_req *pre_req)
> > +{
> > +   pre_req->req = NULL;
> > +   pre_req->bio = NULL;
> > +   list_add_tail(_req->list_req, >lh_pre_req_free);
> > +   hpb->num_inflight_pre_req--;
> > +}
> > +
> > +static void ufshpb_pre_req_compl_fn(struct request *req,
> > blk_status_t error)
> > +{
> > +   struct ufshpb_req *pre_req = (struct ufshpb_req *)req-
> > >end_io_data;
> > +   struct ufshpb_lu *hpb = pre_req->hpb;
> > +   unsigned long flags;
> > +   struct scsi_sense_hdr sshdr;
> > +
> > +   if (error) {
> > +   dev_err(>sdev_ufs_lu->sdev_dev, "block status
> > %d", error);
> > +   scsi_normalize_sense(pre_req->sense,
> > SCSI_SENSE_BUFFERSIZE,
> > +);
> > +   dev_err(>sdev_ufs_lu->sdev_dev,
> > +   "code %x sense_key %x asc %x ascq %x",
> > +   sshdr.response_code,
> > +   sshdr.sense_key, sshdr.asc, sshdr.ascq);
> > +   dev_err(>sdev_ufs_lu->sdev_dev,
> > +   "byte4 %x byte5 %x byte6 %x additional_len
> > %x",
> > +   sshdr.byte4, sshdr.byte5,
> > +   sshdr.byte6, sshdr.additional_length);
> > +   }
>  
>  
> How can you print out sense_key and sense code here? sense code will
> not be copied to pre_req->sense. you should directly use
> scsi_request->sense or let pre_req->sense point to scsi_request->sense.

OK, I will fix it.

> You update the new version patch so quickly. In another word, I am
> wondering if you tested your patch before submitting?

I will check more carefully for the next patch.

Thanks,
Daejun

[PATCH v26 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-03 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  35 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   2 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  22 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 622 +++--
 drivers/scsi/ufs/ufshpb.h  |  67 ++-
 7 files changed, 679 insertions(+), 79 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index bf5cb8846de1..0017eaf89cbe 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,33 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
diff --git a/drivers/scsi/ufs/ufs-sysfs.c b/drivers/scsi/ufs/ufs-sysfs.c
index 2546e7a1ac4f..00fb519406cf 100644
--- a/drivers/scsi/ufs/ufs-sysfs.c
+++ b/drivers/scsi/ufs/ufs-sysfs.c
@@ -841,6 +841,7 @@ out:
\
 static DEVICE_ATTR_RO(_name)
 
 UFS_ATTRIB

[PATCH v26 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-03-03 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 253 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 254 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 5852ff44c3cc..851c01a26207 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2656,6 +2656,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 8abadb0e010a..c75a6816a03f 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,230 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v26 2/4] scsi: ufs: L2P map management for HPB read

2021-03-03 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1091 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1181 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 65563635e20e..957763db1006 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -472,6 +472,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -482,6 +517,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 49b3d5d24fa6..5852ff44c3cc 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5021,6 +5021,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9221,6 +9224,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 1a72f6541510..8abadb0e010a 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v26 1/4] scsi: ufs: Introduce HPB feature

2021-03-03 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 166 ++
 9 files changed, 976 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..bf5cb8846de1 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v26 0/4] scsi: ufs: Add Host Performance Booster Support

2021-03-03 Thread Daejun Park

e region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

 Documentation/ABI/testing/sysfs-driver-ufs |  150 ++
 drivers/scsi/ufs/Kconfig   |9 +
 drivers/scsi/ufs/Makefile  |1 +
 drivers/scsi/ufs/ufs-sysfs.c   |   20 +
 drivers/scsi/ufs/ufs.h |   54 +-
 drivers/scsi/ufs/ufshcd.c  |   73 +-
 drivers/scsi/ufs/ufshcd.h  |   29 +
 drivers/scsi/ufs/ufshpb.c  | 2385 
 drivers/scsi/ufs/ufshpb.h  |  276 +++
 9 files changed, 2995 insertions(+), 2 deletions(-)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

-- 
2.25.1

RE: Re: [PATCH v25 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-03 Thread Daejun Park

> > @@ -1812,8 +2307,9 @@ void ufshpb_get_geo_info(struct ufs_hba *hba, u8 
> > *geo_buf)
> >  void ufshpb_get_dev_info(struct ufs_hba *hba, u8 *desc_buf)
> >  {
> >  struct ufshpb_dev_info *hpb_dev_info = >ufshpb_dev;
> > -int version;
> > +int version, ret;
> >  u8 hpb_mode;
> > +u32 max_hpb_sigle_cmd = 0;
>  
> Maybe max_hpb_single_cmd?
>  
> > 
> >  hpb_mode = desc_buf[DEVICE_DESC_PARAM_HPB_CONTROL];
> >  if (hpb_mode == HPB_HOST_CONTROL) {
> > @@ -1824,13 +2320,27 @@ void ufshpb_get_dev_info(struct ufs_hba *hba,
> > u8 *desc_buf)
> >  }
> > 
> >  version = get_unaligned_be16(desc_buf + DEVICE_DESC_PARAM_HPB_VER);
> > -if (version != HPB_SUPPORT_VERSION) {
> > +if ((version != HPB_SUPPORT_VERSION) &&
> > +(version != HPB_SUPPORT_LEGACY_VERSION)) {
> >  dev_err(hba->dev, "%s: HPB %x version is not supported.\n",
> >  __func__, version);
> >  hpb_dev_info->hpb_disabled = true;
> >  return;
> >  }
> > 
> > +if (version == HPB_SUPPORT_LEGACY_VERSION)
> > +hpb_dev_info->is_legacy = true;
> > +
> > +pm_runtime_get_sync(hba->dev);
> > +ret = ufshcd_query_attr_retry(hba, UPIU_QUERY_OPCODE_READ_ATTR,
> > +QUERY_ATTR_IDN_MAX_HPB_SINGLE_CMD, 0, 0, 
> > _hpb_sigle_cmd);
>  
> Same here
>  
> > +pm_runtime_put_sync(hba->dev);
> > +
> > +if (ret)
> > +dev_err(hba->dev, "%s: idn: read max size of single hpb 
> > cmd query
> > request failed",
> > +__func__);
> > +hpb_dev_info->max_hpb_single_cmd = max_hpb_sigle_cmd;
>  
> Same here
>  

Done.

Thanks,
Daejun

RE: Re: [PATCH v25 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-03 Thread Daejun Park

> > +bool ufshpb_is_legacy(struct ufs_hba *hba)
> > +{
> > +return hba->ufshpb_dev.is_legacy;
> > +}
> > +
> >  static struct ufshpb_lu *ufshpb_get_hpb_data(struct scsi_device *sdev)
> >  {
> >  return sdev->hostdata;
> > @@ -64,9 +69,19 @@ static bool ufshpb_is_write_or_discard_cmd(struct
> > scsi_cmnd *cmd)
> > op_is_discard(req_op(cmd->request));
> >  }
> > 
> > -static bool ufshpb_is_support_chunk(int transfer_len)
> > +static bool ufshpb_is_support_chunk(struct ufshpb_lu *hpb, int 
> > transfer_len)
> >  {
> > -return transfer_len <= HPB_MULTI_CHUNK_HIGH;
> > +return transfer_len <= hpb->pre_req_max_tr_len;
>  
> In the case of HPB1.0, this is wrong - you are allowing transfer_len > 1 
> for HPB1.0 devices.
>  
> Can Guo.

OK, I will check whether it is HPB 1.0 or not.

Thanks,
Daejun

RE: Re: [PATCH v25 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-03 Thread Daejun Park

> > 
> >  static void __ufshpb_evict_region(struct ufshpb_lu *hpb,
> >struct ufshpb_region *rgn)
> >  {
> > @@ -1209,6 +1579,16 @@ static void ufshpb_lu_parameter_init(struct
> > ufs_hba *hba,
> >  u32 entries_per_rgn;
> >  u64 rgn_mem_size, tmp;
> >  
> > +/* for pre_req */
> > +if (hpb_dev_info->max_hpb_single_cmd)
> > +hpb->pre_req_min_tr_len = hpb_dev_info-
> > >max_hpb_single_cmd;
> > +else
> > +hpb->pre_req_min_tr_len = HPB_MULTI_CHUNK_LOW;
>  
>  
> Here is not correct. according to Spec:
>  
> The size is calculated as ( bMAX_DATA_SIZE_FOR_HPB_SINGLE_CMD +1 )*4KB.
> 00h: 4KB
> 01h: 8KB
> 02h: 12KB
> 03h: 16KB
> ...
> FEh: 1020KB
> FFh: 1024KB
>  
> so, here if hpb_dev_info->max_hpb_single_cmd is 0x00, means 4KB, not
> 36KB.

OK,
 
> > +hpb->pre_req_max_tr_len = max(HPB_MULTI_CHUNK_HIGH,
> > +  hpb->pre_req_min_tr_len);
> > +
> > 
> >  out:
> >  /* All LUs are initialized */
> >  if (atomic_dec_and_test(>ufshpb_dev.slave_conf_cnt))
> > @@ -1812,8 +2307,9 @@ void ufshpb_get_geo_info(struct ufs_hba *hba,
> > u8 *geo_buf)
> >  void ufshpb_get_dev_info(struct ufs_hba *hba, u8 *desc_buf)
> >  {
> >  struct ufshpb_dev_info *hpb_dev_info = >ufshpb_dev;
> > -int version;
> > +int version, ret;
> >  u8 hpb_mode;
> > +u32 max_hpb_sigle_cmd = 0;
> >  
> >  hpb_mode = desc_buf[DEVICE_DESC_PARAM_HPB_CONTROL];
> >  if (hpb_mode == HPB_HOST_CONTROL) {
> > @@ -1824,13 +2320,27 @@ void ufshpb_get_dev_info(struct ufs_hba *hba,
> > u8 *desc_buf)
> >  }
> >  
> >  version = get_unaligned_be16(desc_buf +
> > DEVICE_DESC_PARAM_HPB_VER);
> > -if (version != HPB_SUPPORT_VERSION) {
> > +if ((version != HPB_SUPPORT_VERSION) &&
> > +(version != HPB_SUPPORT_LEGACY_VERSION)) {
> >  dev_err(hba->dev, "%s: HPB %x version is not
> > supported.\n",
> >  __func__, version);
> >  hpb_dev_info->hpb_disabled = true;
> >  return;
> >  }
> >  
> > +if (version == HPB_SUPPORT_LEGACY_VERSION)
> > +hpb_dev_info->is_legacy = true;
> > +
> > +pm_runtime_get_sync(hba->dev);
> > +ret = ufshcd_query_attr_retry(hba, UPIU_QUERY_OPCODE_READ_ATTR,
> > +QUERY_ATTR_IDN_MAX_HPB_SINGLE_CMD, 0, 0,
> > _hpb_sigle_cmd);
> > +pm_runtime_put_sync(hba->dev);
> > +
> > +if (ret)
> > +dev_err(hba->dev, "%s: idn: read max size of single hpb
> > cmd query request failed",
> > +__func__);
> > +hpb_dev_info->max_hpb_single_cmd = max_hpb_sigle_cmd;
> > +
>  
> Here you didn't add 1, if you read out the
> QUERY_ATTR_IDN_MAX_HPB_SINGLE_CMD == 7, means device can support
> maximum HPB Data size for using single HPB command is 7+1
> ((7+1)*4=32KB), not 7.

OK, Done.

Thanks,
Daejun

RE: Re: [PATCH v25 4/4] scsi: ufs: Add HPB 2.0 support

2021-03-03 Thread Daejun Park

> > +
> > +/*
> > + * In this driver, WRITE_BUFFER CMD support 36KB (len=9) ~ 512KB
> > (len=128) as
> > + * default. It is possible to change range of transfer_len through
> > sysfs.
> > + */
> > +static inline bool ufshpb_is_required_wb(struct ufshpb_lu *hpb, int
> > len)
> > +{
> > +   return (len >= hpb->pre_req_min_tr_len &&
>  
> Here is wrong, should be : len > hpb->pre_req_min_tr_len.

OK, Done.

Thanks,
Daejun

RE: RE: [PATCH v4 7/9] scsi: ufshpb: Add "Cold" regions timer

2021-03-02 Thread Daejun Park

Hi Avri,
> > 
> > Hi Avri,
> > 
> > > +static void ufshpb_read_to_handler(struct work_struct *work)
> > > +{
> > > +struct delayed_work *dwork = to_delayed_work(work);
> > > +struct ufshpb_lu *hpb;
> > > +struct victim_select_info *lru_info;
> > > +struct ufshpb_region *rgn;
> > > +unsigned long flags;
> > > +LIST_HEAD(expired_list);
> > > +
> > > +hpb = container_of(dwork, struct ufshpb_lu, ufshpb_read_to_work);
> > > +
> > > +if (test_and_set_bit(TIMEOUT_WORK_PENDING, 
> > >work_data_bits))
> > > +return;
> > > +
> > > +spin_lock_irqsave(>rgn_state_lock, flags);
> > > +
> > > +lru_info = >lru_info;
> > > +
> > > +list_for_each_entry(rgn, _info->lh_lru_rgn, list_lru_rgn) {
> > > +bool timedout = ktime_after(ktime_get(), 
> > > rgn->read_timeout);
> > > +
> > > +if (timedout) {
> > 
> > It is important not just to check the timeout, but how much time has passed.
> > If the time exceeded is twice the threshold, the read_timeout_expiries
> > value should be reduced by 2 instead of 1.
> Theoretically this shouldn't happened.
> Please note that POLLING_INTERVAL_MS << READ_TO_MS.
> Better add appropriate check when making those configurable.

OK, I agree.

Thanks,
Daejun

>  
> Thanks,
> Avri
> > 
> > > +rgn->read_timeout_expiries--;
> > 
> > Thanks,
> > Daejun
>  
>  
>  
>

RE: [PATCH v4 3/9] scsi: ufshpb: Add region's reads counter

2021-03-02 Thread Daejun Park

Hi Avri,

> +static void ufshpb_normalization_work_handler(struct work_struct *work)
> +{
> +struct ufshpb_lu *hpb;
> +int rgn_idx;
> +
> +hpb = container_of(work, struct ufshpb_lu, 
> ufshpb_normalization_work);
> +
> +for (rgn_idx = 0; rgn_idx < hpb->rgns_per_lu; rgn_idx++) {
> +struct ufshpb_region *rgn = hpb->rgn_tbl + rgn_idx;

*HERE*
> +if (rgn->reads) {
> +unsigned long flags;
> +
> +spin_lock_irqsave(>rgn_lock, flags);

I thinks this lock should protect rgn->reads when it is accessed.

> +rgn->reads = (rgn->reads >> 1);
> +spin_unlock_irqrestore(>rgn_lock, flags);
> +}
*HERE*

> +
> +if (rgn->rgn_state != HPB_RGN_ACTIVE || rgn->reads)
> +continue;
> +
> +/* if region is active but has no reads - inactivate it */
> +spin_lock(>rsp_list_lock);
> +ufshpb_update_inactive_info(hpb, rgn->rgn_idx);
> +spin_unlock(>rsp_list_lock);
> +}
> +
> +clear_bit(WORK_PENDING, >work_data_bits);

Why we use work_data_bits? It may be checked by worker API.

Thanks,
Daejun

RE: [PATCH v4 7/9] scsi: ufshpb: Add "Cold" regions timer

2021-03-02 Thread Daejun Park

Hi Avri,

> +static void ufshpb_read_to_handler(struct work_struct *work)
> +{
> +struct delayed_work *dwork = to_delayed_work(work);
> +struct ufshpb_lu *hpb;
> +struct victim_select_info *lru_info;
> +struct ufshpb_region *rgn;
> +unsigned long flags;
> +LIST_HEAD(expired_list);
> +
> +hpb = container_of(dwork, struct ufshpb_lu, ufshpb_read_to_work);
> +
> +if (test_and_set_bit(TIMEOUT_WORK_PENDING, >work_data_bits))
> +return;
> +
> +spin_lock_irqsave(>rgn_state_lock, flags);
> +
> +lru_info = >lru_info;
> +
> +list_for_each_entry(rgn, _info->lh_lru_rgn, list_lru_rgn) {
> +bool timedout = ktime_after(ktime_get(), rgn->read_timeout);
> +
> +if (timedout) {

It is important not just to check the timeout, but how much time has passed.
If the time exceeded is twice the threshold, the read_timeout_expiries
value should be reduced by 2 instead of 1.

> +rgn->read_timeout_expiries--;

Thanks,
Daejun

RE: [PATCH v4 6/9] scsi: ufshpb: Add hpb dev reset response

2021-03-02 Thread Daejun Park

Hi Avri,

> diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
> index cf704b82e72a..f33aa28e0a0a 100644
> --- a/drivers/scsi/ufs/ufshpb.c
> +++ b/drivers/scsi/ufs/ufshpb.c
> @@ -642,7 +642,8 @@ int ufshpb_prep(struct ufs_hba *hba, struct ufshcd_lrb 
> *lrbp)
>  if (rgn->reads == ACTIVATION_THRESHOLD)
>  activate = true;
>  spin_unlock_irqrestore(>rgn_lock, flags);
> -if (activate) {
> +if (activate ||
> +test_and_clear_bit(RGN_FLAG_UPDATE, >rgn_flags)) {

How about merge rgn->rgn_flags to rgn_state?

Thanks,
Daejun

[PATCH v25 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-02-25 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 253 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 254 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 5852ff44c3cc..851c01a26207 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2656,6 +2656,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 8abadb0e010a..c75a6816a03f 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,230 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v25 4/4] scsi: ufs: Add HPB 2.0 support

2021-02-25 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  35 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   2 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  22 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 624 +++--
 drivers/scsi/ufs/ufshpb.h  |  67 ++-
 7 files changed, 681 insertions(+), 79 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index bf5cb8846de1..0017eaf89cbe 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,33 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
diff --git a/drivers/scsi/ufs/ufs-sysfs.c b/drivers/scsi/ufs/ufs-sysfs.c
index 2546e7a1ac4f..00fb519406cf 100644
--- a/drivers/scsi/ufs/ufs-sysfs.c
+++ b/drivers/scsi/ufs/ufs-sysfs.c
@@ -841,6 +841,7 @@ out:
\
 static DEVICE_ATTR_RO(_name)
 
 UFS_ATTRIB

[PATCH v25 2/4] scsi: ufs: L2P map management for HPB read

2021-02-25 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1091 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1181 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 65563635e20e..957763db1006 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -472,6 +472,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -482,6 +517,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 49b3d5d24fa6..5852ff44c3cc 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5021,6 +5021,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9221,6 +9224,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 1a72f6541510..8abadb0e010a 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v25 1/4] scsi: ufs: Introduce HPB feature

2021-02-25 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 166 ++
 9 files changed, 976 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..bf5cb8846de1 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v25 0/4] scsi: ufs: Add Host Performance Booster Support

2021-02-25 Thread Daejun Park

rty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached sub-region
  scsi: ufs: Add HPB 2.0 support

 Documentation/ABI/testing/sysfs-driver-ufs |  150 ++
 drivers/scsi/ufs/Kconfig   |9 +
 drivers/scsi/ufs/Makefile  |1 +
 drivers/scsi/ufs/ufs-sysfs.c   |   20 +
 drivers/scsi/ufs/ufs.h |   54 +-
 drivers/scsi/ufs/ufshcd.c  |   73 +-
 drivers/scsi/ufs/ufshcd.h  |   29 +
 drivers/scsi/ufs/ufshpb.c  | 2387 
 drivers/scsi/ufs/ufshpb.h  |  276 +++
 9 files changed, 2997 insertions(+), 2 deletions(-)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

-- 
2.25.1

RE: Re: [PATCH v24 2/4] scsi: ufs: L2P map management for HPB read

2021-02-25 Thread Daejun Park

> > +static int ufshpb_init_mem_wq(void)
> > +{
> > +   int ret;
> > +   unsigned int pool_size;
> > +
> > +   ufshpb_mctx_cache = kmem_cache_create("ufshpb_mctx_cache",
> > +   sizeof(struct
> > ufshpb_map_ctx),
> > +   0, 0, NULL);
> > +   if (!ufshpb_mctx_cache) {
> > +   pr_err("ufshpb: cannot init mctx cache\n");
> > +   return -ENOMEM;
> > +   }
> > +
> > +   pool_size = PAGE_ALIGN(ufshpb_host_map_kbytes * 1024) /
> > PAGE_SIZE;
> > +   pr_info("%s:%d ufshpb_host_map_kbytes %u pool_size %u\n",
> > +  __func__, __LINE__, ufshpb_host_map_kbytes,
> > pool_size);
> > +
> 
> I think print function name is not proper while booting.
> And one HPB is associated with one HBA, if there are two UFS
> controllers, how can I differentiate them? 

I will use dev_info instead of pr_info.

Thanks,
Daejun

> Bean
> 
> 
>

RE: Re: [PATCH v24 1/4] scsi: ufs: Introduce HPB feature

2021-02-25 Thread Daejun Park

> > 
> > +void ufshpb_init(struct ufs_hba *hba)
> > +{
> > +struct ufshpb_dev_info *hpb_dev_info = >ufshpb_dev;
> > +int try;
> > +int ret;
> > +
> > +if (!ufshpb_is_allowed(hba))
> > +return;
> > +
>  
> Here it is better to check "dev_info->hpb_enable", if HPB is not
> enabled from UFS device level,  doesn't need to create mempool and take
> other memory resource.

I will add checking dev_info->hpb_enable value.
if (!ufshpb_is_allowed(hba) || !dev_info->hpb_enable)

Thanks,
Daejun

> Bean
>  
>  
>  
>  
>  
>

RE: RE: [PATCH v24 4/4] scsi: ufs: Add HPB 2.0 support

2021-02-24 Thread Daejun Park

> > if (dev_info->wspecversion >= UFS_DEV_HPB_SUPPORT_VERSION &&
> > (b_ufs_feature_sup & UFS_DEV_HPB_SUPPORT)) {
> > -   dev_info->hpb_enabled = true;
> > +   bool hpb_en = false;
> > +
> > ufshpb_get_dev_info(hba, desc_buf);
> > +
> > +   err = ufshcd_query_flag_retry(hba,
> > UPIU_QUERY_OPCODE_READ_FLAG,
> > + QUERY_FLAG_IDN_HPB_EN, 0, 
> > _en);
> > +   if (ufshpb_is_legacy(hba) || (!err && hpb_en))
> If is_legacy you shouldn't send fHPBen in the first place, not ignoring its 
> failure.
OK,

> Also, using a Boolean is limiting you to HPB2.0 vs. HPB1.0.
> What would you do in new flags/attributes/descriptors that HPB3.0 will 
> introduce?
It can be managed as enum. but it is enough now, I think.

> > +   dev_info->hpb_enabled = true;
> > }

Thanks,
Daejun

RE: RE: [PATCH v24 4/4] scsi: ufs: Add HPB 2.0 support

2021-02-24 Thread Daejun Park

> > 
> > > +static int ufshpb_issue_umap_all_req(struct ufshpb_lu *hpb)
> > Maybe ufshpb_issue_umap_all_req is just a wrapper for
> > ufshpb_issue_umap_req?
> > e.g it calls ufshpb_issue_umap_req(hpb, int read_buferr_id = 0x3) ?
> > Then on host mode inactivation:
> > static int ufshpb_issue_umap_single_req(struct ufshpb_lu *hpb)
> > {
> > return ufshpb_issue_umap_req(hpb, 0x1);
> > }
> Better yet, ufshpb_execute_umap_req can get *rgn as an extra argument.
> ufshpb_issue_umap_all_req will call it with NULL, while
> ufshpb_issue_umap_single_req will call it with the rgn to inactivate.
> 
> Then,  ufshpb_set_unmap_cmd takes the form:
> static void ufshpb_set_unmap_cmd(unsigned char *cdb, struct ufshpb_region 
> *rgn)
> {
> cdb[0] = UFSHPB_WRITE_BUFFER;
> 
> if (rgn) {
> cdb[1] = UFSHPB_WRITE_BUFFER_INACT_SINGLE_ID;
> put_unaligned_be16(rgn->rgn_idx, [2]);
> } else {
> cdb[1] = UFSHPB_WRITE_BUFFER_INACT_ALL_ID;
> }
> 
> cdb[9] = 0x00;
> }
> 
> Does it make sense?

I got it.

Thanks,
Daejun

RE: RE: [PATCH v22 4/4] scsi: ufs: Add HPB 2.0 support

2021-02-24 Thread Daejun Park

> > +   copied = ufshpb_fill_ppn_from_page(hpb, srgn->mctx, srgn_offset,
> > +  pre_req->wb.len - offset,
> > +  [offset]);
> > +
> > +   if (copied < 0)
> > +   goto mctx_error;
> > +
> > +   offset += copied;
> > +   srgn_offset += offset;
> This seems wrong.
> How come the region offset is affected from the offset inside the pages?

I will change as : srgn_offset += copied
> 
> > +
> > +   if (srgn_offset == hpb->entries_per_srgn) {
> > +   srgn_offset = 0;
> > +
> > +   if (++srgn_idx == hpb->srgns_per_rgn) {
> > +   srgn_idx = 0;
> > +   rgn_idx++;
> > +   }
> > +   }
> > +
> > +   if (offset < pre_req->wb.len)
> > +   goto next_offset;
> If the 512k resides in a single subregion, and span across pages, fill_ppn 
> should take care of that.
> If the 512k spans across subregion regions, than it spans across 2 subregions 
> at most,
> and maybe you can use it.

I think it can be support span across pages.
 
The following is about the case of HPB entries are span to two page of
the mctx. fill_ppn() fills HPB entries in the first page of the mctx.
srgn_offset will be updated to check next page. In the next round,
fill_ppn() fills the entries in the second page of mctx.

Thanks,
Daejun

RE: RE: RE: [PATCH v22 4/4] scsi: ufs: Add HPB 2.0 support

2021-02-24 Thread Daejun Park

> > > > @@ -2656,7 +2656,12 @@ static int ufshcd_queuecommand(struct
> > > Scsi_Host
> > > > > *host, struct scsi_cmnd *cmd)
> > > > >
> > > > > lrbp->req_abort_skip = false;
> > > > >
> > > > > -   ufshpb_prep(hba, lrbp);
> > > > > +   err = ufshpb_prep(hba, lrbp);
> > > > > +   if (err == -EAGAIN) {
> > > > > +   lrbp->cmd = NULL;
> > > > > +   ufshcd_release(hba);
> > > > > +   goto out;
> > > > > +   }
> > > > Did I miss-read it, or are you bailing out of wb failed e.g. because no 
> > > > tag is
> > > available?
> > > > Why not continue with read10?
> > >
> > > We try to sending HPB read several times within the requeue_timeout_ms.
> > > Because it strategy has more benefit for overall performance in this
> > > situation that many requests are queueing.
> > This extra logic, IMO, should be optional.  Default none.
> > And yes, in this case requeue_timeout should be a parameter for each OEM to
> > scale.
> And either way, hpb internal flows should not cause dumping the command.
> Worse case - continue as READ10

However, this can improve the overall performance and should be used
carefully. The problem can be solved by setting the default timeout
ms to 0. And OEM can change it.

Thanks,
Daejun

[PATCH v24 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-02-23 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 253 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 254 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 5852ff44c3cc..851c01a26207 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2656,6 +2656,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 5d7b60c273cc..1a4a9c548ef3 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,230 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v24 4/4] scsi: ufs: Add HPB 2.0 support

2021-02-23 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  35 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   2 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  18 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 616 +++--
 drivers/scsi/ufs/ufshpb.h  |  67 ++-
 7 files changed, 669 insertions(+), 79 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index bf5cb8846de1..0017eaf89cbe 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,33 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
diff --git a/drivers/scsi/ufs/ufs-sysfs.c b/drivers/scsi/ufs/ufs-sysfs.c
index 2546e7a1ac4f..00fb519406cf 100644
--- a/drivers/scsi/ufs/ufs-sysfs.c
+++ b/drivers/scsi/ufs/ufs-sysfs.c
@@ -841,6 +841,7 @@ out:
\
 static DEVICE_ATTR_RO(_name)
 
 UFS_ATTRIB

[PATCH v24 2/4] scsi: ufs: L2P map management for HPB read

2021-02-23 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1091 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1181 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 65563635e20e..957763db1006 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -472,6 +472,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -482,6 +517,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 49b3d5d24fa6..5852ff44c3cc 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5021,6 +5021,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9221,6 +9224,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index a36ca89ccd8b..5d7b60c273cc 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v24 1/4] scsi: ufs: Introduce HPB feature

2021-02-23 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 166 ++
 9 files changed, 976 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..bf5cb8846de1 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v24 0/4] scsi: ufs: Add Host Performance Booster Support

2021-02-23 Thread Daejun Park

Changelog:

v23 -> v24
1. Fix build error reported by kernel test robot.

v22 -> v23
1. Add support compatibility of HPB 1.0.
2. Fix read id for single HPB read command.
3. Fix number of pre-allocated requests for write buffer.
4. Add fast path for response UPIU that has same LUN in sense data.
5. Remove WARN_ON for preventing kernel crash.
7. Fix wrong argument for read buffer command.

v21 -> v22
1. Add support processing response UPIU in suspend state.
2. Add support HPB hint from other LU.
3. Add sending write buffer with 0x03 after HPB init.

v20 -> v21
1. Add bMAX_DATA_SIZE_FOR_HPB_SINGLE_CMD attr. and fHPBen flag support.

v19 -> v20
1. Add documentation for sysfs entries of hpb->stat.
2. Fix read buffer command for under-sized sub-region.
3. Fix wrong condition checking for kick map work.
4. Delete redundant response UPIU checking.
5. Add LUN checking in response UPIU.
6. Fix possible deadlock problem due to runtime PM.
7. Add instant changing of sub-region state from response UPIU.
8. Fix endian problem in prefetched PPN.
9. Add JESD220-3A (HPB v2.0) support.

v18 -> 19
1. Fix null pointer error when printing sysfs from non-HPB LU.
2. Apply HPB read opcode in lrbp->cmd->cmnd (from Can Guo's review).
3. Rebase the patch on 5.12/scsi-queue.

v17 -> v18
Fix build error which reported by kernel test robot.

v16 -> v17
1. Rename hpb_state_lock to rgn_state_lock and move it to corresponding
patch.
2. Remove redundant information messages.

v15 -> v16
1. Add missed sysfs ABI documentation.

v14 -> v15
1. Remove duplicated sysfs ABI entries in documentation.
2. Add experiment result of HPB performance testing with iozone.

v13 -> v14
1. Cleanup codes by commentted in Greg's review.
2. Add documentation for sysfs entries (from Greg's review).
3. Add experiment result of HPB performance testing.

v12 -> v13
1. Cleanup codes by comments from Can Guo.
2. Add HPB related descriptor/flag/attributes in sysfs.
3. Change base commit from 5.10/scsi-queue to 5.11/scsi-queue.

v11 -> v12
1. Fixed to return error value when HPB fails to initialize pinned active 
region.
2. Fixed to disable HPB feature if HPB fails to allocate essential memory
and workqueue.
3. Fixed to change proper sub-region state when region is already evicted.

v10 -> v11
Add a newline at end the last line on Kconfig file.

v9 -> v10
1. Fixed 64-bit division error
2. Fixed problems commentted in Bart's review.

v8 -> v9
1. Change sysfs initialization.
2. Change reading descriptor during HPB initialization
3. Fixed problems commentted in Bart's review.
4. Change base commit from 5.9/scsi-queue to 5.10/scsi-queue.

v7 -> v8
Remove wrongly added tags.

v6 -> v7
1. Remove UFS feature layer.
2. Cleanup for sparse error.

v5 -> v6
Change base commit to b53293fa662e28ae0cdd40828dc641c09f133405

v4 -> v5
Delete unused macro define.

v3 -> v4
1. Cleanup.

v2 -> v3
1. Add checking input module parameter value.
2. Change base commit from 5.8/scsi-queue to 5.9/scsi-queue.
3. Cleanup for unused variables and label.

v1 -> v2
1. Change the full boilerplate text to SPDX style.
2. Adopt dynamic allocation for sub-region data structure.
3. Cleanup.

NAND flash memory-based storage devices use Flash Translation Layer (FTL)
to translate logical addresses of I/O requests to corresponding flash
memory addresses. Mobile storage devices typically have RAM with
constrained size, thus lack in memory to keep the whole mapping table.
Therefore, mapping tables are partially retrieved from NAND flash on
demand, causing random-read performance degradation.

To improve random read performance, JESD220-3 (HPB v1.0) proposes HPB
(Host Performance Booster) which uses host system memory as a cache for the
FTL mapping table. By using HPB, FTL data can be read from host memory
faster than from NAND flash memory. 

The current version only supports the DCM (device control mode).
This patch consists of 3 parts to support HPB feature.

1) HPB probe and initialization process
2) READ -> HPB READ using cached map information
3) L2P (logical to physical) map management

In the HPB probe and init process, the device information of the UFS is
queried. After checking supported features, the data structure for the HPB
is initialized according to the device information.

A read I/O in the active sub-region where the map is cached is changed to
HPB READ by the HPB.

The HPB manages the L2P map using information received from the
device. For active sub-region, the HPB caches through ufshpb_map
request. For the in-active region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2

[PATCH v23 4/4] scsi: ufs: Add HPB 2.0 support

2021-02-23 Thread Daejun Park

This patch supports the HPB 2.0.

The HPB 2.0 supports read of varying sizes from 4KB to 512KB.
In the case of Read (<= 32KB) is supported as single HPB read.
In the case of Read (36KB ~ 512KB) is supported by as a combination of
write buffer command and HPB read command to deliver more PPN.
The write buffer commands may not be issued immediately due to busy tags.
To use HPB read more aggressively, the driver can requeue the write buffer
command. The requeue threshold is implemented as timeout and can be
modified with requeue_timeout_ms entry in sysfs.

Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs |  35 +-
 drivers/scsi/ufs/ufs-sysfs.c   |   2 +
 drivers/scsi/ufs/ufs.h |   3 +-
 drivers/scsi/ufs/ufshcd.c  |  19 +-
 drivers/scsi/ufs/ufshcd.h  |   7 +
 drivers/scsi/ufs/ufshpb.c  | 616 +++--
 drivers/scsi/ufs/ufshpb.h  |  65 ++-
 7 files changed, 668 insertions(+), 79 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index bf5cb8846de1..0017eaf89cbe 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1253,14 +1253,14 @@ Description:This entry shows the number of HPB 
pinned regions assigned to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/hit_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/hit_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that changed to HPB read.
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/miss_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/miss_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of reads that cannot be changed to
@@ -1268,7 +1268,7 @@ Description:  This entry shows the number of reads 
that cannot be changed to
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_noti_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_noti_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of response UPIUs that has
@@ -1276,7 +1276,7 @@ Description:  This entry shows the number of response 
UPIUs that has
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_active_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_active_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of active sub-regions recommended by
@@ -1284,7 +1284,7 @@ Description:  This entry shows the number of active 
sub-regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/rb_inactive_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/rb_inactive_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of inactive regions recommended by
@@ -1292,10 +1292,33 @@ Description:This entry shows the number of inactive 
regions recommended by
 
The file is read only.
 
-What:  /sys/class/scsi_device/*/device/hpb_sysfs/map_req_cnt
+What:  /sys/class/scsi_device/*/device/hpb_stat_sysfs/map_req_cnt
 Date:  February 2021
 Contact:   Daejun Park 
 Description:   This entry shows the number of read buffer commands for
activating sub-regions recommended by response UPIUs.
 
The file is read only.
+
+What:  
/sys/class/scsi_device/*/device/hpb_param_sysfs/requeue_timeout_ms
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the requeue timeout threshold for write buffer
+   command in ms. This value can be changed by writing proper 
integer to
+   this entry.
+
+What:  
/sys/bus/platform/drivers/ufshcd/*/attributes/max_data_size_hpb_single_cmd
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the maximum HPB data size for using single HPB
+   command.
+
+   ===  
+   00h  4KB
+   01h  8KB
+   02h  12KB
+   ...
+   FFh  1024KB
+   ===  
+
+   The file is read only.
diff --git a/drivers/scsi/ufs/ufs-sysfs.c b/drivers/scsi/ufs/ufs-sysfs.c
index 2546e7a1ac4f..00fb519406cf 100644
--- a/drivers/scsi/ufs/ufs-sysfs.c
+++ b/drivers/scsi/ufs/ufs-sysfs.c
@@ -841,6 +841,7 @@ out:
\
 static DEVICE_ATTR_RO(_name)
 
 UFS_ATTRIB

[PATCH v23 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-02-23 Thread Daejun Park

This patch changes the read I/O to the HPB read I/O.

If the logical address of the read I/O belongs to active sub-region, the
HPB driver modifies the read I/O command to HPB read. It modifies the UPIU
command of UFS instead of modifying the existing SCSI command.

In the HPB version 1.0, the maximum read I/O size that can be converted to
HPB read is 4KB.

The dirty map of the active sub-region prevents an incorrect HPB read that
has stale physical page number which is updated by previous write I/O.

Reviewed-by: Can Guo 
Reviewed-by: Bart Van Assche 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufshcd.c |   2 +
 drivers/scsi/ufs/ufshpb.c | 253 +-
 drivers/scsi/ufs/ufshpb.h |   2 +
 3 files changed, 254 insertions(+), 3 deletions(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 5852ff44c3cc..851c01a26207 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -2656,6 +2656,8 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, 
struct scsi_cmnd *cmd)
 
lrbp->req_abort_skip = false;
 
+   ufshpb_prep(hba, lrbp);
+
ufshcd_comp_scsi_upiu(hba, lrbp);
 
err = ufshcd_map_sg(hba, lrbp);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index 5d7b60c273cc..1a4a9c548ef3 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -46,6 +46,29 @@ static void ufshpb_set_state(struct ufshpb_lu *hpb, int 
state)
atomic_set(>hpb_state, state);
 }
 
+static int ufshpb_is_valid_srgn(struct ufshpb_region *rgn,
+   struct ufshpb_subregion *srgn)
+{
+   return rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID;
+}
+
+static bool ufshpb_is_read_cmd(struct scsi_cmnd *cmd)
+{
+   return req_op(cmd->request) == REQ_OP_READ;
+}
+
+static bool ufshpb_is_write_or_discard_cmd(struct scsi_cmnd *cmd)
+{
+   return op_is_write(req_op(cmd->request)) ||
+  op_is_discard(req_op(cmd->request));
+}
+
+static bool ufshpb_is_support_chunk(int transfer_len)
+{
+   return transfer_len <= HPB_MULTI_CHUNK_HIGH;
+}
+
 static bool ufshpb_is_general_lun(int lun)
 {
return lun < UFS_UPIU_MAX_UNIT_NUM_ID;
@@ -80,8 +103,8 @@ static void ufshpb_kick_map_work(struct ufshpb_lu *hpb)
 }
 
 static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
-struct ufshcd_lrb *lrbp,
-struct utp_hpb_rsp *rsp_field)
+   struct ufshcd_lrb *lrbp,
+   struct utp_hpb_rsp *rsp_field)
 {
/* Check HPB_UPDATE_ALERT */
if (!(lrbp->ucd_rsp_ptr->header.dword_2 &
@@ -107,6 +130,230 @@ static bool ufshpb_is_hpb_rsp_valid(struct ufs_hba *hba,
return true;
 }
 
+static void ufshpb_set_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int set_bit_len;
+   int bitmap_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if ((srgn_offset + cnt) > bitmap_len)
+   set_bit_len = bitmap_len - srgn_offset;
+   else
+   set_bit_len = cnt;
+
+   if (rgn->rgn_state != HPB_RGN_INACTIVE &&
+   srgn->srgn_state == HPB_SRGN_VALID)
+   bitmap_set(srgn->mctx->ppn_dirty, srgn_offset, set_bit_len);
+
+   srgn_offset = 0;
+   if (++srgn_idx == hpb->srgns_per_rgn) {
+   srgn_idx = 0;
+   rgn_idx++;
+   }
+
+   cnt -= set_bit_len;
+   if (cnt > 0)
+   goto next_srgn;
+}
+
+static bool ufshpb_test_ppn_dirty(struct ufshpb_lu *hpb, int rgn_idx,
+ int srgn_idx, int srgn_offset, int cnt)
+{
+   struct ufshpb_region *rgn;
+   struct ufshpb_subregion *srgn;
+   int bitmap_len;
+   int bit_len;
+
+next_srgn:
+   rgn = hpb->rgn_tbl + rgn_idx;
+   srgn = rgn->srgn_tbl + srgn_idx;
+
+   if (likely(!srgn->is_last))
+   bitmap_len = hpb->entries_per_srgn;
+   else
+   bitmap_len = hpb->last_srgn_entries;
+
+   if (!ufshpb_is_valid_srgn(rgn, srgn))
+   return true;
+
+   /*
+* If the region state is active, mctx must be allocated.
+* In this case, check whether the region is evicted or
+* mctx allcation fail.
+*/
+   if (unlikely(!srgn->mctx)) {
+   dev_err(>sdev_ufs_lu->sdev_dev,
+   "no m

[PATCH v23 2/4] scsi: ufs: L2P map management for HPB read

2021-02-23 Thread Daejun Park

This is a patch for managing L2P map in HPB module.

The HPB divides logical addresses into several regions. A region consists
of several sub-regions. The sub-region is a basic unit where L2P mapping is
managed. The driver loads L2P mapping data of each sub-region. The loaded
sub-region is called active-state. The HPB driver unloads L2P mapping data
as region unit. The unloaded region is called inactive-state.

Sub-region/region candidates to be loaded and unloaded are delivered from
the UFS device. The UFS device delivers the recommended active sub-region
and inactivate region to the driver using sensedata.
The HPB module performs L2P mapping management on the host through the
delivered information.

A pinned region is a pre-set regions on the UFS device that is always
activate-state.

The data structure for map data request and L2P map uses mempool API,
minimizing allocation overhead while avoiding static allocation.

The mininum size of the memory pool used in the HPB is implemented
as a module parameter, so that it can be configurable by the user.

To gurantee a minimum memory pool size of 4MB: ufshpb_host_map_kbytes=4096

The map_work manages active/inactive by 2 "to-do" lists.
Each hpb lun maintains 2 "to-do" lists:
  hpb->lh_inact_rgn - regions to be inactivated, and
  hpb->lh_act_srgn - subregions to be activated
Those lists are maintained on IO completion.

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Signed-off-by: Daejun Park 
---
 drivers/scsi/ufs/ufs.h|   36 ++
 drivers/scsi/ufs/ufshcd.c |4 +
 drivers/scsi/ufs/ufshpb.c | 1091 -
 drivers/scsi/ufs/ufshpb.h |   65 +++
 4 files changed, 1181 insertions(+), 15 deletions(-)

diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 65563635e20e..957763db1006 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -472,6 +472,41 @@ struct utp_cmd_rsp {
u8 sense_data[UFS_SENSE_SIZE];
 };
 
+struct ufshpb_active_field {
+   __be16 active_rgn;
+   __be16 active_srgn;
+};
+#define HPB_ACT_FIELD_SIZE 4
+
+/**
+ * struct utp_hpb_rsp - Response UPIU structure
+ * @residual_transfer_count: Residual transfer count DW-3
+ * @reserved1: Reserved double words DW-4 to DW-7
+ * @sense_data_len: Sense data length DW-8 U16
+ * @desc_type: Descriptor type of sense data
+ * @additional_len: Additional length of sense data
+ * @hpb_op: HPB operation type
+ * @lun: LUN of response UPIU
+ * @active_rgn_cnt: Active region count
+ * @inactive_rgn_cnt: Inactive region count
+ * @hpb_active_field: Recommended to read HPB region and subregion
+ * @hpb_inactive_field: To be inactivated HPB region and subregion
+ */
+struct utp_hpb_rsp {
+   __be32 residual_transfer_count;
+   __be32 reserved1[4];
+   __be16 sense_data_len;
+   u8 desc_type;
+   u8 additional_len;
+   u8 hpb_op;
+   u8 lun;
+   u8 active_rgn_cnt;
+   u8 inactive_rgn_cnt;
+   struct ufshpb_active_field hpb_active_field[2];
+   __be16 hpb_inactive_field[2];
+};
+#define UTP_HPB_RSP_SIZE 40
+
 /**
  * struct utp_upiu_rsp - general upiu response structure
  * @header: UPIU header structure DW-0 to DW-2
@@ -482,6 +517,7 @@ struct utp_upiu_rsp {
struct utp_upiu_header header;
union {
struct utp_cmd_rsp sr;
+   struct utp_hpb_rsp hr;
struct utp_upiu_query qr;
};
 };
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 49b3d5d24fa6..5852ff44c3cc 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -5021,6 +5021,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct 
ufshcd_lrb *lrbp)
 */
pm_runtime_get_noresume(hba->dev);
}
+
+   if (scsi_status == SAM_STAT_GOOD)
+   ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -9221,6 +9224,7 @@ EXPORT_SYMBOL(ufshcd_shutdown);
 void ufshcd_remove(struct ufs_hba *hba)
 {
ufs_bsg_remove(hba);
+   ufshpb_remove(hba);
ufs_sysfs_remove_nodes(hba->dev);
blk_cleanup_queue(hba->tmf_queue);
blk_mq_free_tag_set(>tmf_tag_set);
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
index a36ca89ccd8b..5d7b60c273cc 100644
--- a/drivers/scsi/ufs/ufshpb.c
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -16,6 +16,16 @@
 #include "ufshpb.h"
 #include "../sd.h"
 
+/* memory management */
+static struct kmem_cache *ufshpb_mctx_cache;
+static mempool_t *ufshpb_mctx_pool;
+static mempool_t *ufshpb_page_pool;
+/* A cache size of 2MB can cache ppn in the 1GB range. */
+static unsigned int ufshpb_host_map_kbytes = 2048;
+static int tot_active_srgn_pages;
+
+static str

[PATCH v23 1/4] scsi: ufs: Introduce HPB feature

2021-02-23 Thread Daejun Park

This is a patch for the HPB initialization and adds HPB function calls to
UFS core driver.

NAND flash-based storage devices, including UFS, have mechanisms to
translate logical addresses of IO requests to the corresponding physical
addresses of the flash storage.
In UFS, Logical-address-to-Physical-address (L2P) map data, which is
required to identify the physical address for the requested IOs, can only
be partially stored in SRAM from NAND flash. Due to this partial loading,
accessing the flash address area where the L2P information for that address
is not loaded in the SRAM can result in serious performance degradation.

The basic concept of HPB is to cache L2P mapping entries in host system
memory so that both physical block address (PBA) and logical block address
(LBA) can be delivered in HPB read command.
The HPB READ command allows to read data faster than a read command in UFS
since it provides the physical address (HPB Entry) of the desired logical
block in addition to its logical address. The UFS device can access the
physical block in NAND directly without searching and uploading L2P mapping
table. This improves read performance because the NAND read operation for
uploading L2P mapping table is removed.

In HPB initialization, the host checks if the UFS device supports HPB
feature and retrieves related device capabilities. Then, some HPB
parameters are configured in the device.

We measured the total start-up time of popular applications and observed
the difference by enabling the HPB.
Popular applications are 12 game apps and 24 non-game apps. Each target
applications were launched in order. The cycle consists of running 36
applications in sequence. We repeated the cycle for observing performance
improvement by L2P mapping cache hit in HPB.

The Following is experiment environment:
 - kernel version: 4.4.0
 - RAM: 8GB
 - UFS 2.1 (64GB)

Result:
+---+--+--+---+
| cycle | baseline | with HPB | diff  |
+---+--+--+---+
| 1 | 272.4| 264.9| -7.5  |
| 2 | 250.4| 248.2| -2.2  |
| 3 | 226.2| 215.6| -10.6 |
| 4 | 230.6| 214.8| -15.8 |
| 5 | 232.0| 218.1| -13.9 |
| 6 | 231.9| 212.6| -19.3 |
+---+--+--+---+

We also measured HPB performance using iozone.
Here is my iozone script:
iozone -r 4k -+n -i2 -ecI -t 16 -l 16 -u 16
-s $IO_RANGE/16 -F mnt/tmp_1 mnt/tmp_2 mnt/tmp_3 mnt/tmp_4 mnt/tmp_5
mnt/tmp_6 mnt/tmp_7 mnt/tmp_8 mnt/tmp_9 mnt/tmp_10 mnt/tmp_11 mnt/tmp_12
mnt/tmp_13 mnt/tmp_14 mnt/tmp_15 mnt/tmp_16

Result:
+--++-+
| IO range | HPB on | HPB off |
+--++-+
|   1 GB   | 294.8  | 300.87  |
|   4 GB   | 293.51 | 179.35  |
|   8 GB   | 294.85 | 162.52  |
|  16 GB   | 293.45 | 156.26  |
|  32 GB   | 277.4  | 153.25  |
+--++-+

Reviewed-by: Bart Van Assche 
Reviewed-by: Can Guo 
Acked-by: Avri Altman 
Tested-by: Bean Huo 
Reported-by: kernel test robot 
Signed-off-by: Daejun Park 
---
 Documentation/ABI/testing/sysfs-driver-ufs | 127 +
 drivers/scsi/ufs/Kconfig   |   9 +
 drivers/scsi/ufs/Makefile  |   1 +
 drivers/scsi/ufs/ufs-sysfs.c   |  18 +
 drivers/scsi/ufs/ufs.h |  15 +
 drivers/scsi/ufs/ufshcd.c  |  49 ++
 drivers/scsi/ufs/ufshcd.h  |  22 +
 drivers/scsi/ufs/ufshpb.c  | 569 +
 drivers/scsi/ufs/ufshpb.h  | 166 ++
 9 files changed, 976 insertions(+)
 create mode 100644 drivers/scsi/ufs/ufshpb.c
 create mode 100644 drivers/scsi/ufs/ufshpb.h

diff --git a/Documentation/ABI/testing/sysfs-driver-ufs 
b/Documentation/ABI/testing/sysfs-driver-ufs
index d1bc23cb6a9d..bf5cb8846de1 100644
--- a/Documentation/ABI/testing/sysfs-driver-ufs
+++ b/Documentation/ABI/testing/sysfs-driver-ufs
@@ -1172,3 +1172,130 @@ Description:This node is used to set or display 
whether UFS WriteBooster is
(if the platform supports UFSHCD_CAP_CLK_SCALING). For a
platform that doesn't support UFSHCD_CAP_CLK_SCALING, we can
disable/enable WriteBooster through this sysfs node.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_version
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows the HPB specification version.
+   The full information about the descriptor could be found at UFS
+   HPB (Host Performance Booster) Extension specifications.
+   Example: version 1.2.3 = 0123h
+
+   The file is read only.
+
+What:  /sys/bus/platform/drivers/ufshcd/*/device_descriptor/hpb_control
+Date:  February 2021
+Contact:   Daejun Park 
+Description:   This entry shows an indication of the HPB control mode.
+   00h: Host control mode
+   01h: Device control mode

[PATCH v23 0/4] scsi: ufs: Add Host Performance Booster Support

2021-02-23 Thread Daejun Park

Changelog:

v22 -> v23
1. Add support compatibility of HPB 1.0.
2. Fix read id for single HPB read command.
3. Fix number of pre-allocated requests for write buffer.
4. Add fast path for response UPIU that has same LUN in sense data.
5. Remove WARN_ON for preventing kernel crash.
7. Fix wrong argument for read buffer command.

v21 -> v22
1. Add support processing response UPIU in suspend state.
2. Add support HPB hint from other LU.
3. Add sending write buffer with 0x03 after HPB init.

v20 -> v21
1. Add bMAX_DATA_SIZE_FOR_HPB_SINGLE_CMD attr. and fHPBen flag support.

v19 -> v20
1. Add documentation for sysfs entries of hpb->stat.
2. Fix read buffer command for under-sized sub-region.
3. Fix wrong condition checking for kick map work.
4. Delete redundant response UPIU checking.
5. Add LUN checking in response UPIU.
6. Fix possible deadlock problem due to runtime PM.
7. Add instant changing of sub-region state from response UPIU.
8. Fix endian problem in prefetched PPN.
9. Add JESD220-3A (HPB v2.0) support.

v18 -> 19
1. Fix null pointer error when printing sysfs from non-HPB LU.
2. Apply HPB read opcode in lrbp->cmd->cmnd (from Can Guo's review).
3. Rebase the patch on 5.12/scsi-queue.

v17 -> v18
Fix build error which reported by kernel test robot.

v16 -> v17
1. Rename hpb_state_lock to rgn_state_lock and move it to corresponding
patch.
2. Remove redundant information messages.

v15 -> v16
1. Add missed sysfs ABI documentation.

v14 -> v15
1. Remove duplicated sysfs ABI entries in documentation.
2. Add experiment result of HPB performance testing with iozone.

v13 -> v14
1. Cleanup codes by commentted in Greg's review.
2. Add documentation for sysfs entries (from Greg's review).
3. Add experiment result of HPB performance testing.

v12 -> v13
1. Cleanup codes by comments from Can Guo.
2. Add HPB related descriptor/flag/attributes in sysfs.
3. Change base commit from 5.10/scsi-queue to 5.11/scsi-queue.

v11 -> v12
1. Fixed to return error value when HPB fails to initialize pinned active 
region.
2. Fixed to disable HPB feature if HPB fails to allocate essential memory
and workqueue.
3. Fixed to change proper sub-region state when region is already evicted.

v10 -> v11
Add a newline at end the last line on Kconfig file.

v9 -> v10
1. Fixed 64-bit division error
2. Fixed problems commentted in Bart's review.

v8 -> v9
1. Change sysfs initialization.
2. Change reading descriptor during HPB initialization
3. Fixed problems commentted in Bart's review.
4. Change base commit from 5.9/scsi-queue to 5.10/scsi-queue.

v7 -> v8
Remove wrongly added tags.

v6 -> v7
1. Remove UFS feature layer.
2. Cleanup for sparse error.

v5 -> v6
Change base commit to b53293fa662e28ae0cdd40828dc641c09f133405

v4 -> v5
Delete unused macro define.

v3 -> v4
1. Cleanup.

v2 -> v3
1. Add checking input module parameter value.
2. Change base commit from 5.8/scsi-queue to 5.9/scsi-queue.
3. Cleanup for unused variables and label.

v1 -> v2
1. Change the full boilerplate text to SPDX style.
2. Adopt dynamic allocation for sub-region data structure.
3. Cleanup.

NAND flash memory-based storage devices use Flash Translation Layer (FTL)
to translate logical addresses of I/O requests to corresponding flash
memory addresses. Mobile storage devices typically have RAM with
constrained size, thus lack in memory to keep the whole mapping table.
Therefore, mapping tables are partially retrieved from NAND flash on
demand, causing random-read performance degradation.

To improve random read performance, JESD220-3 (HPB v1.0) proposes HPB
(Host Performance Booster) which uses host system memory as a cache for the
FTL mapping table. By using HPB, FTL data can be read from host memory
faster than from NAND flash memory. 

The current version only supports the DCM (device control mode).
This patch consists of 3 parts to support HPB feature.

1) HPB probe and initialization process
2) READ -> HPB READ using cached map information
3) L2P (logical to physical) map management

In the HPB probe and init process, the device information of the UFS is
queried. After checking supported features, the data structure for the HPB
is initialized according to the device information.

A read I/O in the active sub-region where the map is cached is changed to
HPB READ by the HPB.

The HPB manages the L2P map using information received from the
device. For active sub-region, the HPB caches through ufshpb_map
request. For the in-active region, the HPB discards the L2P map.
When a write I/O occurs in an active sub-region area, associated dirty
bitmap checked as dirty for preventing stale read.

HPB is shown to have a performance improvement of 58 - 67% for random read
workload. [1]

[1]:
https://www.usenix.org/conference/hotstorage17/program/presentation/jeong

Daejun Park (4):
  scsi: ufs: Introduce HPB feature
  scsi: ufs: L2P map management for HPB read
  scsi: ufs: Prepare HPB read for cached

RE: RE: [PATCH v22 4/4] scsi: ufs: Add HPB 2.0 support

2021-02-23 Thread Daejun Park

> > @@ -2656,7 +2656,12 @@ static int ufshcd_queuecommand(struct Scsi_Host
> > *host, struct scsi_cmnd *cmd)
> > 
> > lrbp->req_abort_skip = false;
> > 
> > -   ufshpb_prep(hba, lrbp);
> > +   err = ufshpb_prep(hba, lrbp);
> > +   if (err == -EAGAIN) {
> > +   lrbp->cmd = NULL;
> > +   ufshcd_release(hba);
> > +   goto out;
> > +   }
> Did I miss-read it, or are you bailing out of wb failed e.g. because no tag 
> is available?
> Why not continue with read10?

We try to sending HPB read several times within the requeue_timeout_ms.
Because it strategy has more benefit for overall performance in this
situation that many requests are queueing.

>  
>  
> > +   if (blk_insert_cloned_request(q, req) != BLK_STS_OK)
> > +   return -EAGAIN;
> Why did you choose to use blk_insert_cloned_request and not e.g. the more 
> common blk_execute_rq_nowait?

It is the process that sending one more command (write buffer) prior to
HPB read command. This API makes write buffer to issue directly. Other APIs,
for example blk_execute_rq_nowait, it can make queueing the command in the
scheduler, so the order of commands can be inversed.
Here is comment of the API.
// blk_insert_cloned_request - Helper for stacking drivers to submit a request

> > +   hpb->stats.pre_req_cnt++;
> > +
> > +   return 0;
> > +}
>  
> > -   ufshpb_set_hpb_read_to_upiu(hpb, lrbp, lpn, ppn, transfer_len);
> > +   if (ufshpb_is_required_wb(hpb, transfer_len)) {
> > +   err = ufshpb_issue_pre_req(hpb, cmd, _id);
> > +   if (err) {
> > +   unsigned long timeout;
> > +
> > +   timeout = cmd->jiffies_at_alloc + msecs_to_jiffies(
> > + hpb->params.requeue_timeout_ms);
> > +   if (time_before(jiffies, timeout))
> > +   return -EAGAIN;
> Why requeue_timeout_ms needs to be a configurable parameter?
> Why rq->timeout is not enough?

We are using this value for re-trying threshold of HPB read.

Thanks,
Daejun

> Thanks,
> Avri
>  
>  
>  
>  
>

RE: RE: [PATCH v22 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-02-23 Thread Daejun Park

> > +static int ufshpb_fill_ppn_from_page(struct ufshpb_lu *hpb,
> > +struct ufshpb_map_ctx *mctx, int pos,
> > +int len, u64 *ppn_buf)
> > +{
> > +   struct page *page;
> > +   int index, offset;
> > +   int copied;
> > +
> > +   index = pos / (PAGE_SIZE / HPB_ENTRY_SIZE);
> > +   offset = pos % (PAGE_SIZE / HPB_ENTRY_SIZE);
> Maybe cache hpb->entries_per_page in ufshpb_lu_parameter_init as well?

They are just defined constants and complier will optimize them.

Thanks,
Daejun

> > +
> > +   if ((offset + len) <= (PAGE_SIZE / HPB_ENTRY_SIZE))
> > +   copied = len;
> > +   else
> > +   copied = (PAGE_SIZE / HPB_ENTRY_SIZE) - offset;
> > +
> > +   page = mctx->m_page[index];
> > +   if (unlikely(!page)) {
> > +   dev_err(>sdev_ufs_lu->sdev_dev,
> > +   "error. cannot find page in mctx\n");
> > +   return -ENOMEM;
> > +   }
> > +
> > +   memcpy(ppn_buf, page_address(page) + (offset * HPB_ENTRY_SIZE),
> > +  copied * HPB_ENTRY_SIZE);
> > +
> > +   return copied;
> > +}
>  
>  
>

RE: RE: RE: [PATCH v22 3/4] scsi: ufs: Prepare HPB read for cached sub-region

2021-02-23 Thread Daejun Park

> > > > > +   err = ufshpb_fill_ppn_from_page(hpb, srgn->mctx, srgn_offset, 
> > > > > 1,
> > > );
> > > > > +   spin_unlock_irqrestore(>rgn_state_lock, flags);
> > > > > +   if (unlikely(err < 0)) {
> > > > > +   /*
> > > > > +* In this case, the region state is active,
> > > > > +* but the ppn table is not allocated.
> > > > > +* Make sure that ppn table must be allocated on
> > > > > +* active state.
> > > > > +*/
> > > > > +   WARN_ON(true);
> > > > > +   dev_err(hba->dev, "get ppn failed. err %d\n", err);
> > > > Maybe just pr_warn instead of risking crashing the machine over that?
> > >
> > > Why it crashing the machine? WARN_ON will just print kernel message.
> > I think that it can be configured, but I am not sure.
> I think it can be configured via the parameter panic_on_warn

OK, I will change WARN_ON to dev_err for print message.

Thanks,
Daejun

1 2 3 4 >

1 - 100 of 305 matches

Mail list logo