Re: [PATCH v7 16/21] memory: mtk-smi: Add bus_sel for mt8183
On Thu, Jun 20, 2019 at 7:38 PM Matthias Brugger wrote:
>
> CCing Sascha
>
> On 20/06/2019 11:35, Matthias Brugger wrote:
> >
> >
> > On 13/06/2019 10:14, Pi-Hsun Shih wrote:
> >> Hi,
> >> When I tested this patch series (Based on linux 5.2.0-rc2, and with
> >> various other patch series about MT8183) with lockdep enabled, and I'm
> >> seeing the following lockdep warning on boot.
> >>
> >> By bisecting the commits, the first commit that introduce this warning
> >> is this patch. The warning also doesn't appear if
> >> https://lore.kernel.org/patchwork/patch/1086582/ and
> >> https://lore.kernel.org/patchwork/patch/1086583/ are not applied.
> >>
> >> Do anyone have idea on why this is happening, or any suggestion on
> >> which part I should be digging into to figure this out? Thanks.
> >>
> >> [4.664194] ==
> >> [4.670368] WARNING: possible circular locking dependency detected
> >> [4.676545] 5.2.0-rc2-next-20190528-44527-g6c94b6475c04 #20 Tainted: G S
> >> [4.684539] --
> >> [4.690714] kworker/4:1/51 is trying to acquire lock:
> >> [4.695760] (ptrval) (regulator_list_mutex){+.+.},
> >> at:regulator_lock_dependent+0xdc/0x6c4
> >> [4.704732]
> >> [4.704732] but task is already holding lock:
> >> [4.710556] (ptrval) (>mlock/1){+.+.},
> >> at:genpd_lock_nested_mtx+0x24/0x30
> >> [4.718740]
> >> [4.718740] which lock already depends on the new lock.
> >> [4.718740]
> >> [4.726908]
> >> [4.726908] the existing dependency chain (in reverse order) is:
> >> [4.734382]
> >> [4.734382] -> #4 (>mlock/1){+.+.}:
> >> [4.739963]__mutex_lock_common+0x1a0/0x1fe8
> >> [4.744836]mutex_lock_nested+0x40/0x50
> >> [4.749275]genpd_lock_nested_mtx+0x24/0x30
> >> [4.754063]genpd_add_subdomain+0x150/0x524
> >> [4.758850]pm_genpd_add_subdomain+0x3c/0x5c
> >> [4.763723]scpsys_probe+0x520/0xe78
> >> [4.767902]platform_drv_probe+0xf4/0x134
> >> [4.772517]really_probe+0x214/0x4dc
> >> [4.776696]driver_probe_device+0xcc/0x1d4
> >> [4.781396]__device_attach_driver+0x10c/0x180
> >> [4.786442]bus_for_each_drv+0x124/0x184
> >> [4.790968]__device_attach+0x1c0/0x2d8
> >> [4.795407]device_initial_probe+0x20/0x2c
> >> [4.800106]bus_probe_device+0x80/0x16c
> >> [4.804546]deferred_probe_work_func+0x120/0x168
> >> [4.809767]process_one_work+0x858/0x1208
> >> [4.814379]worker_thread+0x9ec/0xcb8
> >> [4.818644]kthread+0x2b8/0x2d0
> >> [4.822391]ret_from_fork+0x10/0x18
> >> [4.826480]
> >> [4.826480] -> #3 (>mlock){+.+.}:
> >> [4.831880]__mutex_lock_common+0x1a0/0x1fe8
> >> [4.836752]mutex_lock_nested+0x40/0x50
> >> [4.841190]genpd_lock_mtx+0x20/0x2c
> >> [4.845369]genpd_runtime_resume+0x140/0x434
> >> [4.850241]__rpm_callback+0xb0/0x1e4
> >> [4.854506]rpm_callback+0x54/0x1a8
> >> [4.858597]rpm_resume+0xc6c/0x10c4
> >> [4.862689]__pm_runtime_resume+0xb4/0x124
> >> [4.867387]device_link_add+0x598/0x8d0
> >
> > For this looks as if you have also patch
> > [PATCH v2 04/12] memory: mtk-smi: Add device-link between smi-larb and
> > smi-common
> > from series
> > [PATCH v2 00/12] Clean up "mediatek,larb" after adding device_link
> > applied.
> >
>
> My guess is, that we run into this, because we call genpd_runtime_resume
> before
> we have finished the subdomain registration in scpsys_probe.
> That implies that you also have the scpsys series for mt8183 added to your
> tree.
>
> Anyway it looks to me as if we have to add the subdomains before we call
> of_genpd_add_provider_onecell().
>
> Regards,
> Matthias
Yes I do have scpsys series for mt8183 added in my tree. (I have about
125 patches from list on top of 531b0a360899 "Add linux-next specific
files for 20190528" to make the board boot)
However the exact same warning still exists after I changed the order
of of_genpd_add_provider_onecell and pm_genpd_add_subdomain, patch as
follow:
(Patch is based on commit f4788d37bc84 "Add linux-next specific files
for 20190614")
diff --git a/drivers/soc/mediatek/mtk-scpsys.c
b/drivers/soc/mediatek/mtk-scpsys.c
index 503222d0d0da..0cd9bdd4004e 100644
--- a/drivers/soc/mediatek/mtk-scpsys.c
+++ b/drivers/soc/mediatek/mtk-scpsys.c
@@ -440,8 +440,7 @@ static struct scp *init_scp(struct platform_device *pdev,
static void mtk_register_power_domains(struct platform_device *pdev,
struct scp *scp, int num)
{
- struct genpd_onecell_data *pd_data;
- int i, ret;
+ int i;
for (i = 0; i < num; i++) {
struct scp_domain *scpd = >domains[i];
@@ -457,18 +456,6 @@ static void mtk_register_power_domains(struct
platform_device *pdev,
[PATCH] Revert "iommu/vt-d: Fix lock inversion between iommu->lock and device_domain_lock"
This reverts commit 7560cc3ca7d9d11555f80c830544e463fcdb28b8.
With 5.2.0-rc5 I can easily trigger this with lockdep and iommu=pt:
==
WARNING: possible circular locking dependency detected
5.2.0-rc5 #78 Not tainted
--
swapper/0/1 is trying to acquire lock:
ea2b3beb (&(>lock)->rlock){+.+.}, at:
domain_context_mapping_one+0xa5/0x4e0
but task is already holding lock:
a681907b (device_domain_lock){}, at:
domain_context_mapping_one+0x8d/0x4e0
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #1 (device_domain_lock){}:
_raw_spin_lock_irqsave+0x3c/0x50
dmar_insert_one_dev_info+0xbb/0x510
domain_add_dev_info+0x50/0x90
dev_prepare_static_identity_mapping+0x30/0x68
intel_iommu_init+0xddd/0x1422
pci_iommu_init+0x16/0x3f
do_one_initcall+0x5d/0x2b4
kernel_init_freeable+0x218/0x2c1
kernel_init+0xa/0x100
ret_from_fork+0x3a/0x50
-> #0 (&(>lock)->rlock){+.+.}:
lock_acquire+0x9e/0x170
_raw_spin_lock+0x25/0x30
domain_context_mapping_one+0xa5/0x4e0
pci_for_each_dma_alias+0x30/0x140
dmar_insert_one_dev_info+0x3b2/0x510
domain_add_dev_info+0x50/0x90
dev_prepare_static_identity_mapping+0x30/0x68
intel_iommu_init+0xddd/0x1422
pci_iommu_init+0x16/0x3f
do_one_initcall+0x5d/0x2b4
kernel_init_freeable+0x218/0x2c1
kernel_init+0xa/0x100
ret_from_fork+0x3a/0x50
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0CPU1
lock(device_domain_lock);
lock(&(>lock)->rlock);
lock(device_domain_lock);
lock(&(>lock)->rlock);
*** DEADLOCK ***
2 locks held by swapper/0/1:
#0: 033eb13d (dmar_global_lock){}, at:
intel_iommu_init+0x1e0/0x1422
#1: a681907b (device_domain_lock){}, at:
domain_context_mapping_one+0x8d/0x4e0
stack backtrace:
CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.2.0-rc5 #78
Hardware name: LENOVO 20KGS35G01/20KGS35G01, BIOS N23ET50W (1.25 )
06/25/2018
Call Trace:
dump_stack+0x85/0xc0
print_circular_bug.cold.57+0x15c/0x195
__lock_acquire+0x152a/0x1710
lock_acquire+0x9e/0x170
? domain_context_mapping_one+0xa5/0x4e0
_raw_spin_lock+0x25/0x30
? domain_context_mapping_one+0xa5/0x4e0
domain_context_mapping_one+0xa5/0x4e0
? domain_context_mapping_one+0x4e0/0x4e0
pci_for_each_dma_alias+0x30/0x140
dmar_insert_one_dev_info+0x3b2/0x510
domain_add_dev_info+0x50/0x90
dev_prepare_static_identity_mapping+0x30/0x68
intel_iommu_init+0xddd/0x1422
? printk+0x58/0x6f
? lockdep_hardirqs_on+0xf0/0x180
? do_early_param+0x8e/0x8e
? e820__memblock_setup+0x63/0x63
pci_iommu_init+0x16/0x3f
do_one_initcall+0x5d/0x2b4
? do_early_param+0x8e/0x8e
? rcu_read_lock_sched_held+0x55/0x60
? do_early_param+0x8e/0x8e
kernel_init_freeable+0x218/0x2c1
? rest_init+0x230/0x230
kernel_init+0xa/0x100
ret_from_fork+0x3a/0x50
domain_context_mapping_one() is taking device_domain_lock first then
iommu lock, while dmar_insert_one_dev_info() is doing the reverse.
That should be introduced by commit:
7560cc3ca7d9 ("iommu/vt-d: Fix lock inversion between iommu->lock and
device_domain_lock", 2019-05-27)
So far I still cannot figure out how the previous deadlock was
triggered (I cannot find iommu lock taken before calling of
iommu_flush_dev_iotlb()), however I'm pretty sure that that change
should be incomplete at least because it does not fix all the places
so we're still taking the locks in different orders, while reverting
that commit is very clean to me so far that we should always take
device_domain_lock first then the iommu lock.
We can continue to try to find the real culprit mentioned in
7560cc3ca7d9, but for now I think we should revert it to fix current
breakage.
CC: Joerg Roedel
CC: Lu Baolu
CC: dave.ji...@intel.com
Signed-off-by: Peter Xu
---
drivers/iommu/intel-iommu.c | 7 +++
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 56297298d6ee..162b3236e72c 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -2504,7 +2504,6 @@ static struct dmar_domain
*dmar_insert_one_dev_info(struct intel_iommu *iommu,
}
}
- spin_lock(>lock);
spin_lock_irqsave(_domain_lock, flags);
if (dev)
found = find_domain(dev);
@@ -2520,16 +2519,17 @@ static
Re: VT-d deadlock issue on device_domain_lock and iommu lock (5.2-rc5)
On Fri, Jun 21, 2019 at 09:58:28AM +0800, Lu Baolu wrote:
> Hi Peter,
>
> I agree with you that 7560cc3ca7d9 ("iommu/vt-d: Fix lock inversion between
> iommu->lock and device_domain_lock") isn't a good fix. There
> is also another thread, https://lkml.org/lkml/2019/6/18/996, which
> reported this.
>
> I think we can revert this patch now. I will try to reproduce the
> original issue and try to find a new fix.
>
> Can you please submit the revert patch?
Sure, I'll post a formal patch soon. Thanks,
--
Peter Xu
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: VT-d deadlock issue on device_domain_lock and iommu lock (5.2-rc5)
Hi Peter,
I agree with you that 7560cc3ca7d9 ("iommu/vt-d: Fix lock inversion
between iommu->lock and device_domain_lock") isn't a good fix. There
is also another thread, https://lkml.org/lkml/2019/6/18/996, which
reported this.
I think we can revert this patch now. I will try to reproduce the
original issue and try to find a new fix.
Can you please submit the revert patch?
Best regards,
Baolu
On 6/20/19 6:44 PM, Peter Xu wrote:
Hi,
With 5.2.0-rc5 I can easily trigger this with lockdep:
Jun 20 14:37:37 xz-x1 kernel:
==
Jun 20 14:37:37 xz-x1 kernel: WARNING: possible circular locking dependency
detected
Jun 20 14:37:37 xz-x1 kernel: 5.2.0-rc5 #78 Not tainted
Jun 20 14:37:37 xz-x1 kernel:
--
Jun 20 14:37:37 xz-x1 kernel: swapper/0/1 is trying to acquire lock:
Jun 20 14:37:37 xz-x1 kernel: ea2b3beb (&(>lock)->rlock){+.+.},
at: domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel:
but task is already holding lock:
Jun 20 14:37:37 xz-x1 kernel: a681907b (device_domain_lock){}, at:
domain_context_mapping_one+0x8d/0x4e0
Jun 20 14:37:37 xz-x1 kernel:
which lock already depends on the new lock.
Jun 20 14:37:37 xz-x1 kernel:
the existing dependency chain (in reverse order)
is:
Jun 20 14:37:37 xz-x1 kernel:
-> #1 (device_domain_lock){}:
Jun 20 14:37:37 xz-x1 kernel:_raw_spin_lock_irqsave+0x3c/0x50
Jun 20 14:37:37 xz-x1 kernel:dmar_insert_one_dev_info+0xbb/0x510
Jun 20 14:37:37 xz-x1 kernel:domain_add_dev_info+0x50/0x90
Jun 20 14:37:37 xz-x1 kernel:
dev_prepare_static_identity_mapping+0x30/0x68
Jun 20 14:37:37 xz-x1 kernel:intel_iommu_init+0xddd/0x1422
Jun 20 14:37:37 xz-x1 kernel:pci_iommu_init+0x16/0x3f
Jun 20 14:37:37 xz-x1 kernel:do_one_initcall+0x5d/0x2b4
Jun 20 14:37:37 xz-x1 kernel:kernel_init_freeable+0x218/0x2c1
Jun 20 14:37:37 xz-x1 kernel:kernel_init+0xa/0x100
Jun 20 14:37:37 xz-x1 kernel:ret_from_fork+0x3a/0x50
Jun 20 14:37:37 xz-x1 kernel:
-> #0 (&(>lock)->rlock){+.+.}:
Jun 20 14:37:37 xz-x1 kernel:lock_acquire+0x9e/0x170
Jun 20 14:37:37 xz-x1 kernel:_raw_spin_lock+0x25/0x30
Jun 20 14:37:37 xz-x1 kernel:domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel:pci_for_each_dma_alias+0x30/0x140
Jun 20 14:37:37 xz-x1 kernel:dmar_insert_one_dev_info+0x3b2/0x510
Jun 20 14:37:37 xz-x1 kernel:domain_add_dev_info+0x50/0x90
Jun 20 14:37:37 xz-x1 kernel:
dev_prepare_static_identity_mapping+0x30/0x68
Jun 20 14:37:37 xz-x1 kernel:intel_iommu_init+0xddd/0x1422
Jun 20 14:37:37 xz-x1 kernel:pci_iommu_init+0x16/0x3f
Jun 20 14:37:37 xz-x1 kernel:do_one_initcall+0x5d/0x2b4
Jun 20 14:37:37 xz-x1 kernel:kernel_init_freeable+0x218/0x2c1
Jun 20 14:37:37 xz-x1 kernel:kernel_init+0xa/0x100
Jun 20 14:37:37 xz-x1 kernel:ret_from_fork+0x3a/0x50
Jun 20 14:37:37 xz-x1 kernel:
other info that might help us debug this:
Jun 20 14:37:37 xz-x1 kernel: Possible unsafe locking scenario:
Jun 20 14:37:37 xz-x1 kernel:CPU0CPU1
Jun 20 14:37:37 xz-x1 kernel:
Jun 20 14:37:37 xz-x1 kernel: lock(device_domain_lock);
Jun 20 14:37:37 xz-x1 kernel:
lock(&(>lock)->rlock);
Jun 20 14:37:37 xz-x1 kernel:
lock(device_domain_lock);
Jun 20 14:37:37 xz-x1 kernel: lock(&(>lock)->rlock);
Jun 20 14:37:37 xz-x1 kernel:
*** DEADLOCK ***
Jun 20 14:37:37 xz-x1 kernel: 2 locks held by swapper/0/1:
Jun 20 14:37:37 xz-x1 kernel: #0: 033eb13d (dmar_global_lock){},
at: intel_iommu_init+0x1e0/0x1422
Jun 20 14:37:37 xz-x1 kernel: #1: a681907b (device_domain_lock){},
at: domain_context_mapping_one+0x8d/0x4e0
Jun 20 14:37:37 xz-x1 kernel:
stack backtrace:
Jun 20 14:37:37 xz-x1 kernel: CPU: 2 PID: 1 Comm: swapper/0 Not tainted
5.2.0-rc5 #78
Jun 20 14:37:37 xz-x1 kernel: Hardware name: LENOVO 20KGS35G01/20KGS35G01, BIOS
N23ET50W (1.25 ) 06/25/2018
Jun 20 14:37:37 xz-x1 kernel: Call Trace:
Jun 20 14:37:37 xz-x1 kernel: dump_stack+0x85/0xc0
Jun 20 14:37:37 xz-x1 kernel: print_circular_bug.cold.57+0x15c/0x195
Jun 20 14:37:37 xz-x1 kernel: __lock_acquire+0x152a/0x1710
Jun 20 14:37:37 xz-x1 kernel: lock_acquire+0x9e/0x170
Jun 20 14:37:37 xz-x1 kernel: ? domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel: _raw_spin_lock+0x25/0x30
Jun 20 14:37:37 xz-x1 kernel: ? domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel:
Re: [PATCH v2 05/12] media: mtk-jpeg: Get rid of mtk_smi_larb_get/put
On 10/06/2019 14:55, Yong Wu wrote:
> MediaTek IOMMU has already added device_link between the consumer
> and smi-larb device. If the jpg device call the pm_runtime_get_sync,
> the smi-larb's pm_runtime_get_sync also be called automatically.
Please help me out find this relation. I seem to miss something basic, because I
can't find any between the jpeg IP and the iommu.
Regards,
Matthias
>
> CC: Rick Chang
> Signed-off-by: Yong Wu
> Reviewed-by: Evan Green
> ---
> drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c | 22 --
> drivers/media/platform/mtk-jpeg/mtk_jpeg_core.h | 2 --
> 2 files changed, 24 deletions(-)
>
> diff --git a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> index f761e4d..2f37538 100644
> --- a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> +++ b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> @@ -29,7 +29,6 @@
> #include
> #include
> #include
> -#include
>
> #include "mtk_jpeg_hw.h"
> #include "mtk_jpeg_core.h"
> @@ -901,11 +900,6 @@ static int mtk_jpeg_queue_init(void *priv, struct
> vb2_queue *src_vq,
>
> static void mtk_jpeg_clk_on(struct mtk_jpeg_dev *jpeg)
> {
> - int ret;
> -
> - ret = mtk_smi_larb_get(jpeg->larb);
> - if (ret)
> - dev_err(jpeg->dev, "mtk_smi_larb_get larbvdec fail %d\n", ret);
> clk_prepare_enable(jpeg->clk_jdec_smi);
> clk_prepare_enable(jpeg->clk_jdec);
> }
> @@ -914,7 +908,6 @@ static void mtk_jpeg_clk_off(struct mtk_jpeg_dev *jpeg)
> {
> clk_disable_unprepare(jpeg->clk_jdec);
> clk_disable_unprepare(jpeg->clk_jdec_smi);
> - mtk_smi_larb_put(jpeg->larb);
> }
>
> static irqreturn_t mtk_jpeg_dec_irq(int irq, void *priv)
> @@ -1059,21 +1052,6 @@ static int mtk_jpeg_release(struct file *file)
>
> static int mtk_jpeg_clk_init(struct mtk_jpeg_dev *jpeg)
> {
> - struct device_node *node;
> - struct platform_device *pdev;
> -
> - node = of_parse_phandle(jpeg->dev->of_node, "mediatek,larb", 0);
> - if (!node)
> - return -EINVAL;
> - pdev = of_find_device_by_node(node);
> - if (WARN_ON(!pdev)) {
> - of_node_put(node);
> - return -EINVAL;
> - }
> - of_node_put(node);
> -
> - jpeg->larb = >dev;
> -
> jpeg->clk_jdec = devm_clk_get(jpeg->dev, "jpgdec");
> if (IS_ERR(jpeg->clk_jdec))
> return PTR_ERR(jpeg->clk_jdec);
> diff --git a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.h
> b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.h
> index 1a6cdfd..e35fb79 100644
> --- a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.h
> +++ b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.h
> @@ -55,7 +55,6 @@ enum mtk_jpeg_ctx_state {
> * @dec_reg_base:JPEG registers mapping
> * @clk_jdec:JPEG hw working clock
> * @clk_jdec_smi:JPEG SMI bus clock
> - * @larb:SMI device
> */
> struct mtk_jpeg_dev {
> struct mutexlock;
> @@ -69,7 +68,6 @@ struct mtk_jpeg_dev {
> void __iomem*dec_reg_base;
> struct clk *clk_jdec;
> struct clk *clk_jdec_smi;
> - struct device *larb;
> };
>
> /**
>
Re: [PATCH v7 20/21] iommu/mediatek: Fix iova_to_phys PA start for 4GB mode
On Tue, 2019-06-18 at 18:35 +0200, Matthias Brugger wrote:
>
> On 10/06/2019 14:17, Yong Wu wrote:
> > In the 4GB mode, the physical address is remapped,
> >
> > Here is the detailed remap relationship.
> > CPU PA ->HW PA
> > 0x4000_ 0x1_4000_ (Add bit32)
> > 0x8000_ 0x1_8000_ ...
> > 0xc000_ 0x1_c000_ ...
> > 0x1__0x1__ (No change)
> >
> > Thus, we always add bit32 for PA when entering mtk_iommu_map.
> > But in the iova_to_phys, the CPU don't need this bit32 if the
> > PA is from 0x1_4000_ to 0x1__.
> > This patch discards the bit32 in this iova_to_phys in the 4GB mode.
> >
> > Fixes: 30e2fccf9512 ("iommu/mediatek: Enlarge the validate PA range
> > for 4GB mode")
> > Signed-off-by: Yong Wu
> > ---
> > drivers/iommu/mtk_iommu.c | 18 ++
> > 1 file changed, 18 insertions(+)
> >
> > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> > index 67cab2d..34f2e40 100644
> > --- a/drivers/iommu/mtk_iommu.c
> > +++ b/drivers/iommu/mtk_iommu.c
> > @@ -119,6 +119,19 @@ struct mtk_iommu_domain {
> >
> > static const struct iommu_ops mtk_iommu_ops;
> >
> > +/*
> > + * In M4U 4GB mode, the physical address is remapped as below:
> > + * CPU PA -> M4U HW PA
> > + * 0x4000_ 0x1_4000_ (Add bit32)
> > + * 0x8000_ 0x1_8000_ ...
> > + * 0xc000_ 0x1_c000_ ...
> > + * 0x1__ 0x1__ (No change)
> > + *
> > + * Thus, We always add BIT32 in the iommu_map and disable BIT32 if PA is >=
> > + * 0x1_4000_ in the iova_to_phys.
> > + */
> > +#define MTK_IOMMU_4GB_MODE_PA_14000 0x14000UL
> > +
> > static LIST_HEAD(m4ulist); /* List all the M4U HWs */
> >
> > #define for_each_m4u(data) list_for_each_entry(data, , list)
> > @@ -415,6 +428,7 @@ static phys_addr_t mtk_iommu_iova_to_phys(struct
> > iommu_domain *domain,
> > dma_addr_t iova)
> > {
> > struct mtk_iommu_domain *dom = to_mtk_domain(domain);
> > + struct mtk_iommu_data *data = mtk_iommu_get_m4u_data();
> > unsigned long flags;
> > phys_addr_t pa;
> >
> > @@ -422,6 +436,10 @@ static phys_addr_t mtk_iommu_iova_to_phys(struct
> > iommu_domain *domain,
> > pa = dom->iop->iova_to_phys(dom->iop, iova);
> > spin_unlock_irqrestore(>pgtlock, flags);
> >
> > + if (data->plat_data->has_4gb_mode && data->dram_is_4gb &&
> > + pa >= MTK_IOMMU_4GB_MODE_PA_14000)
> > + pa &= ~BIT_ULL(32);
> > +
>
> Hm, I wonder if we could fix this as first patch in the series, especially
> before:
> "[PATCH 06/21] iommu/io-pgtable-arm-v7s: Extend MediaTek 4GB Mode"
OK.
>
> This would make it easier for the stable maintainer to cherry-pick the fix.
> Without 100% understanding the code, it seems suspicious to me, that you first
> move the setting of the bit32 and bit33 into v7s and later explicitly clean
> the
> bits here.
>
> So my take on this is, that patch 6/21 introduced the regression you are
> trying
> to fix here. As said that is speculation as I don't understand the code in its
> whole.
>
> Any clarification would be useful.
I guess the commit message in [06/21] will be helpful.
In the previous mt8173 and mt2712, the M4U HW support "4GB mode" in
which the range of dram is from 0x4000_ to 0x1_3fff_ and it was
remapped to 0x1__ ~0x1__(For readable, I have wrote the
re-map relationship into the code in this patch.). but mt8183 don't need
remap the dram address(0x4000_ ~ 0x3__).
In order to unify the code, we add bit32 for "4GB mode". But actually
the PA doesn't always have bit32, thus, I have to remove bit32 when PA >
0x1_4000_.
So sorry that the "4GB mode" is a little unreadable and special, And the
4GB patch(30e2fccf9512 ("iommu/mediatek: Enlarge the validate PA range
for 4GB mode") has introduced several fix patches.
>
> Regards,
> Matthias
>
> > return pa;
> > }
> >
> >
>
> ___
> Linux-mediatek mailing list
> linux-media...@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-mediatek
Re: [PATCH v7 19/21] iommu/mediatek: Rename enable_4GB to dram_is_4gb
On Tue, 2019-06-18 at 18:06 +0200, Matthias Brugger wrote:
>
> On 10/06/2019 14:17, Yong Wu wrote:
> > This patch only rename the variable name from enable_4GB to
> > dram_is_4gb for readable.
>
> From my understanding this is true when available RAM > 4GB so I think the
> name
> should be something like dram_bigger_4gb otherwise it may create confusion
> again.
Strictly, It is not "dram_bigger_4gb". actually if the dram size is over
3GB (the first 1GB is the register space), the "4GB mode" will be
enabled. then how about the name "dram_enable_32bit"?(the PA 32bit will
be enabled in the 4GB mode.)
There is another option, please see the last part in [1] suggested by
Evan, something like below:
data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
if (!data->plat_data->has_4gb_mode)
data->enable_4GB = false;
Then mtk_iommu_map would only have:
if (data->enable_4GB)
paddr |= BIT_ULL(32);
Which one do you prefer?
[1] https://lore.kernel.org/patchwork/patch/1028421/
>
> Also from my point of view this patch should be done before
> "[PATCH 06/21] iommu/io-pgtable-arm-v7s: Extend MediaTek 4GB Mode"
OK.
>
> Regards,
> Matthias
>
> >
> > Signed-off-by: Yong Wu
> > Reviewed-by: Evan Green
> > ---
> > drivers/iommu/mtk_iommu.c | 10 +-
> > drivers/iommu/mtk_iommu.h | 2 +-
> > 2 files changed, 6 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> > index 86158d8..67cab2d 100644
> > --- a/drivers/iommu/mtk_iommu.c
> > +++ b/drivers/iommu/mtk_iommu.c
> > @@ -382,7 +382,7 @@ static int mtk_iommu_map(struct iommu_domain *domain,
> > unsigned long iova,
> > int ret;
> >
> > /* The "4GB mode" M4U physically can not use the lower remap of Dram. */
> > - if (data->plat_data->has_4gb_mode && data->enable_4GB)
> > + if (data->plat_data->has_4gb_mode && data->dram_is_4gb)
> > paddr |= BIT_ULL(32);
> >
> > spin_lock_irqsave(>pgtlock, flags);
> > @@ -554,13 +554,13 @@ static int mtk_iommu_hw_init(const struct
> > mtk_iommu_data *data)
> > writel_relaxed(regval, data->base + REG_MMU_INT_MAIN_CONTROL);
> >
> > if (data->plat_data->m4u_plat == M4U_MT8173)
> > - regval = (data->protect_base >> 1) | (data->enable_4GB << 31);
> > + regval = (data->protect_base >> 1) | (data->dram_is_4gb << 31);
> > else
> > regval = lower_32_bits(data->protect_base) |
> > upper_32_bits(data->protect_base);
> > writel_relaxed(regval, data->base + REG_MMU_IVRP_PADDR);
> >
> > - if (data->enable_4GB && data->plat_data->has_vld_pa_rng) {
> > + if (data->dram_is_4gb && data->plat_data->has_vld_pa_rng) {
> > /*
> > * If 4GB mode is enabled, the validate PA range is from
> > * 0x1__ to 0x1__. here record bit[32:30].
> > @@ -611,8 +611,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
> > return -ENOMEM;
> > data->protect_base = ALIGN(virt_to_phys(protect), MTK_PROTECT_PA_ALIGN);
> >
> > - /* Whether the current dram is over 4GB */
> > - data->enable_4GB = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
> > + /* Whether the current dram is 4GB. */
> > + data->dram_is_4gb = !!(max_pfn > (BIT_ULL(32) >> PAGE_SHIFT));
> >
> > res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> > data->base = devm_ioremap_resource(dev, res);
> > diff --git a/drivers/iommu/mtk_iommu.h b/drivers/iommu/mtk_iommu.h
> > index 753266b..e8114b2 100644
> > --- a/drivers/iommu/mtk_iommu.h
> > +++ b/drivers/iommu/mtk_iommu.h
> > @@ -65,7 +65,7 @@ struct mtk_iommu_data {
> > struct mtk_iommu_domain *m4u_dom;
> > struct iommu_group *m4u_group;
> > struct mtk_smi_iommusmi_imu; /* SMI larb iommu info */
> > - boolenable_4GB;
> > + booldram_is_4gb;
> > booltlb_flush_active;
> >
> > struct iommu_device iommu;
> >
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v7 17/21] memory: mtk-smi: Get rid of need_larbid
On Tue, 2019-06-18 at 15:45 +0200, Matthias Brugger wrote:
>
> On 10/06/2019 14:17, Yong Wu wrote:
> > The "mediatek,larb-id" has already been parsed in MTK IOMMU driver.
> > It's no need to parse it again in SMI driver. Only clean some codes.
> > This patch is fit for all the current mt2701, mt2712, mt7623, mt8173
> > and mt8183.
> >
> > After this patch, the "mediatek,larb-id" only be needed for mt2712
> > which have 2 M4Us. In the other SoCs, we can get the larb-id from M4U
> > in which the larbs in the "mediatek,larbs" always are ordered.
> >
> > Correspondingly, the larb_nr in the "struct mtk_smi_iommu" could also
> > be deleted.
> >
>
> I think we can get rid of struct mtk_smi_iommu and just add the
> struct mtk_smi_larb_iommu larb_imu[MTK_LARB_NR_MAX] directly to
> mtk_iommu_data,
> passing just that array to the components bind function.
Thanks. I will try this in a new patch.
>
> Never the less this patch looks fine:
> Reviewed-by: Matthias Brugger
Really appreciate for reviewing so many patches.
>
> > CC: Matthias Brugger
> > Signed-off-by: Yong Wu
> > Reviewed-by: Evan Green
> > ---
> > drivers/iommu/mtk_iommu.c| 1 -
> > drivers/iommu/mtk_iommu_v1.c | 2 --
> > drivers/memory/mtk-smi.c | 26 ++
> > include/soc/mediatek/smi.h | 1 -
> > 4 files changed, 2 insertions(+), 28 deletions(-)
> >
> > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
> > index ec4ce74..6053b8b 100644
> > --- a/drivers/iommu/mtk_iommu.c
> > +++ b/drivers/iommu/mtk_iommu.c
> > @@ -634,7 +634,6 @@ static int mtk_iommu_probe(struct platform_device *pdev)
> > "mediatek,larbs", NULL);
> > if (larb_nr < 0)
> > return larb_nr;
> > - data->smi_imu.larb_nr = larb_nr;
> >
> > for (i = 0; i < larb_nr; i++) {
> > struct device_node *larbnode;
> > diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c
> > index 52b01e3..73308ad 100644
> > --- a/drivers/iommu/mtk_iommu_v1.c
> > +++ b/drivers/iommu/mtk_iommu_v1.c
> > @@ -624,8 +624,6 @@ static int mtk_iommu_probe(struct platform_device *pdev)
> > larb_nr++;
> > }
> >
> > - data->smi_imu.larb_nr = larb_nr;
> > -
> > platform_set_drvdata(pdev, data);
> >
> > ret = mtk_iommu_hw_init(data);
> > diff --git a/drivers/memory/mtk-smi.c b/drivers/memory/mtk-smi.c
> > index 08cf40d..10e6493 100644
> > --- a/drivers/memory/mtk-smi.c
> > +++ b/drivers/memory/mtk-smi.c
> > @@ -67,7 +67,6 @@ struct mtk_smi_common_plat {
> > };
> >
> > struct mtk_smi_larb_gen {
> > - bool need_larbid;
> > int port_in_larb[MTK_LARB_NR_MAX + 1];
> > void (*config_port)(struct device *);
> > unsigned int larb_direct_to_common_mask;
> > @@ -153,18 +152,9 @@ void mtk_smi_larb_put(struct device *larbdev)
> > struct mtk_smi_iommu *smi_iommu = data;
> > unsigned int i;
> >
> > - if (larb->larb_gen->need_larbid) {
> > - larb->mmu = _iommu->larb_imu[larb->larbid].mmu;
> > - return 0;
> > - }
> > -
> > - /*
> > -* If there is no larbid property, Loop to find the corresponding
> > -* iommu information.
> > -*/
> > - for (i = 0; i < smi_iommu->larb_nr; i++) {
> > + for (i = 0; i < MTK_LARB_NR_MAX; i++) {
> > if (dev == smi_iommu->larb_imu[i].dev) {
> > - /* The 'mmu' may be updated in iommu-attach/detach. */
> > + larb->larbid = i;
> > larb->mmu = _iommu->larb_imu[i].mmu;
> > return 0;
> > }
> > @@ -243,7 +233,6 @@ static void mtk_smi_larb_config_port_gen1(struct device
> > *dev)
> > };
> >
> > static const struct mtk_smi_larb_gen mtk_smi_larb_mt2701 = {
> > - .need_larbid = true,
> > .port_in_larb = {
> > LARB0_PORT_OFFSET, LARB1_PORT_OFFSET,
> > LARB2_PORT_OFFSET, LARB3_PORT_OFFSET
> > @@ -252,7 +241,6 @@ static void mtk_smi_larb_config_port_gen1(struct device
> > *dev)
> > };
> >
> > static const struct mtk_smi_larb_gen mtk_smi_larb_mt2712 = {
> > - .need_larbid = true,
> > .config_port= mtk_smi_larb_config_port_gen2_general,
> > .larb_direct_to_common_mask = BIT(8) | BIT(9), /* bdpsys */
> > };
> > @@ -291,7 +279,6 @@ static int mtk_smi_larb_probe(struct platform_device
> > *pdev)
> > struct device *dev = >dev;
> > struct device_node *smi_node;
> > struct platform_device *smi_pdev;
> > - int err;
> >
> > larb = devm_kzalloc(dev, sizeof(*larb), GFP_KERNEL);
> > if (!larb)
> > @@ -321,15 +308,6 @@ static int mtk_smi_larb_probe(struct platform_device
> > *pdev)
> > }
> > larb->smi.dev = dev;
> >
> > - if (larb->larb_gen->need_larbid) {
> > - err = of_property_read_u32(dev->of_node, "mediatek,larb-id",
> > - >larbid);
> > - if (err) {
> > - dev_err(dev, "missing larbid
Re: [PATCH v7 16/21] memory: mtk-smi: Add bus_sel for mt8183
CCing Sascha
On 20/06/2019 11:35, Matthias Brugger wrote:
>
>
> On 13/06/2019 10:14, Pi-Hsun Shih wrote:
>> Hi,
>> When I tested this patch series (Based on linux 5.2.0-rc2, and with
>> various other patch series about MT8183) with lockdep enabled, and I'm
>> seeing the following lockdep warning on boot.
>>
>> By bisecting the commits, the first commit that introduce this warning
>> is this patch. The warning also doesn't appear if
>> https://lore.kernel.org/patchwork/patch/1086582/ and
>> https://lore.kernel.org/patchwork/patch/1086583/ are not applied.
>>
>> Do anyone have idea on why this is happening, or any suggestion on
>> which part I should be digging into to figure this out? Thanks.
>>
>> [4.664194] ==
>> [4.670368] WARNING: possible circular locking dependency detected
>> [4.676545] 5.2.0-rc2-next-20190528-44527-g6c94b6475c04 #20 Tainted: G S
>> [4.684539] --
>> [4.690714] kworker/4:1/51 is trying to acquire lock:
>> [4.695760] (ptrval) (regulator_list_mutex){+.+.},
>> at:regulator_lock_dependent+0xdc/0x6c4
>> [4.704732]
>> [4.704732] but task is already holding lock:
>> [4.710556] (ptrval) (>mlock/1){+.+.},
>> at:genpd_lock_nested_mtx+0x24/0x30
>> [4.718740]
>> [4.718740] which lock already depends on the new lock.
>> [4.718740]
>> [4.726908]
>> [4.726908] the existing dependency chain (in reverse order) is:
>> [4.734382]
>> [4.734382] -> #4 (>mlock/1){+.+.}:
>> [4.739963]__mutex_lock_common+0x1a0/0x1fe8
>> [4.744836]mutex_lock_nested+0x40/0x50
>> [4.749275]genpd_lock_nested_mtx+0x24/0x30
>> [4.754063]genpd_add_subdomain+0x150/0x524
>> [4.758850]pm_genpd_add_subdomain+0x3c/0x5c
>> [4.763723]scpsys_probe+0x520/0xe78
>> [4.767902]platform_drv_probe+0xf4/0x134
>> [4.772517]really_probe+0x214/0x4dc
>> [4.776696]driver_probe_device+0xcc/0x1d4
>> [4.781396]__device_attach_driver+0x10c/0x180
>> [4.786442]bus_for_each_drv+0x124/0x184
>> [4.790968]__device_attach+0x1c0/0x2d8
>> [4.795407]device_initial_probe+0x20/0x2c
>> [4.800106]bus_probe_device+0x80/0x16c
>> [4.804546]deferred_probe_work_func+0x120/0x168
>> [4.809767]process_one_work+0x858/0x1208
>> [4.814379]worker_thread+0x9ec/0xcb8
>> [4.818644]kthread+0x2b8/0x2d0
>> [4.822391]ret_from_fork+0x10/0x18
>> [4.826480]
>> [4.826480] -> #3 (>mlock){+.+.}:
>> [4.831880]__mutex_lock_common+0x1a0/0x1fe8
>> [4.836752]mutex_lock_nested+0x40/0x50
>> [4.841190]genpd_lock_mtx+0x20/0x2c
>> [4.845369]genpd_runtime_resume+0x140/0x434
>> [4.850241]__rpm_callback+0xb0/0x1e4
>> [4.854506]rpm_callback+0x54/0x1a8
>> [4.858597]rpm_resume+0xc6c/0x10c4
>> [4.862689]__pm_runtime_resume+0xb4/0x124
>> [4.867387]device_link_add+0x598/0x8d0
>
> For this looks as if you have also patch
> [PATCH v2 04/12] memory: mtk-smi: Add device-link between smi-larb and
> smi-common
> from series
> [PATCH v2 00/12] Clean up "mediatek,larb" after adding device_link
> applied.
>
My guess is, that we run into this, because we call genpd_runtime_resume before
we have finished the subdomain registration in scpsys_probe.
That implies that you also have the scpsys series for mt8183 added to your tree.
Anyway it looks to me as if we have to add the subdomains before we call
of_genpd_add_provider_onecell().
Regards,
Matthias
> Regards,
> Matthias
>
>> [4.871829]mtk_smi_larb_probe+0x2b0/0x340
>> [4.876528]platform_drv_probe+0xf4/0x134
>> [4.881141]really_probe+0x214/0x4dc
>> [4.885320]driver_probe_device+0xcc/0x1d4
>> [4.890020]__device_attach_driver+0x10c/0x180
>> [4.895066]bus_for_each_drv+0x124/0x184
>> [4.899591]__device_attach+0x1c0/0x2d8
>> [4.904031]device_initial_probe+0x20/0x2c
>> [4.908730]bus_probe_device+0x80/0x16c
>> [4.913169]deferred_probe_work_func+0x120/0x168
>> [4.918387]process_one_work+0x858/0x1208
>> [4.923000]worker_thread+0x9ec/0xcb8
>> [4.927264]kthread+0x2b8/0x2d0
>> [4.931009]ret_from_fork+0x10/0x18
>> [4.935098]
>> [4.935098] -> #2 (dpm_list_mtx){+.+.}:
>> [4.940412]__mutex_lock_common+0x1a0/0x1fe8
>> [4.945284]mutex_lock_nested+0x40/0x50
>> [4.949722]device_pm_lock+0x1c/0x24
>> [4.953900]device_link_add+0x98/0x8d0
>> [4.958252]_regulator_get+0x3f0/0x504
>> [4.962606]_devm_regulator_get+0x58/0xb8
>> [4.967218]devm_regulator_get+0x28/0x34
>> [4.971746]
Re: use exact allocation for dma coherent memory
On Wed, Jun 19, 2019 at 01:29:03PM -0300, Jason Gunthorpe wrote: > > Yes. This will blow up badly on many platforms, as sq->queue > > might be vmapped, ioremapped, come from a pool without page backing. > > Gah, this addr gets fed into io_remap_pfn_range/remap_pfn_range too.. > > Potnuri, you should fix this.. > > You probably need to use dma_mmap_from_dev_coherent() in the mmap ? The function to use is dma_mmap_coherent, dma_mmap_from_dev_coherent is just an internal helper. That beiŋ said the drivers/infiniband code has a lot of *remap_pfn_range, and a lot of them look like they might be for DMA memory. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
VT-d deadlock issue on device_domain_lock and iommu lock (5.2-rc5)
Hi,
With 5.2.0-rc5 I can easily trigger this with lockdep:
Jun 20 14:37:37 xz-x1 kernel:
==
Jun 20 14:37:37 xz-x1 kernel: WARNING: possible circular locking dependency
detected
Jun 20 14:37:37 xz-x1 kernel: 5.2.0-rc5 #78 Not tainted
Jun 20 14:37:37 xz-x1 kernel:
--
Jun 20 14:37:37 xz-x1 kernel: swapper/0/1 is trying to acquire lock:
Jun 20 14:37:37 xz-x1 kernel: ea2b3beb (&(>lock)->rlock){+.+.},
at: domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel:
but task is already holding lock:
Jun 20 14:37:37 xz-x1 kernel: a681907b (device_domain_lock){}, at:
domain_context_mapping_one+0x8d/0x4e0
Jun 20 14:37:37 xz-x1 kernel:
which lock already depends on the new lock.
Jun 20 14:37:37 xz-x1 kernel:
the existing dependency chain (in reverse order)
is:
Jun 20 14:37:37 xz-x1 kernel:
-> #1 (device_domain_lock){}:
Jun 20 14:37:37 xz-x1 kernel:_raw_spin_lock_irqsave+0x3c/0x50
Jun 20 14:37:37 xz-x1 kernel:dmar_insert_one_dev_info+0xbb/0x510
Jun 20 14:37:37 xz-x1 kernel:domain_add_dev_info+0x50/0x90
Jun 20 14:37:37 xz-x1 kernel:
dev_prepare_static_identity_mapping+0x30/0x68
Jun 20 14:37:37 xz-x1 kernel:intel_iommu_init+0xddd/0x1422
Jun 20 14:37:37 xz-x1 kernel:pci_iommu_init+0x16/0x3f
Jun 20 14:37:37 xz-x1 kernel:do_one_initcall+0x5d/0x2b4
Jun 20 14:37:37 xz-x1 kernel:kernel_init_freeable+0x218/0x2c1
Jun 20 14:37:37 xz-x1 kernel:kernel_init+0xa/0x100
Jun 20 14:37:37 xz-x1 kernel:ret_from_fork+0x3a/0x50
Jun 20 14:37:37 xz-x1 kernel:
-> #0 (&(>lock)->rlock){+.+.}:
Jun 20 14:37:37 xz-x1 kernel:lock_acquire+0x9e/0x170
Jun 20 14:37:37 xz-x1 kernel:_raw_spin_lock+0x25/0x30
Jun 20 14:37:37 xz-x1 kernel:domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel:pci_for_each_dma_alias+0x30/0x140
Jun 20 14:37:37 xz-x1 kernel:dmar_insert_one_dev_info+0x3b2/0x510
Jun 20 14:37:37 xz-x1 kernel:domain_add_dev_info+0x50/0x90
Jun 20 14:37:37 xz-x1 kernel:
dev_prepare_static_identity_mapping+0x30/0x68
Jun 20 14:37:37 xz-x1 kernel:intel_iommu_init+0xddd/0x1422
Jun 20 14:37:37 xz-x1 kernel:pci_iommu_init+0x16/0x3f
Jun 20 14:37:37 xz-x1 kernel:do_one_initcall+0x5d/0x2b4
Jun 20 14:37:37 xz-x1 kernel:kernel_init_freeable+0x218/0x2c1
Jun 20 14:37:37 xz-x1 kernel:kernel_init+0xa/0x100
Jun 20 14:37:37 xz-x1 kernel:ret_from_fork+0x3a/0x50
Jun 20 14:37:37 xz-x1 kernel:
other info that might help us debug this:
Jun 20 14:37:37 xz-x1 kernel: Possible unsafe locking scenario:
Jun 20 14:37:37 xz-x1 kernel:CPU0CPU1
Jun 20 14:37:37 xz-x1 kernel:
Jun 20 14:37:37 xz-x1 kernel: lock(device_domain_lock);
Jun 20 14:37:37 xz-x1 kernel:
lock(&(>lock)->rlock);
Jun 20 14:37:37 xz-x1 kernel:
lock(device_domain_lock);
Jun 20 14:37:37 xz-x1 kernel: lock(&(>lock)->rlock);
Jun 20 14:37:37 xz-x1 kernel:
*** DEADLOCK ***
Jun 20 14:37:37 xz-x1 kernel: 2 locks held by swapper/0/1:
Jun 20 14:37:37 xz-x1 kernel: #0: 033eb13d (dmar_global_lock){},
at: intel_iommu_init+0x1e0/0x1422
Jun 20 14:37:37 xz-x1 kernel: #1: a681907b (device_domain_lock){},
at: domain_context_mapping_one+0x8d/0x4e0
Jun 20 14:37:37 xz-x1 kernel:
stack backtrace:
Jun 20 14:37:37 xz-x1 kernel: CPU: 2 PID: 1 Comm: swapper/0 Not tainted
5.2.0-rc5 #78
Jun 20 14:37:37 xz-x1 kernel: Hardware name: LENOVO 20KGS35G01/20KGS35G01, BIOS
N23ET50W (1.25 ) 06/25/2018
Jun 20 14:37:37 xz-x1 kernel: Call Trace:
Jun 20 14:37:37 xz-x1 kernel: dump_stack+0x85/0xc0
Jun 20 14:37:37 xz-x1 kernel: print_circular_bug.cold.57+0x15c/0x195
Jun 20 14:37:37 xz-x1 kernel: __lock_acquire+0x152a/0x1710
Jun 20 14:37:37 xz-x1 kernel: lock_acquire+0x9e/0x170
Jun 20 14:37:37 xz-x1 kernel: ? domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel: _raw_spin_lock+0x25/0x30
Jun 20 14:37:37 xz-x1 kernel: ? domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel: domain_context_mapping_one+0xa5/0x4e0
Jun 20 14:37:37 xz-x1 kernel: ? domain_context_mapping_one+0x4e0/0x4e0
Jun 20 14:37:37 xz-x1 kernel: pci_for_each_dma_alias+0x30/0x140
Jun 20 14:37:37 xz-x1 kernel: dmar_insert_one_dev_info+0x3b2/0x510
Jun 20 14:37:37 xz-x1 kernel: domain_add_dev_info+0x50/0x90
Jun 20 14:37:37 xz-x1 kernel: dev_prepare_static_identity_mapping+0x30/0x68
Jun 20 14:37:37 xz-x1 kernel: intel_iommu_init+0xddd/0x1422
Jun 20 14:37:37 xz-x1 kernel: ?
Re: [PATCH v7 16/21] memory: mtk-smi: Add bus_sel for mt8183
On 13/06/2019 10:14, Pi-Hsun Shih wrote:
> Hi,
> When I tested this patch series (Based on linux 5.2.0-rc2, and with
> various other patch series about MT8183) with lockdep enabled, and I'm
> seeing the following lockdep warning on boot.
>
> By bisecting the commits, the first commit that introduce this warning
> is this patch. The warning also doesn't appear if
> https://lore.kernel.org/patchwork/patch/1086582/ and
> https://lore.kernel.org/patchwork/patch/1086583/ are not applied.
>
> Do anyone have idea on why this is happening, or any suggestion on
> which part I should be digging into to figure this out? Thanks.
>
> [4.664194] ==
> [4.670368] WARNING: possible circular locking dependency detected
> [4.676545] 5.2.0-rc2-next-20190528-44527-g6c94b6475c04 #20 Tainted: G S
> [4.684539] --
> [4.690714] kworker/4:1/51 is trying to acquire lock:
> [4.695760] (ptrval) (regulator_list_mutex){+.+.},
> at:regulator_lock_dependent+0xdc/0x6c4
> [4.704732]
> [4.704732] but task is already holding lock:
> [4.710556] (ptrval) (>mlock/1){+.+.},
> at:genpd_lock_nested_mtx+0x24/0x30
> [4.718740]
> [4.718740] which lock already depends on the new lock.
> [4.718740]
> [4.726908]
> [4.726908] the existing dependency chain (in reverse order) is:
> [4.734382]
> [4.734382] -> #4 (>mlock/1){+.+.}:
> [4.739963]__mutex_lock_common+0x1a0/0x1fe8
> [4.744836]mutex_lock_nested+0x40/0x50
> [4.749275]genpd_lock_nested_mtx+0x24/0x30
> [4.754063]genpd_add_subdomain+0x150/0x524
> [4.758850]pm_genpd_add_subdomain+0x3c/0x5c
> [4.763723]scpsys_probe+0x520/0xe78
> [4.767902]platform_drv_probe+0xf4/0x134
> [4.772517]really_probe+0x214/0x4dc
> [4.776696]driver_probe_device+0xcc/0x1d4
> [4.781396]__device_attach_driver+0x10c/0x180
> [4.786442]bus_for_each_drv+0x124/0x184
> [4.790968]__device_attach+0x1c0/0x2d8
> [4.795407]device_initial_probe+0x20/0x2c
> [4.800106]bus_probe_device+0x80/0x16c
> [4.804546]deferred_probe_work_func+0x120/0x168
> [4.809767]process_one_work+0x858/0x1208
> [4.814379]worker_thread+0x9ec/0xcb8
> [4.818644]kthread+0x2b8/0x2d0
> [4.822391]ret_from_fork+0x10/0x18
> [4.826480]
> [4.826480] -> #3 (>mlock){+.+.}:
> [4.831880]__mutex_lock_common+0x1a0/0x1fe8
> [4.836752]mutex_lock_nested+0x40/0x50
> [4.841190]genpd_lock_mtx+0x20/0x2c
> [4.845369]genpd_runtime_resume+0x140/0x434
> [4.850241]__rpm_callback+0xb0/0x1e4
> [4.854506]rpm_callback+0x54/0x1a8
> [4.858597]rpm_resume+0xc6c/0x10c4
> [4.862689]__pm_runtime_resume+0xb4/0x124
> [4.867387]device_link_add+0x598/0x8d0
For this looks as if you have also patch
[PATCH v2 04/12] memory: mtk-smi: Add device-link between smi-larb and
smi-common
from series
[PATCH v2 00/12] Clean up "mediatek,larb" after adding device_link
applied.
Regards,
Matthias
> [4.871829]mtk_smi_larb_probe+0x2b0/0x340
> [4.876528]platform_drv_probe+0xf4/0x134
> [4.881141]really_probe+0x214/0x4dc
> [4.885320]driver_probe_device+0xcc/0x1d4
> [4.890020]__device_attach_driver+0x10c/0x180
> [4.895066]bus_for_each_drv+0x124/0x184
> [4.899591]__device_attach+0x1c0/0x2d8
> [4.904031]device_initial_probe+0x20/0x2c
> [4.908730]bus_probe_device+0x80/0x16c
> [4.913169]deferred_probe_work_func+0x120/0x168
> [4.918387]process_one_work+0x858/0x1208
> [4.923000]worker_thread+0x9ec/0xcb8
> [4.927264]kthread+0x2b8/0x2d0
> [4.931009]ret_from_fork+0x10/0x18
> [4.935098]
> [4.935098] -> #2 (dpm_list_mtx){+.+.}:
> [4.940412]__mutex_lock_common+0x1a0/0x1fe8
> [4.945284]mutex_lock_nested+0x40/0x50
> [4.949722]device_pm_lock+0x1c/0x24
> [4.953900]device_link_add+0x98/0x8d0
> [4.958252]_regulator_get+0x3f0/0x504
> [4.962606]_devm_regulator_get+0x58/0xb8
> [4.967218]devm_regulator_get+0x28/0x34
> [4.971746]pwm_backlight_probe+0x61c/0x1b90
> [4.976617]platform_drv_probe+0xf4/0x134
> [4.981230]really_probe+0x214/0x4dc
> [4.985409]driver_probe_device+0xcc/0x1d4
> [4.990108]device_driver_attach+0xe4/0x104
> [4.994894]__driver_attach+0x134/0x14c
> [4.999333]bus_for_each_dev+0x120/0x180
> [5.003859]driver_attach+0x48/0x54
> [5.007950]bus_add_driver+0x2ac/0x44c
> [5.012303]driver_register+0x160/0x288
> [5.016742]
[RFC PATCH v7 4/5] block: add a helper function to merge the segments
This patch adds a helper function whether a queue can merge
the segments by the DMA MAP layer (e.g. via IOMMU).
Signed-off-by: Yoshihiro Shimoda
---
block/blk-settings.c | 22 ++
include/linux/blkdev.h | 2 ++
2 files changed, 24 insertions(+)
diff --git a/block/blk-settings.c b/block/blk-settings.c
index 45f2c52..6a78ea0 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -4,6 +4,7 @@
*/
#include
#include
+#include
#include
#include
#include
@@ -831,6 +832,27 @@ void blk_queue_write_cache(struct request_queue *q, bool
wc, bool fua)
}
EXPORT_SYMBOL_GPL(blk_queue_write_cache);
+/**
+ * blk_queue_can_use_dma_map_merging - configure queue for merging segments.
+ * @q: the request queue for the device
+ * @dev: the device pointer for dma
+ *
+ * Tell the block layer about merging the segments by dma map of @q.
+ */
+bool blk_queue_can_use_dma_map_merging(struct request_queue *q,
+ struct device *dev)
+{
+ unsigned long boundary = dma_get_merge_boundary(dev);
+
+ if (!boundary)
+ return false;
+
+ /* No need to update max_segment_size. see blk_queue_virt_boundary() */
+ blk_queue_virt_boundary(q, boundary);
+
+ return true;
+}
+
static int __init blk_settings_init(void)
{
blk_max_low_pfn = max_low_pfn - 1;
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 592669b..a7a839d 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1091,6 +1091,8 @@ extern void blk_queue_dma_alignment(struct request_queue
*, int);
extern void blk_queue_update_dma_alignment(struct request_queue *, int);
extern void blk_queue_rq_timeout(struct request_queue *, unsigned int);
extern void blk_queue_write_cache(struct request_queue *q, bool enabled, bool
fua);
+extern bool blk_queue_can_use_dma_map_merging(struct request_queue *q,
+ struct device *dev);
/*
* Number of physical segments as sent to the device.
--
2.7.4
[RFC PATCH v7 0/5] treewide: improve R-Car SDHI performance
This patch series is based on iommu.git / next branch.
Since SDHI host internal DMAC of the R-Car Gen3 cannot handle two or
more segments, the performance rate (especially, eMMC HS400 reading)
is not good. However, if IOMMU is enabled on the DMAC, since IOMMU will
map multiple scatter gather buffers as one contignous iova, the DMAC can
handle the iova as well and then the performance rate is possible to
improve. In fact, I have measured the performance by using bonnie++,
"Sequential Input - block" rate was improved on r8a7795.
To achieve this, this patch series modifies IOMMU and Block subsystem
at first. Since I'd like to get any feedback from each subsystem whether
this way is acceptable for upstream, I submit it to treewide with RFC.
Changes from v6:
- [1/5 for DMA MAP] A new patch.
- [2/5 for IOMMU] A new patch.
- [3/5 for BLOCK] Add Reviewed-by.
- [4/5 for BLOCK] Use a new DMA MAP API instead of device_iommu_mapped().
- [5/5 for MMC] Likewise, and some minor fix.
- Remove patch 4/5 of v6 from this v7 patch series.
https://patchwork.kernel.org/project/linux-renesas-soc/list/?series=131769
Changes from v5:
- Almost all patches are new code.
- [4/5 for MMC] This is a refactor patch so that I don't add any
{Tested,Reviewed}-by tags.
- [5/5 for MMC] Modify MMC subsystem to use bigger segments instead of
the renesas_sdhi driver.
- [5/5 for MMC] Use BLK_MAX_SEGMENTS (128) instead of local value
SDHI_MAX_SEGS_IN_IOMMU (512). Even if we use BLK_MAX_SEGMENTS,
the performance is still good.
https://patchwork.kernel.org/project/linux-renesas-soc/list/?series=127511
Changes from v4:
- [DMA MAPPING] Add a new device_dma_parameters for iova contiguous.
- [IOMMU] Add a new capable for "merging" segments.
- [IOMMU] Add a capable ops into the ipmmu-vmsa driver.
- [MMC] Sort headers in renesas_sdhi_core.c.
- [MMC] Remove the following codes that made on v3 that can be achieved by
DMA MAPPING and IOMMU subsystem:
-- Check if R-Car Gen3 IPMMU is used or not on patch 3.
-- Check if all multiple segment buffers are aligned to PAGE_SIZE on patch 3.
https://patchwork.kernel.org/project/linux-renesas-soc/list/?series=125593
Changes from v3:
- Use a helper function device_iommu_mapped on patch 1 and 3.
- Check if R-Car Gen3 IPMMU is used or not on patch 3.
- Check if all multiple segment buffers are aligned to PAGE_SIZE on patch 3.
- Add Reviewed-by Wolfram-san on patch 1 and 2. Note that I also got his
Reviewed-by on patch 3, but I changed it from v2. So, I didn't add
his Reviewed-by at this time.
https://patchwork.kernel.org/project/linux-renesas-soc/list/?series=120985
Changes from v2:
- Add some conditions in the init_card().
- Add a comment in the init_card().
- Add definitions for some "MAX_SEGS".
https://patchwork.kernel.org/project/linux-renesas-soc/list/?series=116729
Changes from v1:
- Remove adding init_card ops into struct tmio_mmc_dma_ops and
tmio_mmc_host and just set init_card on renesas_sdhi_core.c.
- Revise typos on "mmc: tmio: No memory size limitation if runs on IOMMU".
- Add Simon-san's Reviewed-by on a tmio patch.
https://patchwork.kernel.org/project/linux-renesas-soc/list/?series=110485
Yoshihiro Shimoda (5):
dma: Introduce dma_get_merge_boundary()
iommu/dma: Add a new dma_map_ops of get_merge_boundary()
block: sort headers on blk-setting.c
block: add a helper function to merge the segments
mmc: queue: Use bigger segments if DMA MAP layer can merge the
segments
Documentation/DMA-API.txt | 8
block/blk-settings.c| 34 --
drivers/iommu/dma-iommu.c | 11 +++
drivers/mmc/core/queue.c| 35 ---
include/linux/blkdev.h | 2 ++
include/linux/dma-mapping.h | 6 ++
include/linux/mmc/host.h| 1 +
kernel/dma/mapping.c| 11 +++
8 files changed, 99 insertions(+), 9 deletions(-)
--
2.7.4
[RFC PATCH v7 1/5] dma: Introduce dma_get_merge_boundary()
This patch adds a new DMA API "dma_get_merge_boundary". This function
returns the DMA merge boundary if the DMA layer can merge the segments.
This patch also adds the implementation for a new dma_map_ops pointer.
Signed-off-by: Yoshihiro Shimoda
---
Documentation/DMA-API.txt | 8
include/linux/dma-mapping.h | 6 ++
kernel/dma/mapping.c| 11 +++
3 files changed, 25 insertions(+)
diff --git a/Documentation/DMA-API.txt b/Documentation/DMA-API.txt
index 0076150..11a2647 100644
--- a/Documentation/DMA-API.txt
+++ b/Documentation/DMA-API.txt
@@ -204,6 +204,14 @@ Returns the maximum size of a mapping for the device. The
size parameter
of the mapping functions like dma_map_single(), dma_map_page() and
others should not be larger than the returned value.
+::
+
+ unsigned long
+ dma_get_merge_boundary(struct device *dev);
+
+Returns the DMA merge boundary. If the device cannot merge any the DMA address
+segments, the function returns 0.
+
Part Id - Streaming DMA mappings
diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
index 6309a72..e81e076 100644
--- a/include/linux/dma-mapping.h
+++ b/include/linux/dma-mapping.h
@@ -131,6 +131,7 @@ struct dma_map_ops {
int (*dma_supported)(struct device *dev, u64 mask);
u64 (*get_required_mask)(struct device *dev);
size_t (*max_mapping_size)(struct device *dev);
+ unsigned long (*get_merge_boundary)(struct device *dev);
};
#define DMA_MAPPING_ERROR (~(dma_addr_t)0)
@@ -467,6 +468,7 @@ int dma_set_mask(struct device *dev, u64 mask);
int dma_set_coherent_mask(struct device *dev, u64 mask);
u64 dma_get_required_mask(struct device *dev);
size_t dma_max_mapping_size(struct device *dev);
+unsigned long dma_get_merge_boundary(struct device *dev);
#else /* CONFIG_HAS_DMA */
static inline dma_addr_t dma_map_page_attrs(struct device *dev,
struct page *page, size_t offset, size_t size,
@@ -572,6 +574,10 @@ static inline size_t dma_max_mapping_size(struct device
*dev)
{
return 0;
}
+static inline unsigned long dma_get_merge_boundary(struct device *dev)
+{
+ return 0;
+}
#endif /* CONFIG_HAS_DMA */
static inline dma_addr_t dma_map_single_attrs(struct device *dev, void *ptr,
diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c
index f7afdad..8e262cf 100644
--- a/kernel/dma/mapping.c
+++ b/kernel/dma/mapping.c
@@ -367,3 +367,14 @@ size_t dma_max_mapping_size(struct device *dev)
return size;
}
EXPORT_SYMBOL_GPL(dma_max_mapping_size);
+
+unsigned long dma_get_merge_boundary(struct device *dev)
+{
+ const struct dma_map_ops *ops = get_dma_ops(dev);
+
+ if (!ops || !ops->get_merge_boundary)
+ return 0; /* can't merge */
+
+ return ops->get_merge_boundary(dev);
+}
+EXPORT_SYMBOL_GPL(dma_get_merge_boundary);
--
2.7.4
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
[RFC PATCH v7 3/5] block: sort headers on blk-setting.c
This patch sorts the headers in alphabetic order to ease the maintenance for this part. Signed-off-by: Yoshihiro Shimoda Reviewed-by: Wolfram Sang --- block/blk-settings.c | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/block/blk-settings.c b/block/blk-settings.c index 2ae348c..45f2c52 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -2,16 +2,16 @@ /* * Functions related to setting various queue properties from drivers */ -#include -#include -#include #include #include -#include /* for max_pfn/max_low_pfn */ #include -#include -#include #include +#include +#include +#include +#include +#include /* for max_pfn/max_low_pfn */ +#include #include "blk.h" #include "blk-wbt.h" -- 2.7.4
[RFC PATCH v7 5/5] mmc: queue: Use bigger segments if DMA MAP layer can merge the segments
When the max_segs of a mmc host is smaller than 512, the mmc
subsystem tries to use 512 segments if DMA MAP layer can merge
the segments, and then the mmc subsystem exposes such information
to the block layer by using blk_queue_can_use_dma_map_merging().
Signed-off-by: Yoshihiro Shimoda
---
drivers/mmc/core/queue.c | 35 ---
include/linux/mmc/host.h | 1 +
2 files changed, 33 insertions(+), 3 deletions(-)
diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c
index 92900a0..ab0ecc6 100644
--- a/drivers/mmc/core/queue.c
+++ b/drivers/mmc/core/queue.c
@@ -24,6 +24,8 @@
#include "card.h"
#include "host.h"
+#define MMC_DMA_MAP_MERGE_SEGMENTS 512
+
static inline bool mmc_cqe_dcmd_busy(struct mmc_queue *mq)
{
/* Allow only 1 DCMD at a time */
@@ -196,6 +198,12 @@ static void mmc_queue_setup_discard(struct request_queue
*q,
blk_queue_flag_set(QUEUE_FLAG_SECERASE, q);
}
+static unsigned int mmc_get_max_segments(struct mmc_host *host)
+{
+ return host->can_dma_map_merge ? MMC_DMA_MAP_MERGE_SEGMENTS :
+host->max_segs;
+}
+
/**
* mmc_init_request() - initialize the MMC-specific per-request data
* @q: the request queue
@@ -209,7 +217,7 @@ static int __mmc_init_request(struct mmc_queue *mq, struct
request *req,
struct mmc_card *card = mq->card;
struct mmc_host *host = card->host;
- mq_rq->sg = mmc_alloc_sg(host->max_segs, gfp);
+ mq_rq->sg = mmc_alloc_sg(mmc_get_max_segments(host), gfp);
if (!mq_rq->sg)
return -ENOMEM;
@@ -368,13 +376,23 @@ static void mmc_setup_queue(struct mmc_queue *mq, struct
mmc_card *card)
blk_queue_bounce_limit(mq->queue, limit);
blk_queue_max_hw_sectors(mq->queue,
min(host->max_blk_count, host->max_req_size / 512));
- blk_queue_max_segments(mq->queue, host->max_segs);
+ if (host->can_dma_map_merge)
+ WARN(!blk_queue_can_use_dma_map_merging(mq->queue,
+ mmc_dev(host)),
+"merging was advertised but not possible");
+ blk_queue_max_segments(mq->queue, mmc_get_max_segments(host));
if (mmc_card_mmc(card))
block_size = card->ext_csd.data_sector_size;
blk_queue_logical_block_size(mq->queue, block_size);
- blk_queue_max_segment_size(mq->queue,
+ /*
+* After blk_queue_can_use_dma_map_merging() was called with succeed,
+* since it calls blk_queue_virt_boundary(), the mmc should not call
+* both blk_queue_max_segment_size().
+*/
+ if (host->can_dma_map_merge)
+ blk_queue_max_segment_size(mq->queue,
round_down(host->max_seg_size, block_size));
dma_set_max_seg_size(mmc_dev(host), queue_max_segment_size(mq->queue));
@@ -424,6 +442,17 @@ int mmc_init_queue(struct mmc_queue *mq, struct mmc_card
*card)
mq->tag_set.cmd_size = sizeof(struct mmc_queue_req);
mq->tag_set.driver_data = mq;
+ /*
+* Since blk_mq_alloc_tag_set() calls .init_request() of mmc_mq_ops,
+* the host->can_dma_map_merge should be set before to get max_segs
+* from mmc_get_max_segments().
+*/
+ if (host->max_segs < MMC_DMA_MAP_MERGE_SEGMENTS &&
+ dma_get_merge_boundary(mmc_dev(host)))
+ host->can_dma_map_merge = 1;
+ else
+ host->can_dma_map_merge = 0;
+
ret = blk_mq_alloc_tag_set(>tag_set);
if (ret)
return ret;
diff --git a/include/linux/mmc/host.h b/include/linux/mmc/host.h
index 43d0f0c..10c3719 100644
--- a/include/linux/mmc/host.h
+++ b/include/linux/mmc/host.h
@@ -398,6 +398,7 @@ struct mmc_host {
unsigned intretune_now:1; /* do re-tuning at next req */
unsigned intretune_paused:1; /* re-tuning is temporarily
disabled */
unsigned intuse_blk_mq:1; /* use blk-mq */
+ unsigned intcan_dma_map_merge:1; /* merging can be used */
int rescan_disable; /* disable card detection */
int rescan_entered; /* used with nonremovable
devices */
--
2.7.4
[RFC PATCH v7 2/5] iommu/dma: Add a new dma_map_ops of get_merge_boundary()
This patch adds a new dma_map_ops of get_merge_boundary() to
expose the DMA merge boundary if the domain type is IOMMU_DOMAIN_DMA.
Signed-off-by: Yoshihiro Shimoda
---
drivers/iommu/dma-iommu.c | 11 +++
1 file changed, 11 insertions(+)
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 205d694..9950cb5 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1091,6 +1091,16 @@ static int iommu_dma_get_sgtable(struct device *dev,
struct sg_table *sgt,
return ret;
}
+static unsigned long iommu_dma_get_merge_boundary(struct device *dev)
+{
+ struct iommu_domain *domain = iommu_get_dma_domain(dev);
+
+ if (domain->type != IOMMU_DOMAIN_DMA)
+ return 0; /* can't merge */
+
+ return (1 << __ffs(domain->pgsize_bitmap)) - 1;
+}
+
static const struct dma_map_ops iommu_dma_ops = {
.alloc = iommu_dma_alloc,
.free = iommu_dma_free,
@@ -1106,6 +1116,7 @@ static const struct dma_map_ops iommu_dma_ops = {
.sync_sg_for_device = iommu_dma_sync_sg_for_device,
.map_resource = iommu_dma_map_resource,
.unmap_resource = iommu_dma_unmap_resource,
+ .get_merge_boundary = iommu_dma_get_merge_boundary,
};
/*
--
2.7.4
