Because vfio and uio can be used at the same time, we need save the value of
requested_addr at?pci_vfio_map_resource() and?pci_uio_map_resource().Your guys
like add a?global?variable "void * pci_requested_addr" or a "void**
requested_addr" parameter at?pci_vfio_map_resource() and?pci_uio_map_resource()
?------------------------------------------------------------------From:Burakov,
Anatoly <anatoly.burakov at intel.com>Time:2014 Nov 7 (Fri) 18 : 03To:??
<liang.xu at cinfotech.cn>, dev at dpdk.org <dev at dpdk.org>Cc:thomas.monjalon
at 6wind.com <thomas.monjalon at 6wind.com>, De Lara Guarch, Pablo
<pablo.de.lara.guarch at intel.com>Subject:RE: [PATCH v6] eal: map uio
resources after hugepages when the base_virtaddr is configured.
font-family: Wingdings;panose-1: 5 0 0 0 0 0 0 0 0 0;font-family: MS
Gothic;panose-1: 2 11 6 9 7 2 5 8 2 4;font-family: Cambria Math;panose-1: 2 4 5
3 5 4 6 3 2 4;font-family: Calibri;panose-1: 2 15 5 2 2 2 4 3 2 4;font-family:
Tahoma;panose-1: 2 11 6 4 3 5 4 4 2 4;font-family: \@MS Gothic;panose-1: 2 11 6
9 7 2 5 8 2 4;p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:
0.0in;margin-bottom: 1.0E-4pt;font-size: 12.0pt;font-family: Times New Roman ,
serif;}
a:link, span.MsoHyperlink {mso-style-priority: 99;color:
#0563c1;text-decoration: underline;}
a:visited, span.MsoHyperlinkFollowed {mso-style-priority: 99;color:
#954f72;text-decoration: underline;}
span.EmailStyle17 {mso-style-type: personal-reply;font-family: Calibri ,
sans-serif;color: #1f497d;}
*.MsoChpDefault {mso-style-type: export-only;font-family: Calibri , sans-serif;}
size: 8.5in 11.0in;margin: 1.0in 1.0in 1.0in 1.0in;div.WordSection1 {page:
WordSection1;}
Ah, yes, sorry about that, haven?t quite woke up yet
L You?re right. So it?s removing the dependency on --base-virtaddr, moving the
function to eal_pci.c
?and adding similar VFIO code that?s left then.
?
Thanks,
Anatoly
?
From: XU Liang [mailto:liang...@cinfotech.cn]
Sent: Friday, November 7, 2014 9:57 AM
To: Burakov, Anatoly; dev at dpdk.org
Cc: thomas.monjalon at 6wind.com; De Lara Guarch, Pablo
Subject: Re: [PATCH v6] eal: map uio resources after hugepages when the
base_virtaddr is configured.
?
How to find the?maximum end virtual address ??I'm not the DPDK expert, but I
will try to do my best.
?
If the segments isn't overlap, "if(seg->addr > last->addr)?last = seg;" already
find the segment with?maximum end virtual address.
?
------------------------------------------------------------------
From:Burakov, Anatoly <anatoly.burakov at intel.com>
Time:2014 Nov 7 (Fri) 17 : 47
To:??
<liang.xu at cinfotech.cn>,
dev at dpdk.org <dev at dpdk.org>
Cc:thomas.monjalon at 6wind.com <thomas.monjalon at 6wind.com>, De Lara Guarch,
Pablo <pablo.de.lara.guarch at intel.com>
Subject:RE: [PATCH v6] eal: map uio resources after hugepages when the
base_virtaddr is configured.
?
The commit message looks fine to me, but VFIO code needs to be adjusted the
same way.
Also, now that I think of it, you can't simply assume that whatever last memseg
you have has the latest virtual address. When IVSHMEM is initialized, it too
reserves some space in the virtual memory, which can be higher than the last
hugepage, but not be the
last hugepage (because IVSHMEM memory is first to be reserved, before the main
memory).
My advice would be to rewrite the function to return the maximum end virtual
address (instead of a last segment) and move it to eal_pci.c (and include
declaration for it in include/eal_pci_init.h).
My apologies for not thinking about any of this during the first 6 iterations
of the patch :(
Thanks,
Anatoly
-----Original Message-----
From: lxu [mailto:liang...@cinfotech.cn]
Sent: Friday, November 7, 2014 8:01 AM
To: dev at dpdk.org
Cc: Burakov, Anatoly; thomas.monjalon at 6wind.com; De Lara Guarch, Pablo
Subject: [PATCH v6] eal: map uio resources after hugepages when the
base_virtaddr is configured.
A multiple process DPDK application must mmap hugepages and pci resource into
same virtual addresses. By default the virtual addresses chosen by the primary
process automatically when calling the mmap. But sometime the virtual addresses
chosen by the primary
process isn't usable at secondary process. Such as the secondary process
linked with more libraries than primary process. The library has been mapped
into this virtual address. The command line parameter 'base-virtaddr' has been
added for this situation. If
it's configured, the hugepages will be mapped into this base address. But the
virtual address of uio resource mapped still does not refer to the parameter.
In that case it would still fail.
This patch try to map uio resources after hugepages when the base_virtaddr is
configured. So the error of "EAL: pci_map_resource(): cannot mmap" can be
resolved by set base-virtaddr into free virtual address space.
Signed-off-by: lxu <liang.xu at cinfotech.cn>
---
lib/librte_eal/linuxapp/eal/eal_pci_uio.c | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 7e62266..a2c9ab6 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -273,6 +273,24 @@ pci_get_uio_dev(struct rte_pci_device *dev, char *dstbuf,
return uio_num;
}
+static inline const struct rte_memseg *
+get_physmem_last(void)
+{
+ const struct rte_memseg * seg = rte_eal_get_physmem_layout();
+ const struct rte_memseg * last = seg;
+ unsigned i = 0;
+
+ for (i=0; i<RTE_MAX_MEMSEG; i++, seg++) {
+ if (seg->addr == NULL)
+ break;
+
+ if(seg->addr > last->addr)
+ last = seg;
+
+ }
+ return last;
+}
+
/* map the PCI resource of a PCI device in virtual memory */ int
pci_uio_map_resource(struct rte_pci_device *dev) @@ -290,6 +308,13 @@
pci_uio_map_resource(struct rte_pci_device *dev)
struct mapped_pci_resource *uio_res;
struct pci_map *maps;
+ /* map uio resource into user required virtual address */
+ static void * requested_addr;
+ if (internal_config.base_virtaddr && NULL == requested_addr) {
+ const struct rte_memseg * last = get_physmem_last();
+ requested_addr = RTE_PTR_ADD(last->addr, last->len);
+ }
+
dev->intr_handle.fd = -1;
dev->intr_handle.type = RTE_INTR_HANDLE_UNKNOWN;
@@ -371,10 +396,12 @@ pci_uio_map_resource(struct rte_pci_device *dev)
if (maps[j].addr != NULL)
fail = 1;
else {
- mapaddr = pci_map_resource(NULL, fd, (off_t)offset,
+ mapaddr = pci_map_resource(requested_addr, fd, (off_t)offset,
(size_t)maps[j].size);
if (mapaddr == NULL)
fail = 1;
+ else if (NULL != requested_addr)
+ requested_addr = RTE_PTR_ADD(mapaddr, maps[j].size);
}
if (fail) {
--
1.9.1
