You are right, process 1 is always allocating 1GB even if I request only 10MB, and memseg->hugepage_sz is 1GB. When I use rte_memseg_get_fd_offset I get an FD and the offset is 0 (correct) because that's the memseg offset not the memzone. To access the memory allocated by process 1 I need to take into account also the offset between memzone and memseg. And then I need to add (memzone->iova - memseg->iova) to the address returned by mmap.
On Thu, Jul 7, 2022 at 2:26 AM Dmitry Kozlyuk <[email protected]> wrote: > > 2022-07-07 00:14 (UTC+0200), Antonio Di Bacco: > > Dear Dmitry, > > > > I tried to follow this approach and if I allocate 1GB on primary > > process number 1, then I can mmap that memory on the primary process > > number 2. > > I also tried to convert the virt addr of the allocation made in > > primary 1 to phys and then I converted the virt addr returned by mmap > > in primary 2 and I got the same phys addr. > > > > Unfortunately, if I try to allocated only 10 MB for example in primary > > 1, then mmap in primary 2 succeeds but it seems that this virt addr > > doesn't correspond to the same phys memory as in primary 1. > > > > In the primary 2, the mmap is used like this: > > > > int flags = MAP_SHARED | MAP_HUGETLB ; > > > > uint64_t* addr = (uint64_t*) mmap(NULL, sz, PROT_READ|PROT_WRITE, > > flags, my_mem_fd, off); > > Hi Antonio, > > From `man 2 mmap`: > > Huge page (Huge TLB) mappings > For mappings that employ huge pages, the requirements for the > arguments of mmap() and munmap() differ somewhat from the requirements > for mappings that use the native system page size. > > For mmap(), offset must be a multiple of the underlying huge page > size. The system automatically aligns length to be a multiple of > the underlying huge page size. > > For munmap(), addr, and length must both be a multiple of the > underlying huge page size. > > Probably process 1 maps a 1 GB hugepage: > DPDK does so if 1 GB hugepages are used even if you only allocate 10 MB. > You can examine memseg to see what size it is (not memzone!). > Hugepage size is a property of each mounted HugeTBL filesystem. > It determines which kernel pool to use. > Pools are over different sets of physical pages. > This means that the kernel doesn't allow to map given page frames > as 1 GB and 2 MB hugepages at the same time via hugetlbfs. > I'm surprised mmap() works at all in your case > and suspect that it is mapping 2 MB hugepages in process 2. > > The solution may be, in process 2: > > base_offset = RTE_ALIGN_FLOOR(offset, hugepage_size) > map_addr = mmap(fd, size=hugepage_size, offset=base_offset) > addr = RTE_PTR_ADD(map_addr, offset - base_offset) > > Note that if [offset; offset+size) crosses a hugepage boundary, > you have to map more than one page.
