[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-06-07 Thread Seth Forshee
** Changed in: linux (Ubuntu Eoan) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1823753 Title: arm64: cma_alloc errors at boot Status in

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-06-04 Thread dann frazier
** Description changed: [Impact] We enabled CONFIG_DMA_CMA to fix bug 1803206, but that led to a regression on other arm64 systems that began spewing these messages on boot - sometimes > 10K of them: [ 19.534097] cma: cma_alloc: alloc failed, req-size: 64 pages, ret: -12 [

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-06-04 Thread dann frazier
** Description changed: - On some arm64 systems[*] we are seeing a spew of messages on the - console: + [Impact] + We enabled CONFIG_DMA_CMA to fix bug 1803206, but that led to a regression + where other systems began spewing on the order of 10K of these messages on boot: [ 19.534097]

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-30 Thread dann frazier
Thanks Paolo and John. I've uploaded a cma.4 kernel to ppa:dannf/cma that includes the following additional fix: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux- next.git/commit/?id=e7a647580852f90b7f9a5b940e7eaffa105f6271 I'll get some non-ARM architecture testing on that before

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-30 Thread John Garry
Note: We have tested https://lkml.org/lkml/2019/5/6/1221, which is now on linux-next *, and the hi162 RoCE driver single patch usage is diminished: $cat cma.disco.dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count ([0-9]+)\,.*/\1/' | sort -n | uniq -c 129 2 3 4 206 8 32 16 2 24 4 32

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-27 Thread Paolo Pisati
Here are dmesg and meminfo for all 3 cases (generuic, cma3 and cma3=64M): https://people.canonical.com/~ppisati/lp1823753/ Other than the different memory alloaction, i didn't notice any evident regression. -- You received this bug notification because you are a member of Kernel Packages,

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-24 Thread dann frazier
I forgot to enable CMA_DEBUGFS in those builds, so I've uploaded another (cma.3) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1823753 Title: arm64: cma_alloc errors at boot Status in

Re: [Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-23 Thread dann frazier
On Thu, May 23, 2019 at 8:31 AM Paolo Pisati <1823...@bugs.launchpad.net> wrote: > > Yes, i have all the hardware to do some testing, prepare some kernels > and we can take it from there. Thanks Paolo. I pushed a test build to ppa:dannf/cma. This bumps cma to 32M, but would also be good to know

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-23 Thread Paolo Pisati
Yes, i have all the hardware to do some testing, prepare some kernels and we can take it from there. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1823753 Title: arm64: cma_alloc

Re: [Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-22 Thread dann frazier
On Wed, May 22, 2019 at 8:41 AM Paolo Pisati <1823...@bugs.launchpad.net> wrote: > > I realize this is just a workaround (and the above 31M cma memory > fragmentation is ugly), but we should definitely bump CMA allocation > space: definitely 32M (since that's what upstream default to) but if 64M >

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-22 Thread Paolo Pisati
I realize this is just a workaround (and the above 31M cma memory fragmentation is ugly), but we should definitely bump CMA allocation space: definitely 32M (since that's what upstream default to) but if 64M solves a problem you have at the moment (until we sort out the driver issue), i'm not

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-14 Thread dann frazier
The hisi_sas maintainer is experimenting with a patch that will change the driver's 33 page allocations to single page allocations. If there is no significant performance impact, that could be a way forward to deal with the fragmentation issue costing us up to 31M of CMA. In addition, there is

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-07 Thread dann frazier
I've split the ratelimiting topic out to bug 1828092 so to keep this one open while we continue to try and solve the underlying cause of the messages. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-07 Thread dann frazier
Why would it fail to init? If CMA allocation fails, the kernel will fulfill the request using alloc_pages_node(): struct page *__dma_direct_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { [...] /* CMA can be used only

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-07 Thread Fred Kimmy
I find this "Mellonox 25Gigabit-64bit-SFP28-2port– PCIe3.0 X8–15B3-1015-2" card can require 260M CMA memory. If we insert 2 or 3...4 Mellonox card, this card will fail to init it, so we should disable this cma config or set 1024M or above CMA memory. -- You received this bug notification because

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-06 Thread dann frazier
As per Seth's suggestion, I tried ratelimiting the cma_alloc messages: diff --git a/mm/cma.c b/mm/cma.c index c7b39dd3b4f6..56d2a046f689 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -477,7 +477,7 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align,

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-06 Thread dann frazier
Note that in current disco things have gotten worse - our single page allocations have blown up to 11753: $ cat cma.disco.dmesg | grep "cma: cma_alloc(cma" | sed -r 's/.*count ([0-9]+)\,.*/\1/' | sort -n | uniq -c 11753 1 3 2 3 4 234 8 32 16 2 24 4 32 256

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-06 Thread dann frazier
The culprit for the 256 33 page allocations (that causes the fragmentation mentioned in comment #8) is the hisi_sas_v3_hw driver: [ 21.220867] Call trace: [ 21.223301] dump_backtrace+0x0/0x1b0 [ 21.226948] show_stack+0x24/0x30 [ 21.230251] dump_stack+0x90/0xb4 [ 21.233554]

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-03 Thread Seth Forshee
That wastage is pretty awful. You might want try a patch to at least ratelimit the cma_alloc errors. Not that it's a real solution, but it might be something we can apply to mitigate the problem short-term while continuing to search for something better. -- You received this bug notification

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-03 Thread dann frazier
To try to understand how the CMA is allocated - and what the problem might be - I booted a HiSilicon D06 w/ plenty of cma cushion (cma=128M), and looked at cma debugfs. In the upstream thread, Robin asked if this was potentially an issue with lots of 1 page allocations - something for which

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-05-03 Thread Terry Rudd
** Changed in: linux (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1823753 Title: arm64: cma_alloc errors at boot Status in linux package in

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-04-25 Thread Seth Forshee
I only had time to look briefly, but here's a few notes. Firstly, a workaround for now is passing cma=0 on the kernel command line. I feel like there must be something wrong here if everything just works fine with CMA turned off, because obviously devices are getting CMA allocations when they

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-04-24 Thread dann frazier
Note that this appears to be a regression introduced by bug 1803206. Disabling DMA_CMA avoids it - but obviously would regress bug 1803206. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-04-17 Thread dann frazier
Upstream thread: http://lists.infradead.org/pipermail/linux-arm-kernel/2019-April/647345.html -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1823753 Title: arm64: cma_alloc errors at

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-04-08 Thread dann frazier
** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1823753 Title: arm64: cma_alloc errors at boot Status in linux

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-04-08 Thread dann frazier
I enabled CMA_DEBUG & CMA_DEBUGFS and booted w/ cma=128M so that we can see the CMA state when no allocations fail. root@d06-4:/sys/kernel/debug/cma/cma-reserved# cat count 32768 root@d06-4:/sys/kernel/debug/cma/cma-reserved# cat used 15630 Looks like we're actually using ~61M - and perhaps

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-04-08 Thread dann frazier
This was not an issue prior to v4.20. Bisection identified the following commit: 886643b766321 arm64: use the generic swiotlb_dma_ops -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu.

[Kernel-packages] [Bug 1823753] Re: arm64: cma_alloc errors at boot

2019-04-08 Thread dann frazier
The disco kernel uses CONFIG_CMA_SIZE_MBYTES=16, while the arm64 defconfig sets it to 32. I tried bumping up this setting by powers of 2 until the messages went away, and it finally did at 128, which seems extreme. Need to figure out what is attempting these allocations, and why they are failing.