Re: DMA: atmel_serial: Opening and closing the serial device repeatedly causes kmalloc-32 slab leak
Le 27/11/2018 à 10:58, Alexandre Belloni a écrit :
> Hello Richard,
>
> On 27/11/2018 10:51:13+0100, richard.gen...@gmail.com wrote:
>> Hi all,
>>
>> I reproduced the memory leak on my board (at91sam9g35-cm) with a 4.20-rc3.
>>
>> It triggered an OOM after a couple of hours running a code like this:
>> #include
>> #include
>> #include
>> #include
>>
>>
>> int main(int argc, char **argv)
>> {
>> int fd;
>> do {
>> fd = open("/dev/ttyS1", O_RDONLY);
>> close(fd);
>> } while (true);
>> return 0;
>> }
>>
>> As Mario pointed out, this only happens when atmel,use-dma-{r,t}x are
>> used in the device-tree.
>>
>> Adding:
>> CONFIG_DEBUG_SLAB=y
>> CONFIG_DEBUG_SLAB_LEAK=y
>> Doesn't show anything suspect in /proc/slab_allocators
>>
>> From what I found until now, it's something done in :
>> dma_request_slave_channel();
>> that leaks kmalloc-32
>> Mabe I missed something, but it seems that everything DMA related is
>> deallocated in atmel_release_{tx,rx}_dma().
>>
>> Is this ringing a bell ?
>>
>
> Yes, this is known issue and it has yet to be worked on.
>
After a talk on freenode, Alex found the problem.
A patch is on its way.
Thanks !
Re: DMA: atmel_serial: Opening and closing the serial device repeatedly causes kmalloc-32 slab leak
Hello Richard,
On 27/11/2018 10:51:13+0100, richard.gen...@gmail.com wrote:
> Hi all,
>
> I reproduced the memory leak on my board (at91sam9g35-cm) with a 4.20-rc3.
>
> It triggered an OOM after a couple of hours running a code like this:
> #include
> #include
> #include
> #include
>
>
> int main(int argc, char **argv)
> {
> int fd;
> do {
> fd = open("/dev/ttyS1", O_RDONLY);
> close(fd);
> } while (true);
> return 0;
> }
>
> As Mario pointed out, this only happens when atmel,use-dma-{r,t}x are
> used in the device-tree.
>
> Adding:
> CONFIG_DEBUG_SLAB=y
> CONFIG_DEBUG_SLAB_LEAK=y
> Doesn't show anything suspect in /proc/slab_allocators
>
> From what I found until now, it's something done in :
> dma_request_slave_channel();
> that leaks kmalloc-32
> Mabe I missed something, but it seems that everything DMA related is
> deallocated in atmel_release_{tx,rx}_dma().
>
> Is this ringing a bell ?
>
Yes, this is known issue and it has yet to be worked on.
--
Alexandre Belloni, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
DMA: atmel_serial: Opening and closing the serial device repeatedly causes kmalloc-32 slab leak
[re-sending with Vinod's correct email address. Sorry for the noise. ]
[re-sending the bug report to the lists]
Le 16/11/2018 à 17:04, Mario Forner a écrit :
> Problem:
> When I open and close the serial device /dev/ttyS4 in a loop
> the amount of kmalloc-32 slabs increases slowly but steadily without limit.
>
> The serial device is configured in acme-aria.dts to use DMA.
>
> If DMA is disabled, the amount of kmalloc32-slabs remains constant
> over several hours, just fluctuating slightly.
>
> The serial device is accessed by atmel_serial.c which is evident from
> the drivers kernel log output.
>
> The bug was noticed on a device which had been running over several weeks and
> has accumulated ~86MB of unrelaimable kmalloc-32 slabs by now. Example:
>
> root@master1083:~# slabtop -o | head -10
> Active / Total Objects (% used): 2704880 / 2716124 (99.6%)
> Active / Total Slabs (% used) : 23150 / 23150 (100.0%)
> Active / Total Caches (% used) : 57 / 76 (75.0%)
> Active / Total Size (% used) : 88893.62K / 89964.32K (98.8%)
> Minimum / Average / Maximum Object : 0.02K / 0.03K / 4096.00K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 2674804 2673840 99%0.03K 21571 124 86284K kmalloc-32
> 7200 7104 98%0.08K144 50 576K kernfs_node_cache
> 6930 5370 77%0.13K231 30 924K dentry
>
> root@master1083:~# uptime
> 15:12:55 up 93 days, 16:54, 1 user, load average: 1.59, 2.31, 2.42
>
> Keywords: atmel, serial, kernel, leak, memory, dma, slab, kmalloc
>
> Kernel information:
>
> Kernel version:
> Linux version 4.2.6 (mario@linux-7rm0) (gcc version 4.9.3 (crosstool-NG ) )
> #115 Fri Nov 16 11:05:22 CET 2018
> Linux version 4.9.124 (mario@linux-7rm0) (gcc version 4.9.3 (crosstool-NG ) )
> #16 Fri Nov 16 13:54:25 CET 2018
>
> Most recent Kernel version which did not have the bug: unknown
>
> How to reproduce the bug:
>
> The kmalloc-32 slab count should be monitored every few minutes with "slabtop
> -o".
> The leak can be triggered by running the following script, given DMA has
> been enabled for ttyS4:
>
> #!/usr/bin/env python
> import serial
> import time
>
> while True:
> try:
> with serial.Serial(port = '/dev/ttyS4'):
> pass
> except Exception as e:
> print e
> finally:
> time.sleep(0.5)
> # end script
>
> Environment:
> Software:
> Linux master 4.2.6 #114 Fri Nov 16 10:14:30 CET 2018 armv5tejl GNU/Linux
>
> Binutils2.25
> Util-linux 2.25.2
> Mount 2.25.2
> Module-init-tools 18
> E2fsprogs 1.42.12
> Linux C Library 2.19
> Dynamic linker (ldd)2.19
> Linux C++ Library 6.0.20
> Procps 3.3.9
> Net-tools 1.60
> Sh-utils8.23
> Udev215
> Modules Loaded iptable_nat option usb_wwan usbserial
>
> Processor information:
> # cat /proc/cpuinfo
> processor : 0
> model name : ARM926EJ-S rev 5 (v5l)
> BogoMIPS: 198.76
> Features: swp half thumb fastmult edsp java
> CPU implementer : 0x41
> CPU architecture: 5TEJ
> CPU variant : 0x0
> CPU part: 0x926
> CPU revision: 5
>
> Hardware: Atmel AT91SAM9
> Revision:
> Serial :
>
> Module information:
> # cat /proc/modules
> iptable_nat 1720 0 - Live 0xbf0a5000
> option 28780 0 - Live 0xbf03c000
> usb_wwan 6876 1 option, Live 0xbf024000
> usbserial 23392 2 option,usb_wwan, Live 0xbf00
>
> Loaded drivers:
> # cat /proc/ioports
> # cat /proc/iomem
> 0030-00307fff : 30.sram
> 0060-006f : /ahb/ohci@0060
> 0070-007f : /ahb/ehci@0070
> 2000-2fff : System RAM
> 20008000-20577287 : Kernel code
> 205a8000-205f02b7 : Kernel data
> f000-f0ff : /ahb/apb/spi@f000
> f8008000-f80080ff : /ahb/apb/timer@f8008000
> f800c000-f800c0ff : /ahb/apb/timer@f800c000
> f801-f80100ff : /ahb/apb/i2c@f801
> f8014000-f80140ff : /ahb/apb/i2c@f8014000
> f801c000-f801c1ff : atmel_serial
> f802-f80201ff : atmel_serial
> f8024000-f80241ff : atmel_serial
> f8028000-f80281ff : atmel_serial
> f802c000-f802c0ff : /ahb/apb/ethernet@f802c000
> f8034000-f80342ff : /ahb/apb/pwm@f8034000
> f804c000-f804c0ff : /ahb/apb/adc@f804c000
> ec00-edff : at_hdmac
> ee00-efff : at_hdmac
> f200-f3ff : atmel_serial
> f400-f5ff : /ahb/apb/pinctrl@f400/gpio@f400
> f600-f7ff : /ahb/apb/pinctrl@f400/gpio@f600
> f800-f9ff : /ahb/apb/pinctrl@f400/gpio@f800
> fa00-fbff : /ahb/apb/pinctrl@f400/gpio@fa00
> fe10-fe1f : /ahb/apb/shdwc@fe10
>
> Other Information:
> The serial device ttyS4 is configuered inside .dts file by
>
> usart3: serial@f8028000 {
> status = "okay";
> compatible = "atmel,at91sam9260-usart";
> reg = <0xf8028000 0x
DMA: atmel_serial: Opening and closing the serial device repeatedly causes kmalloc-32 slab leak
[re-sending the bug report to the lists]
Le 16/11/2018 à 17:04, Mario Forner a écrit :
> Problem:
> When I open and close the serial device /dev/ttyS4 in a loop
> the amount of kmalloc-32 slabs increases slowly but steadily without limit.
>
> The serial device is configured in acme-aria.dts to use DMA.
>
> If DMA is disabled, the amount of kmalloc32-slabs remains constant
> over several hours, just fluctuating slightly.
>
> The serial device is accessed by atmel_serial.c which is evident from
> the drivers kernel log output.
>
> The bug was noticed on a device which had been running over several weeks and
> has accumulated ~86MB of unrelaimable kmalloc-32 slabs by now. Example:
>
> root@master1083:~# slabtop -o | head -10
> Active / Total Objects (% used): 2704880 / 2716124 (99.6%)
> Active / Total Slabs (% used) : 23150 / 23150 (100.0%)
> Active / Total Caches (% used) : 57 / 76 (75.0%)
> Active / Total Size (% used) : 88893.62K / 89964.32K (98.8%)
> Minimum / Average / Maximum Object : 0.02K / 0.03K / 4096.00K
>
> OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
> 2674804 2673840 99%0.03K 21571 124 86284K kmalloc-32
> 7200 7104 98%0.08K144 50 576K kernfs_node_cache
> 6930 5370 77%0.13K231 30 924K dentry
>
> root@master1083:~# uptime
> 15:12:55 up 93 days, 16:54, 1 user, load average: 1.59, 2.31, 2.42
>
> Keywords: atmel, serial, kernel, leak, memory, dma, slab, kmalloc
>
> Kernel information:
>
> Kernel version:
> Linux version 4.2.6 (mario@linux-7rm0) (gcc version 4.9.3 (crosstool-NG ) )
> #115 Fri Nov 16 11:05:22 CET 2018
> Linux version 4.9.124 (mario@linux-7rm0) (gcc version 4.9.3 (crosstool-NG ) )
> #16 Fri Nov 16 13:54:25 CET 2018
>
> Most recent Kernel version which did not have the bug: unknown
>
> How to reproduce the bug:
>
> The kmalloc-32 slab count should be monitored every few minutes with "slabtop
> -o".
> The leak can be triggered by running the following script, given DMA has
> been enabled for ttyS4:
>
> #!/usr/bin/env python
> import serial
> import time
>
> while True:
> try:
> with serial.Serial(port = '/dev/ttyS4'):
> pass
> except Exception as e:
> print e
> finally:
> time.sleep(0.5)
> # end script
>
> Environment:
> Software:
> Linux master 4.2.6 #114 Fri Nov 16 10:14:30 CET 2018 armv5tejl GNU/Linux
>
> Binutils2.25
> Util-linux 2.25.2
> Mount 2.25.2
> Module-init-tools 18
> E2fsprogs 1.42.12
> Linux C Library 2.19
> Dynamic linker (ldd)2.19
> Linux C++ Library 6.0.20
> Procps 3.3.9
> Net-tools 1.60
> Sh-utils8.23
> Udev215
> Modules Loaded iptable_nat option usb_wwan usbserial
>
> Processor information:
> # cat /proc/cpuinfo
> processor : 0
> model name : ARM926EJ-S rev 5 (v5l)
> BogoMIPS: 198.76
> Features: swp half thumb fastmult edsp java
> CPU implementer : 0x41
> CPU architecture: 5TEJ
> CPU variant : 0x0
> CPU part: 0x926
> CPU revision: 5
>
> Hardware: Atmel AT91SAM9
> Revision:
> Serial :
>
> Module information:
> # cat /proc/modules
> iptable_nat 1720 0 - Live 0xbf0a5000
> option 28780 0 - Live 0xbf03c000
> usb_wwan 6876 1 option, Live 0xbf024000
> usbserial 23392 2 option,usb_wwan, Live 0xbf00
>
> Loaded drivers:
> # cat /proc/ioports
> # cat /proc/iomem
> 0030-00307fff : 30.sram
> 0060-006f : /ahb/ohci@0060
> 0070-007f : /ahb/ehci@0070
> 2000-2fff : System RAM
> 20008000-20577287 : Kernel code
> 205a8000-205f02b7 : Kernel data
> f000-f0ff : /ahb/apb/spi@f000
> f8008000-f80080ff : /ahb/apb/timer@f8008000
> f800c000-f800c0ff : /ahb/apb/timer@f800c000
> f801-f80100ff : /ahb/apb/i2c@f801
> f8014000-f80140ff : /ahb/apb/i2c@f8014000
> f801c000-f801c1ff : atmel_serial
> f802-f80201ff : atmel_serial
> f8024000-f80241ff : atmel_serial
> f8028000-f80281ff : atmel_serial
> f802c000-f802c0ff : /ahb/apb/ethernet@f802c000
> f8034000-f80342ff : /ahb/apb/pwm@f8034000
> f804c000-f804c0ff : /ahb/apb/adc@f804c000
> ec00-edff : at_hdmac
> ee00-efff : at_hdmac
> f200-f3ff : atmel_serial
> f400-f5ff : /ahb/apb/pinctrl@f400/gpio@f400
> f600-f7ff : /ahb/apb/pinctrl@f400/gpio@f600
> f800-f9ff : /ahb/apb/pinctrl@f400/gpio@f800
> fa00-fbff : /ahb/apb/pinctrl@f400/gpio@fa00
> fe10-fe1f : /ahb/apb/shdwc@fe10
>
> Other Information:
> The serial device ttyS4 is configuered inside .dts file by
>
> usart3: serial@f8028000 {
> status = "okay";
> compatible = "atmel,at91sam9260-usart";
> reg = <0xf8028000 0x200>;
> interrupts = <8 IRQ_TYPE_LEVEL_HIGH 5>;
> pinctrl-names
