memory allocation issue loading a kernel module
Hi everyone, I was wondering if there is a way to flush out pages in memory that might not be required. I have a device driver that allocates 16 distict buffers each 32K in size. This is done with a bus_dma call as they will be accessed by a PCI device. The problem is that if I do a compile on my system prior to trying to kldload the module, there isn't enough physical memory for the driver. I am assuming it is the disk cache that is eating up that memory and I want to flush out enough pages for my bus_dma allocation to work. Is this possible? Any and all comments are appreciated. Thanks in advance, Sean ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory allocation issue loading a kernel module
Sean McNeil wrote: Hi everyone, I was wondering if there is a way to flush out pages in memory that might not be required. I have a device driver that allocates 16 distict buffers each 32K in size. This is done with a bus_dma call as they will be accessed by a PCI device. The problem is that if I do a compile on my system prior to trying to kldload the module, there isn't enough physical memory for the driver. I am assuming it is the disk cache that is eating up that memory and I want to flush out enough pages for my bus_dma allocation to work. Is this possible? Any and all comments are appreciated. The problem has probably nothing to do with the disk cache eating up memory but I believe what you're seeing is physical address space fragmentation. On x86, when devices want to perform DMA operations, they are given physical addresses, not virtual ones as with other architectures like sparc64 which have an IOMMU. This means that for each of your 32k buffers, you need 8 _physically contiguous_ pages of memory. Unfortunately, the more a system is running, the more the physical address space tends to be fragmented, and it becomes impossible to reserve large chunks of physically contiguous memory, hence why the kldload is failing. If I remember correctly, Alan Cox intended to write a binary buddy allocator to handle the physical address space (or do coalescing another way, I'm not sure...) so that this particular problem is solved. Cheers, Maxime ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory allocation issue loading a kernel module
Yes, thanks for the clarification. I still am inclined to believe, though, that the disk driver is what is fragmenting the physical memory with disk cacheing. It is only a theory, but it sounded plausible. Thanks again, Sean On Tue, 2003-11-25 at 00:13, Maxime Henrion wrote: Sean McNeil wrote: Hi everyone, I was wondering if there is a way to flush out pages in memory that might not be required. I have a device driver that allocates 16 distict buffers each 32K in size. This is done with a bus_dma call as they will be accessed by a PCI device. The problem is that if I do a compile on my system prior to trying to kldload the module, there isn't enough physical memory for the driver. I am assuming it is the disk cache that is eating up that memory and I want to flush out enough pages for my bus_dma allocation to work. Is this possible? Any and all comments are appreciated. The problem has probably nothing to do with the disk cache eating up memory but I believe what you're seeing is physical address space fragmentation. On x86, when devices want to perform DMA operations, they are given physical addresses, not virtual ones as with other architectures like sparc64 which have an IOMMU. This means that for each of your 32k buffers, you need 8 _physically contiguous_ pages of memory. Unfortunately, the more a system is running, the more the physical address space tends to be fragmented, and it becomes impossible to reserve large chunks of physically contiguous memory, hence why the kldload is failing. If I remember correctly, Alan Cox intended to write a binary buddy allocator to handle the physical address space (or do coalescing another way, I'm not sure...) so that this particular problem is solved. Cheers, Maxime ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory allocation issue loading a kernel module
Sean McNeil wrote: Yes, thanks for the clarification. I still am inclined to believe, though, that the disk driver is what is fragmenting the physical memory with disk cacheing. It is only a theory, but it sounded plausible. Maybe, but the root cause is not the disk caching. It may be that this subsystem is doing a lot of allocations/deallocations and thus leads to physical address space fragmentation, but the root cause is how we deal with physical address space, and the correct fix is not to add hacks into the disk caching code. I'm insisting on this because I don't want to see people adding hacks here and there to workaround the problem. Even if you get the disk caching code to cause less fragmentation, fragmentation _will_ happen. At best it'll take a bit longer. Cheers, Maxime ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory allocation issue loading a kernel module
Oh, I absolutely agree. I do not want any hacks. I was hoping that there might be an existing mechanism that was in place that would allow for the purging of unused physical pages by resource hogs. I am reminded of an old OS I was fond of: AmigaOS. It had a real nice feature where applications, drivers, etc. would register a low memory callback. Whenever the OS reached certain water-marks, these callbacks would get invoked. It was a nice clean way for shared libraries and drivers to cache things in memory, but then throw them away when things got tight. It is not a big deal for me. This is for a customer of mine and they are OK with loading the module early during boot when memory isn't fragmented yet. Just thinking out text, Sean On Tue, 2003-11-25 at 00:31, Maxime Henrion wrote: Sean McNeil wrote: Yes, thanks for the clarification. I still am inclined to believe, though, that the disk driver is what is fragmenting the physical memory with disk cacheing. It is only a theory, but it sounded plausible. Maybe, but the root cause is not the disk caching. It may be that this subsystem is doing a lot of allocations/deallocations and thus leads to physical address space fragmentation, but the root cause is how we deal with physical address space, and the correct fix is not to add hacks into the disk caching code. I'm insisting on this because I don't want to see people adding hacks here and there to workaround the problem. Even if you get the disk caching code to cause less fragmentation, fragmentation _will_ happen. At best it'll take a bit longer. Cheers, Maxime ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory allocation issue loading a kernel module
On Tuesday 25 November 2003 18:43, Maxime Henrion wrote: If I remember correctly, Alan Cox intended to write a binary buddy allocator to handle the physical address space (or do coalescing another way, I'm not sure...) so that this particular problem is solved. Another way to solve it is the bktr approach which has a KLD that just reserves some memory early on (ie you load it in the loader). This means that when you test your module the memory chunk stays around no matter how often you reload. You could get more RAM too 8-) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 9A8C 569F 685A D928 5140 AE4B 319B 41F4 5D17 FDD5 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: memory allocation issue loading a kernel module
Perfect!! This is exactly the thing I need. I will investigate. Memory is an option, but this project is pretty much done. Knowing how to do the bktr approach is something worth the excercise. More RAM won't teach me anything new ;-) Sean On Tue, 2003-11-25 at 16:09, Daniel O'Connor wrote: On Tuesday 25 November 2003 18:43, Maxime Henrion wrote: If I remember correctly, Alan Cox intended to write a binary buddy allocator to handle the physical address space (or do coalescing another way, I'm not sure...) so that this particular problem is solved. Another way to solve it is the bktr approach which has a KLD that just reserves some memory early on (ie you load it in the loader). This means that when you test your module the memory chunk stays around no matter how often you reload. You could get more RAM too 8-) ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to [EMAIL PROTECTED]
