Re: Linux 2.6.20.3

2007-03-13 Thread Bill Irwin
On Tue, Mar 13, 2007 at 05:29:17PM -0700, Nish Aravamudan wrote: Ok, truly bizarre, I found that I was not running stock 2.6.20.3, but had your small hugetlb patch on top. So I went back and patched 2.6.20.1 with your patch, rebooted, got a soft lockup. Went back to stock 2.6.20.1 and did not.

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-19 Thread Bill Irwin
On Sun, Mar 18, 2007 at 03:50:10AM +0100, Nick Piggin wrote: Yes, that should be the case. So would this mean that nonlinear protections don't work on regular files? I guess that's OK if Oracle and UML both use tmpfs/shm? Sometimes ramfs is also used in the Oracle case. I presume that's even

Re: Wanted: simple, safe x86 stack overflow detection

2007-02-28 Thread Bill Irwin
On Wed, Feb 28, 2007 at 09:41:44PM +0100, Andi Kleen wrote: Likely already too late then -- if critical state is overwritten you crashed before. Also a lot of stack intensive codes relatively large unused holes so it might miss the canary completely Anyways if you want a crash on context

Re: Wanted: simple, safe x86 stack overflow detection

2007-02-28 Thread Bill Irwin
On Feb 28 2007 15:20, Bill Irwin wrote: I don't know about the rest of the world, but halting the system in the case of memory corruption sounds like an extremely good idea to me. On Thu, Mar 01, 2007 at 12:36:47AM +0100, Jan Engelhardt wrote: Just because a rather unimportant driver (e.g

Re: [PATCH (updated)] timer: Run calc_load halfway through each round_jiffies second

2007-03-01 Thread Bill Irwin
On Thu, Mar 01, 2007 at 10:52:01PM +, Simon Arlott wrote: Whenever jiffies is started at a multiple of 5*HZ or wraps, calc_load is run exactly on the second which is when tasks using round_jiffies will be scheduled to run. This has a bad effect on the load average, making it tend

Re: Kernel Oops with shm namespace cleanups

2007-03-01 Thread Bill Irwin
On Wed, Feb 28, 2007 at 02:13:29PM -0600, Adam Litke wrote: Hey. While testing 2.6.21-rc2 with libhugetlbfs, the shm-fork test case causes the kernel to oops. To reproduce: Execute 'make check' in the latest libhugetlbfs source on a 2.6.21-rc2 kernel with 100 huge pages allocated. Using

Re: [PATCH] Fix get_unmapped_area and fsync for hugetlb shm segments

2007-03-01 Thread Bill Irwin
On Thu, Mar 01, 2007 at 03:46:08PM -0800, Adam Litke wrote: static inline int is_file_hugepages(struct file *file) { - return file-f_op == hugetlbfs_file_operations; + if (file-f_op == hugetlbfs_file_operations) + return 1; + if (is_file_shm_hugepages(file)) +

Re: The performance and behaviour of the anti-fragmentation related patches

2007-03-01 Thread Bill Irwin
On Thu, Mar 01, 2007 at 10:12:50AM +, Mel Gorman wrote: These are figures based on kernels patches with Andy Whitcrofts reclaim patches. You will see that the zone-based kernel is getting success rates closer to 40% as one would expect although there is still something amiss. Yes,

Re: The performance and behaviour of the anti-fragmentation related patches

2007-03-02 Thread Bill Irwin
On Fri, 02 Mar 2007 12:43:42 -0500 Rik van Riel [EMAIL PROTECTED] wrote: I can't share all the details, since a lot of the problems are customer workloads. One particular case is a 32GB system with a database that takes most of memory. The amount of actually freeable page cache memory is in

Re: The performance and behaviour of the anti-fragmentation related patches

2007-03-02 Thread Bill Irwin
On Fri, Mar 02, 2007 at 01:23:28PM -0500, Rik van Riel wrote: With 32 CPUs diving into the page reclaim simultaneously, each trying to scan a fraction of memory, this is disastrous for performance. A 256GB system should be even worse. Thundering herds of a sort pounding the LRU locks from

Re: The performance and behaviour of the anti-fragmentation related patches

2007-03-02 Thread Bill Irwin
At some point in the past, Mel Gorman wrote: I can't think of a workload that totally makes a mess out of list-based. However, list-based makes no guarantees on availability. If a system administrator knows they need between 10,000 and 100,000 huge pages and doesn't want to waste memory

Re: kernel BUG at arch/x86_64/mm/../../i386/mm/hugetlbpage.c:140!

2007-03-02 Thread Bill Irwin
On Fri, Mar 02, 2007 at 04:51:15PM +0300, Alexander Y. Fomichev wrote: I'm hit a bug on 2.6.21-rc1 at startup of mysql with 'large-pages' flag set. (at this point mysql trying to allocate pages from hugetlb pool by sysv shm syscalls). Seems like it could be triggered by previous badness and

Re: Kernel Oops with shm namespace cleanups

2007-03-02 Thread Bill Irwin
On Thu, 2007-03-01 at 16:08 -0800, Bill Irwin wrote: Looks like I should grab these testcases for the sake of due diligence (not to say I intend to alter maintenance style from primarily review, approval, and bugfixing, not that I've been doing as much of any of those as I should). To which

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-03 Thread Bill Irwin
On Wed, Feb 28, 2007 at 09:41:44PM +0100, Andi Kleen wrote: I suppose one could have a CONFIG_DEBUG_STACK_OVERFLOW that gets the stacks from vmalloc which would catch any overflow with its guard pages. This is you would need to change __pa() to handle that too because there might be still some

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-06 Thread Bill Irwin
Bill Irwin wrote: Chuck, is any of this of any use to you? On Tue, Mar 06, 2007 at 01:59:39PM -0500, Chuck Ebbert wrote: I said simple. :) In the 4k/4k stack i386 kernel, is there any fundamental reason it can't be 4k/8k? We seem to be mostly hitting problems in overflowing the IRQ stack

Re: kernel BUG at arch/x86_64/mm/../../i386/mm/hugetlbpage.c:140!

2007-03-06 Thread Bill Irwin
On Saturday 03 March 2007, Bill Irwin wrote: If you have a known-working kernel version, git-bisect might help you track down where it was introduced. Given the messages prior to the hugetlbpage.c BUG_ON I'd say that this is something else besides the specific code listed by line number

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-06 Thread Bill Irwin
On Tue, 6 Mar 2007, Chuck Ebbert wrote: In the 4k/4k stack i386 kernel, is there any fundamental reason it can't be 4k/8k? We seem to be mostly hitting problems in overflowing the IRQ stack... I think. Overhead would only be 4k per CPU for that. On Tue, Mar 06, 2007 at 07:43:41PM +, Hugh

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-06 Thread Bill Irwin
At some point in the past, I wrote: I'm certainly in favor of the move; IRQ stacks could be made rather deep and cheaply at that. I may get around to writing it this week if no one else does it first. On Tue, Mar 06, 2007 at 08:28:35PM -0800, Arjan van de Ven wrote: the irq stacks aren't the

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-06 Thread Bill Irwin
On Tue, Mar 06, 2007 at 10:51:01PM -0800, Andrew Morton wrote: Does anybody really pass a NULL `type' arg into filemap_nopage()? The major vs. minor fault accounting patch that introduced the argument didn't make non-NULL type arguments a requirement. It's essentially an optional second return

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, 7 Mar 2007 09:27:55 +0100 Ingo Molnar [EMAIL PROTECTED] wrote: btw., if we decide that nonlinear isnt worth the continuing maintainance pain, we could internally implement/emulate sys_remap_file_pages() via a call to mremap() and essentially deprecate it, without breaking the ABI -

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
* Nick Piggin [EMAIL PROTECTED] wrote: After these patches, I don't think there is too much burden. The main thing left really is just the objrmap stuff, but that is just handled with a minimal 'dumb' algorithm that doesn't cost much. On Wed, Mar 07, 2007 at 10:22:52AM +0100, Ingo Molnar

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, Mar 07, 2007 at 10:28:21AM +0100, Nick Piggin wrote: Depending on whether anyone wants it, and what features they want, we could emulate the old syscall, and make a new restricted one which is much less intrusive. For example, if we can operate only on MAP_ANONYMOUS memory and specify

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, Mar 07, 2007 at 10:22:52AM +0100, Ingo Molnar wrote: ok. What do you think about the sys_remap_file_pages_prot() thing that Paolo has done in a nicely split up form - does that complicate things in any fundamental way? That is what is useful to UML. * Bill Irwin [EMAIL PROTECTED

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, 7 Mar 2007 01:29:03 -0800 Bill Irwin [EMAIL PROTECTED] wrote: Guess what major real-life application not only uses nonlinear daily but would even be very happy to see it extended with non-vma-creating protections and more? On Wed, Mar 07, 2007 at 01:39:42AM -0800, Andrew Morton wrote

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, Mar 07, 2007 at 11:47:42AM +0100, Peter Zijlstra wrote: Well, now they don't, but it could be done or even exploited as a DoS. On Wed, Mar 07, 2007 at 12:00:36PM +0100, Nick Piggin wrote: But so could nonlinear page reclaim. I think we need to restrict nonlinear mappings to root if

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-07 Thread Bill Irwin
On Tue, 2007-03-06 at 22:44 -0800, Bill Irwin wrote: What do you see as the obstacle to eliminating nested IRQ's? On Wed, Mar 07, 2007 at 04:34:52AM -0800, Arjan van de Ven wrote: political will, or maybe just the lack of convincing people so far Political issues are significantly more

Re: [PATCH] Fix get_unmapped_area and fsync for hugetlb shm segments

2007-03-07 Thread Bill Irwin
Bill Irwin [EMAIL PROTECTED] writes: A comment to prepare others for the impending doubletake might be nice. Or maybe just open-coding the equality check for hugetlbfs_file_operations in is_file_shm_hugepages() if others find it as jarring as I. Please extend my ack to any follow-up fiddling

Re: [PATCH] Fix get_unmapped_area and fsync for hugetlb shm segments

2007-03-07 Thread Bill Irwin
On Wed, Mar 07, 2007 at 05:26:48PM -0600, Adam Litke wrote: :) Enter my remove-is_file_hugepages() patches (which I posted a few weeks ago). I'll rework them and repost soon. That should help to make all of this cleaner. Those were great. I've wanted something like them for a long, long

Re: [PATCH] hugetlbfs read() support

2007-07-19 Thread Bill Irwin
On Thu, Jul 19, 2007 at 10:07:59AM -0700, Nishanth Aravamudan wrote: But I do think a second reason to do this is to make hugetlbfs behave like a normal fs -- that is read(), write(), etc. work on files in the mountpoint. But that is simply my opinion. Mine as well. -- wli - To unsubscribe

Re: [2/6] add config option to vmalloc stacks (was: Re: [-mm patch] i386: enable 4k stacks by default)

2007-04-30 Thread Bill Irwin
On Mon, Apr 30, 2007 at 08:15:11PM +0100, Christoph Hellwig wrote: So if you want to invest some time into getting this into mergeable shape I'd suggest you redo the patch series in the following way: patch 1: dynamic allocated irq stacks patch 2: make irqstacks unconditional, but allow

Re: [4/6] go BUG on vmallocspace in __pa() (was: Re: [-mm patch] i386: enable 4k stacks by default)

2007-04-30 Thread Bill Irwin
On Mon, Apr 30, 2007 at 10:38:19AM -0700, William Lee Irwin III wrote: Here's what I did for i386 for someone concerned about blowing the stack. On Mon, Apr 30, 2007 at 10:45:10AM -0700, William Lee Irwin III wrote: Add checks to __pa() so it goes BUG() on vmallocspace addresses. On Mon, 30

Re: [5/6] dynamically allocate IRQ stacks (was: Re: [-mm patch] i386: enable 4k stacks by default)

2007-04-30 Thread Bill Irwin
On Mon, 30 Apr 2007, William Lee Irwin III wrote: -static char softirq_stack[NR_CPUS * THREAD_SIZE] -__attribute__((__aligned__(THREAD_SIZE))); +static DEFINE_PER_CPU(char *, softirq_stack); +static DEFINE_PER_CPU(char *, hardirq_stack); -static char hardirq_stack[NR_CPUS *

Re: [2/6] add config option to vmalloc stacks (was: Re: [-mm patch] i386: enable 4k stacks by default)

2007-04-30 Thread Bill Irwin
On Mon, Apr 30, 2007 at 08:15:11PM +0100, Christoph Hellwig wrote: So if you want to invest some time into getting this into mergeable shape I'd suggest you redo the patch series in the following way: patch 1: dynamic allocated irq stacks patch 2: make irqstacks unconditional, but allow

[0/3] i386 stack handling updates

2007-04-30 Thread Bill Irwin
This is a series of patches to improve i386 stack handling in several manners. The net result is to improve system stability via IRQ stacks while remaining memory efficient and to improve stack debugging. [1/3] dynamically allocate irq stacks This conserves memory while using IRQ stacks.

[1/3] dynamically allocate IRQ stacks

2007-04-30 Thread Bill Irwin
Dynamically allocate IRQ stacks in order to conserve memory when using IRQ stacks. cpu_possible_map is not now initialized in such a manner as to provide a meaningful indication of how many CPU's might be in the system, and features to appear in the sequel also require indirection, so they

[2/3] unconditional i386 IRQ stacks

2007-04-30 Thread Bill Irwin
IRQ stacks are a valuable stability feature. This patch makes them unconditional, as there is no circumstance under which they do not improve stability and they have no meaningful performance impact. Signed-off-by: William Irwin [EMAIL PROTECTED] Index: stack-paranoia/include/asm-i386/irq.h

[3/3] use vmalloc() to arrange guard pages for stacks

2007-04-30 Thread Bill Irwin
This patch introduces CONFIG_DEBUG_STACK, which vmalloc()'s task and IRQ stacks in order to establish guard pages. In such a manner any stack overflow that references pages immediately adjacent to the stack is immediately trapped with a fault, which precludes silent memory corruption or

Re: 2.6.22 -mm merge plans

2007-04-30 Thread Bill Irwin
On Mon, Apr 30, 2007 at 04:20:07PM -0700, Andrew Morton wrote: proper-prototype-for-hugetlb_get_unmapped_area.patch ... convert-hugetlbfs-to-use-vm_ops-fault.patch ... get_unmapped_area-handles-map_fixed-in-hugetlbfs.patch ... get_unmapped_area-doesnt-need-hugetlbfs-hacks-anymore.patch ...

Re: [PATCH] [30/30] x86_64: Add missing !X86_PAE dependincy to the 2G/2G split.

2007-05-01 Thread Bill Irwin
On Tue, May 01, 2007 at 06:26:23AM +0200, Eric Dumazet wrote: Hum... We lose a usefull 2G/2G split. Should'nt we use a patch to change PAGE_OFFSET to 0x800 instead of 0x7800 and keep 2G/2G split ? On Tue, May 01, 2007 at 08:21:32AM +0200, Andi Kleen wrote: I dropped the patch for now.

Re: [3/3] use vmalloc() to arrange guard pages for stacks

2007-05-01 Thread Bill Irwin
Bill Irwin wrote: This patch introduces CONFIG_DEBUG_STACK, which vmalloc()'s task and IRQ stacks in order to establish guard pages. In such a manner any stack overflow that references pages immediately adjacent to the stack is immediately trapped with a fault, which precludes silent memory

Re: [3/3] use vmalloc() to arrange guard pages for stacks

2007-05-01 Thread Bill Irwin
Bill Irwin wrote: It's a shame that the resource scalability implications of vmallocspace allocations prevent this from being useful in production. One could, in principle, establish guard pages within ZONE_NORMAL, but for 4KB stacks it's somewhat awkward to dredge up 3 contigous pages, and 4

Re: [PATCH] [30/30] x86_64: Add missing !X86_PAE dependincy to the 2G/2G split.

2007-05-01 Thread Bill Irwin
Bill Irwin a écrit : as a stopgap measure, but I'm not all that interested in grabbing patch credits where others could do it easily enough. Either of the config alterations is fine by me as they now stand; maybe Eric Dumazet might care to do something like my suggestion at some point. On Tue

Re: per-thread rusage

2007-05-01 Thread Bill Irwin
On Mon, Apr 09, 2007 at 04:53:15PM -0700, Andrew Morton wrote: Seems sane. Could we please get it tested and get a full description in place? Something which provides enough detail for the manpage maintainers. Also, a quick comparison between Linux's RUSAGE_THREAD and $other-os's

Re: [1/3] dynamically allocate IRQ stacks

2007-05-01 Thread Bill Irwin
At some point in the past, I wrote: +static void * __init __alloc_irqstack(int cpu) +{ +if (!cpu) +return __alloc_bootmem(THREAD_SIZE, THREAD_SIZE, +__pa(MAX_DMA_ADDRESS)); + +return (void *)__get_free_pages(GFP_KERNEL, +

Re: per-thread rusage

2007-05-01 Thread Bill Irwin
On 5/1/07, Bill Irwin [EMAIL PROTECTED] wrote: A sort of note for me to refer back to when I get the rest of the way here. AIX does this with getrusage(RUSAGE_THREAD,...), Solaris with getrusage(RUSAGE_LWP,...), On Tue, May 01, 2007 at 11:39:46AM -0700, Ulrich Drepper wrote: RUSAGE_LWP

Re: per-thread rusage

2007-05-01 Thread Bill Irwin
On 5/1/07, Bill Irwin [EMAIL PROTECTED] wrote: The basic idea is to try to do it similarly to how everyone else does so userspace (I suppose this would include glibc) don't have to bend over backward to accommodate it. Or basically to do what everyone expects. On Tue, May 01, 2007 at 03:10

Re: per-thread rusage

2007-05-01 Thread Bill Irwin
On Tue, May 01, 2007 at 06:27:24PM -0400, Theodore Tso wrote: There are two ways of implementing this. One is to have the JVM periodically poll using a pthread_getrusage() interface. A better choice might be some kind of per-thread CPU limit, that would result in a thread-specific

Re: [2/6] add config option to vmalloc stacks (was: Re: [-mm patch] i386: enable 4k stacks by default)

2007-05-01 Thread Bill Irwin
On Mon, Apr 30, 2007 at 08:15:11PM +0100, Christoph Hellwig wrote: So if you want to invest some time into getting this into mergeable shape I'd suggest you redo the patch series in the following way: patch 1: dynamic allocated irq stacks On Tue, May 01, 2007 at 05:36:06PM -0500, Matt Mackall

Re: per-thread rusage

2007-05-01 Thread Bill Irwin
At some point in the past, I wrote: I just so happen to think we should implement a variety of CPU resource limits beyond what we now do, so this, too, interests me. On Wed, May 02, 2007 at 12:04:58AM +0100, Alan Cox wrote: Agreed - and make them all 64bit while doing the cleanup. One thing

Re: [2/6] add config option to vmalloc stacks (was: Re: [-mm patch] i386: enable 4k stacks by default)

2007-05-01 Thread Bill Irwin
At some point in the past, I wrote: These IRQ stacks are per-cpu, not per-IRQ. It may make sense to implement per-IRQ stacks, in which case dynamic allocation at the time of request_irq() will make sense. On Wed, May 02, 2007 at 12:07:45AM +0100, Alan Cox wrote: This depends if active IRQ

Re: [2/6] add config option to vmalloc stacks (was: Re: [-mm patch] i386: enable 4k stacks by default)

2007-05-01 Thread Bill Irwin
On Tue, May 01, 2007 at 05:36:06PM -0500, Matt Mackall wrote: Can we register them lazily at request_irq time? On Tue, May 01, 2007 at 03:51:25PM -0700, Bill Irwin wrote: These IRQ stacks are per-cpu, not per-IRQ. It may make sense to implement per-IRQ stacks, in which case dynamic allocation

Re: [RFC BUG?] dereference PAGE_OFFSET address (rc7-mm2)

2007-05-02 Thread Bill Irwin
On Wed, May 02, 2007 at 12:51:40AM -0700, Andrew Morton wrote: Thanks for the report. I can reproduce it. Bisection shows that x86_64-mm-paravirt-initial-pagetable.patch caused this. I didn't check whether the patch actually permits us to read kernel memory. Probably it does. Probably we'd

Re: [RFC BUG?] dereference PAGE_OFFSET address (rc7-mm2)

2007-05-02 Thread Bill Irwin
prefer that it didn't ;) On Wed, May 02, 2007 at 01:46:17AM -0700, Bill Irwin wrote: Brown paper bag time. I don't know how it got past me. Brain dump before crashing for the night: The patch refuses to clobber already-present pagetable entries of whatever origin. There are pagetables prior

Re: [RFC BUG?] dereference PAGE_OFFSET address (rc7-mm2)

2007-05-02 Thread Bill Irwin
Bill Irwin wrote: Brain dump before crashing for the night: The patch refuses to clobber already-present pagetable entries of whatever origin. There are pagetables prior to this setup covering the address range just above PAGE_OFFSET. If this theory is correct, you should only be able to go

Re: [RFC BUG?] dereference PAGE_OFFSET address (rc7-mm2)

2007-05-02 Thread Bill Irwin
On Wed, May 02, 2007 at 09:28:46AM -0700, Jeremy Fitzhardinge wrote: I think this should be fixed now. Eric made all those writes unconditional (to fix a problem with PSE superpages not being created). The patch is in Andi's queue. Bill Irwin [EMAIL PROTECTED] writes: It needs verification

Re: [4/6] go BUG on vmallocspace in __pa()

2007-05-02 Thread Bill Irwin
William Lee Irwin III wrote: +unsigned long __kvaddr_to_paddr(unsigned long kvaddr) +{ +if (high_memory) +BUG_ON(kvaddr = VMALLOC_START); +else +BUG_ON(kvaddr = (unsigned long)__va(MAXMEM)); +return kvaddr - PAGE_OFFSET; +} On Wed, May 02, 2007 at

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-02 Thread Bill Irwin
On Wed, May 02, 2007 at 06:56:09PM -0700, Jeremy Fitzhardinge wrote: +static void __cpuinit __free_irqstack(int cpu, void *stk) +{ + int i; + + if (!cpu) + return; + + unmap_vm_area(per_cpu(irqstack_area, cpu)); + + for (i = 0; i THREAD_SIZE/PAGE_SIZE; ++i)

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-02 Thread Bill Irwin
) +__free_page(per_cpu(irqstack_pages, cpu)[i]); +} On Wed, May 02, 2007 at 07:25:34PM -0700, Bill Irwin wrote: [...] Not sure if cpu 0 can ever be offlined, but it is remapped and so on. So its virtual mapping should be undone if it can be offlined, but we probably shouldn't attempt to free

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-02 Thread Bill Irwin
On Wed, May 02, 2007 at 06:56:09PM -0700, Jeremy Fitzhardinge wrote: This fixes two bugs: - the stack allocation must be marked __cpuinit, since it gets called on resume as well. - presumably the interrupt stack should be freed on unplug if its going to get reallocated on every plug.

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-03 Thread Bill Irwin
Bill Irwin wrote: Updated patch follows. Please add your Signed-off-by: if it meets your approval; On Wed, May 02, 2007 at 11:01:05PM -0700, Jeremy Fitzhardinge wrote: What does it apply to? I'm getting conflicts if I replace my patch with this. Or does it replace one of your patches? I

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-03 Thread Bill Irwin
On Wed, May 02, 2007 at 10:48:09PM -0700, Bill Irwin wrote: Updated patch follows. Please add your Signed-off-by: if it meets your approval; I am operating on the assumption I should never do so myself. I'm a bit unsure of how to handle cpu 0 vs. potential freeing of per_cpu areas and error

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-03 Thread Bill Irwin
Bill Irwin wrote: I had the same question about yours and just brute-force merged. Not a big deal for me to rediff against whatever everyone's working off of. On Thu, May 03, 2007 at 12:07:29AM -0700, Jeremy Fitzhardinge wrote: I picked up one version of your patches you posted a couple

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-03 Thread Bill Irwin
On Thu, May 03, 2007 at 12:39:30AM -0700, Bill Irwin wrote: As an aside, it looks like failures here need to eventually propagate to __cpu_up(). irq_ctx_init() needs to return a status, and its callers need to check it. irq_ctx_init() probably also needs to be __cpuinit. Ignoring the general

Re: + per-cpuset-hugetlb-accounting-and-administration.patch added to -mm tree

2007-05-04 Thread Bill Irwin
On Thu, May 03, 2007 at 06:38:21PM -0700, Paul Jackson wrote: Adding Christoph Lameter [EMAIL PROTECTED] to the cc list, as he knows more about hugetlb pages than I do. This patch strikes me as a bit odd. Granted, it's solving what could be a touchy problem with a fairly simple solution,

Re: [2/6] add config option to vmalloc stacks (was: Re: [-mm patch] i386: enable 4k stacks by default)

2007-05-04 Thread Bill Irwin
On Mon, Apr 30, 2007 at 10:43:10AM -0700, William Lee Irwin III wrote: + Allocates the stack physically discontiguously and from high + memory. Furthermore an unmapped guard page follows the stack. + This is not for end-users. It's intended to trigger fatal + system errors

Re: Linux 2.6.20.3

2007-03-13 Thread Bill Irwin
On Tue, Mar 13, 2007 at 05:29:17PM -0700, Nish Aravamudan wrote: > Ok, truly bizarre, I found that I was not running stock 2.6.20.3, but > had your small hugetlb patch on top. > So I went back and patched 2.6.20.1 with your patch, rebooted, got a > soft lockup. Went back to stock 2.6.20.1 and did

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-19 Thread Bill Irwin
On Sun, Mar 18, 2007 at 03:50:10AM +0100, Nick Piggin wrote: > Yes, that should be the case. So would this mean that nonlinear protections > don't work on regular files? I guess that's OK if Oracle and UML both use > tmpfs/shm? Sometimes ramfs is also used in the Oracle case. I presume that's

Re: Wanted: simple, safe x86 stack overflow detection

2007-02-28 Thread Bill Irwin
On Wed, Feb 28, 2007 at 09:41:44PM +0100, Andi Kleen wrote: > Likely already too late then -- if critical state is overwritten > you crashed before. Also a lot of stack intensive codes > relatively large unused holes so it might miss the canary completely > Anyways if you want a crash on context

Re: Wanted: simple, safe x86 stack overflow detection

2007-02-28 Thread Bill Irwin
On Feb 28 2007 15:20, Bill Irwin wrote: >> I don't know about the rest of the world, but halting the system in the >> case of memory corruption sounds like an extremely good idea to me. On Thu, Mar 01, 2007 at 12:36:47AM +0100, Jan Engelhardt wrote: > Just because a rather "

Re: [PATCH (updated)] timer: Run calc_load halfway through each round_jiffies second

2007-03-01 Thread Bill Irwin
On Thu, Mar 01, 2007 at 10:52:01PM +, Simon Arlott wrote: > Whenever jiffies is started at a multiple of 5*HZ or wraps, calc_load is > run exactly on the second which is when tasks using round_jiffies will > be scheduled to run. This has a bad effect on the load average, making > it tend

Re: Kernel Oops with shm namespace cleanups

2007-03-01 Thread Bill Irwin
On Wed, Feb 28, 2007 at 02:13:29PM -0600, Adam Litke wrote: > Hey. While testing 2.6.21-rc2 with libhugetlbfs, the shm-fork test case > causes the kernel to oops. To reproduce: Execute 'make check' in the > latest libhugetlbfs source on a 2.6.21-rc2 kernel with 100 huge pages > allocated.

Re: [PATCH] Fix get_unmapped_area and fsync for hugetlb shm segments

2007-03-01 Thread Bill Irwin
On Thu, Mar 01, 2007 at 03:46:08PM -0800, Adam Litke wrote: > static inline int is_file_hugepages(struct file *file) > { > - return file->f_op == _file_operations; > + if (file->f_op == _file_operations) > + return 1; > + if (is_file_shm_hugepages(file)) > +

Re: The performance and behaviour of the anti-fragmentation related patches

2007-03-01 Thread Bill Irwin
On Thu, Mar 01, 2007 at 10:12:50AM +, Mel Gorman wrote: > These are figures based on kernels patches with Andy Whitcrofts reclaim > patches. You will see that the zone-based kernel is getting success rates > closer to 40% as one would expect although there is still something amiss. Yes,

Re: The performance and behaviour of the anti-fragmentation related patches

2007-03-02 Thread Bill Irwin
On Fri, 02 Mar 2007 12:43:42 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote: >> I can't share all the details, since a lot of the problems are customer >> workloads. >> One particular case is a 32GB system with a database that takes most >> of memory. The amount of actually freeable page cache

Re: The performance and behaviour of the anti-fragmentation related patches

2007-03-02 Thread Bill Irwin
On Fri, Mar 02, 2007 at 01:23:28PM -0500, Rik van Riel wrote: > With 32 CPUs diving into the page reclaim simultaneously, > each trying to scan a fraction of memory, this is disastrous > for performance. A 256GB system should be even worse. Thundering herds of a sort pounding the LRU locks from

Re: The performance and behaviour of the anti-fragmentation related patches

2007-03-02 Thread Bill Irwin
At some point in the past, Mel Gorman wrote: >> I can't think of a workload that totally makes a mess out of list-based. >> However, list-based makes no guarantees on availability. If a system >> administrator knows they need between 10,000 and 100,000 huge pages and >> doesn't want to waste

Re: kernel BUG at arch/x86_64/mm/../../i386/mm/hugetlbpage.c:140!

2007-03-02 Thread Bill Irwin
On Fri, Mar 02, 2007 at 04:51:15PM +0300, Alexander Y. Fomichev wrote: > I'm hit a bug on 2.6.21-rc1 at startup of mysql with 'large-pages' flag set. > (at this point mysql trying to allocate pages from hugetlb pool by sysv > shm syscalls). Seems like it could be triggered by previous badness >

Re: Kernel Oops with shm namespace cleanups

2007-03-02 Thread Bill Irwin
On Thu, 2007-03-01 at 16:08 -0800, Bill Irwin wrote: >> Looks like I should grab these testcases for the sake of due diligence >> (not to say I intend to alter maintenance style from primarily review, >> approval, and bugfixing, not that I've been doing as much of any of th

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-03 Thread Bill Irwin
On Wed, Feb 28, 2007 at 09:41:44PM +0100, Andi Kleen wrote: > I suppose one could have a CONFIG_DEBUG_STACK_OVERFLOW that gets > the stacks from vmalloc which would catch any overflow with its > guard pages. This is you would need to change __pa() to handle > that too because there might be still

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-06 Thread Bill Irwin
Bill Irwin wrote: >> Chuck, is any of this of any use to you? On Tue, Mar 06, 2007 at 01:59:39PM -0500, Chuck Ebbert wrote: > I said "simple." :) > In the 4k/4k stack i386 kernel, is there any fundamental reason it > can't be 4k/8k? We seem to be mostly hitting problems

Re: kernel BUG at arch/x86_64/mm/../../i386/mm/hugetlbpage.c:140!

2007-03-06 Thread Bill Irwin
On Saturday 03 March 2007, Bill Irwin wrote: >> If you have a known-working kernel version, git-bisect might help you >> track down where it was introduced. Given the messages prior to the >> hugetlbpage.c BUG_ON I'd say that this is something else besides the >> spec

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-06 Thread Bill Irwin
On Tue, 6 Mar 2007, Chuck Ebbert wrote: >> In the 4k/4k stack i386 kernel, is there any fundamental reason it >> can't be 4k/8k? We seem to be mostly hitting problems in overflowing >> the IRQ stack... I think. Overhead would only be 4k per CPU for that. On Tue, Mar 06, 2007 at 07:43:41PM +,

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-06 Thread Bill Irwin
At some point in the past, I wrote: >> I'm certainly in favor of the move; IRQ stacks could be made >> rather deep and cheaply at that. I may get around to writing it this >> week if no one else does it first. On Tue, Mar 06, 2007 at 08:28:35PM -0800, Arjan van de Ven wrote: > the irq stacks

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-06 Thread Bill Irwin
On Tue, Mar 06, 2007 at 10:51:01PM -0800, Andrew Morton wrote: > Does anybody really pass a NULL `type' arg into filemap_nopage()? The major vs. minor fault accounting patch that introduced the argument didn't make non-NULL type arguments a requirement. It's essentially an optional second return

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, 7 Mar 2007 09:27:55 +0100 Ingo Molnar <[EMAIL PROTECTED]> wrote: >> btw., if we decide that nonlinear isnt worth the continuing maintainance >> pain, we could internally implement/emulate sys_remap_file_pages() via a >> call to mremap() and essentially deprecate it, without breaking the

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
* Nick Piggin <[EMAIL PROTECTED]> wrote: >> After these patches, I don't think there is too much burden. The main >> thing left really is just the objrmap stuff, but that is just handled >> with a minimal 'dumb' algorithm that doesn't cost much. On Wed, Mar 07, 2007 at 10:22:52AM +0100, Ingo

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, Mar 07, 2007 at 10:28:21AM +0100, Nick Piggin wrote: > Depending on whether anyone wants it, and what features they want, we > could emulate the old syscall, and make a new restricted one which is > much less intrusive. > For example, if we can operate only on MAP_ANONYMOUS memory and

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, Mar 07, 2007 at 10:22:52AM +0100, Ingo Molnar wrote: >>> ok. What do you think about the sys_remap_file_pages_prot() thing that >>> Paolo has done in a nicely split up form - does that complicate things >>> in any fundamental way? That is what is useful to

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, 7 Mar 2007 01:29:03 -0800 Bill Irwin <[EMAIL PROTECTED]> wrote: >> Guess what major real-life application not only uses nonlinear daily >> but would even be very happy to see it extended with non-vma-creating >> protections and more? On Wed, Mar 07, 2007 at 01:39:4

Re: [patch 4/6] mm: merge populate and nopage into fault (fixes nonlinear)

2007-03-07 Thread Bill Irwin
On Wed, Mar 07, 2007 at 11:47:42AM +0100, Peter Zijlstra wrote: >> Well, now they don't, but it could be done or even exploited as a DoS. On Wed, Mar 07, 2007 at 12:00:36PM +0100, Nick Piggin wrote: > But so could nonlinear page reclaim. I think we need to restrict nonlinear > mappings to root if

Re: Wanted: simple, safe x86 stack overflow detection

2007-03-07 Thread Bill Irwin
On Tue, 2007-03-06 at 22:44 -0800, Bill Irwin wrote: >> What do you see as the obstacle to eliminating nested IRQ's? On Wed, Mar 07, 2007 at 04:34:52AM -0800, Arjan van de Ven wrote: > political will, or maybe just the lack of convincing people so far Political issues are significa

Re: [PATCH] Fix get_unmapped_area and fsync for hugetlb shm segments

2007-03-07 Thread Bill Irwin
Bill Irwin <[EMAIL PROTECTED]> writes: >> A comment to prepare others for the impending doubletake might be nice. >> Or maybe just open-coding the equality check for _file_operations >> in is_file_shm_hugepages() if others find it as jarring as I. Please >> extend my

Re: [PATCH] Fix get_unmapped_area and fsync for hugetlb shm segments

2007-03-07 Thread Bill Irwin
On Wed, Mar 07, 2007 at 05:26:48PM -0600, Adam Litke wrote: > :) Enter my remove-is_file_hugepages() patches (which I posted a few > weeks ago). I'll rework them and repost soon. That should help to make > all of this cleaner. Those were great. I've wanted something like them for a long, long

Re: [4/6] go BUG on vmallocspace in __pa()

2007-05-02 Thread Bill Irwin
William Lee Irwin III wrote: >> +unsigned long __kvaddr_to_paddr(unsigned long kvaddr) >> +{ >> +if (high_memory) >> +BUG_ON(kvaddr >= VMALLOC_START); >> +else >> +BUG_ON(kvaddr >= (unsigned long)__va(MAXMEM)); >> +return kvaddr - PAGE_OFFSET; >> +} On Wed, May

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-02 Thread Bill Irwin
On Wed, May 02, 2007 at 06:56:09PM -0700, Jeremy Fitzhardinge wrote: > +static void __cpuinit __free_irqstack(int cpu, void *stk) > +{ > + int i; > + > + if (!cpu) > + return; > + > + unmap_vm_area(per_cpu(irqstack_area, cpu)); > + > + for (i = 0; i <

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-02 Thread Bill Irwin
cpu)); >> + >> +for (i = 0; i < THREAD_SIZE/PAGE_SIZE; ++i) >> +__free_page(per_cpu(irqstack_pages, cpu)[i]); >> +} On Wed, May 02, 2007 at 07:25:34PM -0700, Bill Irwin wrote: [...] Not sure if cpu 0 can ever be offlined, but it is remapped and so on. So

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-02 Thread Bill Irwin
On Wed, May 02, 2007 at 06:56:09PM -0700, Jeremy Fitzhardinge wrote: > This fixes two bugs: > - the stack allocation must be marked __cpuinit, since it gets called >on resume as well. > - presumably the interrupt stack should be freed on unplug if its >going to get reallocated on every

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-03 Thread Bill Irwin
Bill Irwin wrote: >> Updated patch follows. Please add your Signed-off-by: if it meets your >> approval; On Wed, May 02, 2007 at 11:01:05PM -0700, Jeremy Fitzhardinge wrote: > What does it apply to? I'm getting conflicts if I replace my patch with > this. Or does it replace o

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-03 Thread Bill Irwin
On Wed, May 02, 2007 at 10:48:09PM -0700, Bill Irwin wrote: > Updated patch follows. Please add your Signed-off-by: if it meets your > approval; I am operating on the assumption I should never do so myself. > I'm a bit unsure of how to handle cpu 0 vs. potential freeing of per_cp

Re: [PATCH] i386: fix suspend/resume with dynamically allocated irq stacks

2007-05-03 Thread Bill Irwin
Bill Irwin wrote: >> I had the same question about yours and just brute-force merged. Not a >> big deal for me to rediff against whatever everyone's working off of. On Thu, May 03, 2007 at 12:07:29AM -0700, Jeremy Fitzhardinge wrote: > I picked up one version of your patches you

  1   2   >