Re: AMD AGP Bug
On Fri, 1 Feb 2002, Peter Wemm wrote: - AMD write cache allocation due to speculative writes being cancelled and then written back later vs no cache snooping on AGP regions. I'm somewhat perplexed about this issue, there's lots of conflicting info going around, a good deal of it which does not make much sense [to me :-)]. I really dont see what PSE has to do with this for several reasons.. if the page/ region is cacheable, why does a 4MB vs 4K page make any difference? cacheable vs no-cache-snooping is a recipe for disaster.. why would 4K pages on a non-coherent region be safer? Or is the problem that write allocation happens on uncacheable/non-write-back regions in 4MB pages? Or something else? Speculative writes can only happen to pages in the TLB (so you don't get speculative TLB misses and replacements), not having a large amount of 4M pages around in the TLB means that addresses covered by these can't possibly be involved in speculative writes. I personaly suspect the reason the cache line flushes of speculatively written to cache lines derive from the AMD-s use of MOESI coherency and mapping that to actual bits. Another minor side effect is that you get direct cache-to-cahce transfers in SMP systems for shared data. Sander Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] All of this is for nothing if we don't go to the stars - JMS/B5 To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Narvi wrote: Speculative writes can only happen to pages in the TLB (so you don't get speculative TLB misses and replacements), not having a large amount of 4M pages around in the TLB means that addresses covered by these can't possibly be involved in speculative writes. I personaly suspect the reason the cache line flushes of speculatively written to cache lines derive from the AMD-s use of MOESI coherency and mapping that to actual bits. Another minor side effect is that you get direct cache-to-cahce transfers in SMP systems for shared data. I think that the problem is more related to the fact that there are 16 TLB entries for 4K data pages, 16 for 4K code pages, and another 8 for 4M pages. Peter is right, in other words, because there is a problem with the interaction of 4K and 4M pages. I've seen this myself, as I've previously reported, as well as seeing other problems (and knowing how to work around them, after weeks of investigation into characterizing them). Note that I was lucky, in that I had modified the FreeBSD kernel to use certain types of mappings in a certain way; I think it would be very difficult, or impossible, for anyone else to duplicate the problem in order to better characterize it beyond DISABLE_PSE, except the chip vendors themselves, if they started from first principles with a simulation. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
On Mon, 4 Feb 2002, Terry Lambert wrote: Narvi wrote: Speculative writes can only happen to pages in the TLB (so you don't get speculative TLB misses and replacements), not having a large amount of 4M pages around in the TLB means that addresses covered by these can't possibly be involved in speculative writes. I personaly suspect the reason the cache line flushes of speculatively written to cache lines derive from the AMD-s use of MOESI coherency and mapping that to actual bits. Another minor side effect is that you get direct cache-to-cahce transfers in SMP systems for shared data. I think that the problem is more related to the fact that there are 16 TLB entries for 4K data pages, 16 for 4K code pages, and another 8 for 4M pages. Peter is right, in other words, because there is a problem with the interaction of 4K and 4M pages. I've seen this myself, as I've previously reported, as well as seeing other problems (and knowing how to work around them, after weeks of investigation into characterizing them). I wasn't aware that I was contradicting Peter 8-) Yes, it is possible that there are issues (some of which may in turn be bugs) in the 4KB and 4MB page handling. I'm not sure that 4K vs. 4M handling is done in any special way when it comes to speculative vs. normal writes. Speculative writes don't cross page boundaries (which means a different thing depending on which TLB you found the page from) - meaning that having a change of objects inside a mapped 4M page is asking for trouble. Writing back the cache lines that may have been written to is imho reasonable behaviour. Note that I was lucky, in that I had modified the FreeBSD kernel to use certain types of mappings in a certain way; I think it would be very difficult, or impossible, for anyone else to duplicate the problem in order to better characterize it beyond DISABLE_PSE, except the chip vendors themselves, if they started from first principles with a simulation. It may even well be possible to get different results with aligned vs. misaligned reads and writes, or a proper mix thereof. It may be possible to build a model to track down the what is really going on part, but its not clear its worth the trouble over just picking what (not) to do. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Narvi wrote: I wasn't aware that I was contradicting Peter 8-) Sorry; looked like it to me... 8-). It may even well be possible to get different results with aligned vs. misaligned reads and writes, or a proper mix thereof. It may be possible to build a model to track down the what is really going on part, but its not clear its worth the trouble over just picking what (not) to do. Unless you were a chip vendor telling people to disable PSE in all rour systems that ran Windows or Linux, and the resulting systems were losing 4-14% of their total performance as a result? ;^). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Glendon Gross wrote: That's right, guys! This is FreeBSD after all... so Mr. Lambert is entitled to charge 10K for that bugfix code if he wants. In fact he is Free to do so. But it's a little pricy for me, although perhaps not for AMD if it means they can fix their cache-paging problems! It cost me two weeks to figure out, and both Intel and AMD wanted to charge me $$$ to file a problem report. I once fixed a Borland library problem which was pretty serious: it would effect all programs that did string manipulation, and linked the standard library, instead of using the compiler generated inlines. They wanted to charge me to file a bug report, too, so I demanded that they agree to send me a T-shirt before I would tell them the fix. As far as I know, they still have the bug. Microsoft also charged me to report a bug in the removable media driver in Windows 95/98 (you can't safely page to it because it fails to propagate some writes it's supposed to propagate, so you can'tt safely install Windows on a JAZ or Syquest cartridge, because paging fails). I made them admit the problem, and they credited the incident report after they admitted it was their issue, but it still started my 90 day support clock, so that it wasn't there when I needed it, later. Since it's not a FreeBSD problem, you'll forgive me if I find it funny that Linux and Windows disables the PSE to work around some problems. It's not that hard to understand or work around, if you know where to look, and knowing where to look is what will cost them. The *other* AMD problem, with AGP and 4K and 4M pages that point to the same memory could arguably be a chipset or a software problem (depending on whether or not the chipset vendor has documented the behaviour before people were let loose to write the software -- errata: the difference between a software problem and a software workaround to a hardware problem). It's already documented how to work around that one; I'm sure a fix is pending for Linux, and Windows will probably leave PSE off until they can figure out a way to update everyone. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Terry Lambert wrote: Glendon Gross wrote: That's right, guys! This is FreeBSD after all... so Mr. Lambert is entitled to charge 10K for that bugfix code if he wants. In fact he is Free to do so. But it's a little pricy for me, although perhaps not for AMD if it means they can fix their cache-paging problems! It cost me two weeks to figure out, and both Intel and AMD wanted to charge me $$$ to file a problem report. [..] Since it's not a FreeBSD problem, you'll forgive me if I find it funny that Linux and Windows disables the PSE to work around some problems. It's not that hard to understand or work around, if you know where to look, and knowing where to look is what will cost them. [..] The *other* AMD problem, with AGP and 4K and 4M pages that point to the same memory could arguably be a chipset or a software problem (depending on whether or not the chipset vendor has documented the behaviour before people were let loose to write the software -- errata: the difference between a software problem and a software workaround to a hardware problem). It's already documented how to work around that one; I'm sure a fix is pending for Linux, and Windows will probably leave PSE off until they can figure out a way to update everyone. I'm a little confused as to which bugs are which that we're talking about now. Which is the one that you're trying to sell the info for? The issues that I know about: - interactions between PG_G, CR4_PGE, 4M and 4K pages and TLB flushing (potentially cross platform issue due to boot time quirks).. I think we have a workaround for this in our code already, but I have a better complete fix for it in a pending change that I'm working on. - AMD invlpg vs 4MB page bug (AMD bug). If you invlpg using an address in the upper 2MB of a 4MB page, the 4MB tlb may not be flushed (we dont suffer from this (I think) because we do some very suboptimal things with 4MB pages), versus doing an invlpg at the base address of the 4MB mapping. - AMD write cache allocation due to speculative writes being cancelled and then written back later vs no cache snooping on AGP regions. I'm somewhat perplexed about this issue, there's lots of conflicting info going around, a good deal of it which does not make much sense [to me :-)]. I really dont see what PSE has to do with this for several reasons.. if the page/ region is cacheable, why does a 4MB vs 4K page make any difference? cacheable vs no-cache-snooping is a recipe for disaster.. why would 4K pages on a non-coherent region be safer? Or is the problem that write allocation happens on uncacheable/non-write-back regions in 4MB pages? Or something else? - hardware prefetch (newer AMD cpus, pentium 4, 0.13 micron pentium3's) related problems. (can be solved with PAT and/or MSRR programming). I'm just trying to figure out if there's something I'm missing. I know we do some very dubious things with PG_G at bootup. Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] All of this is for nothing if we don't go to the stars - JMS/B5 To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Peter Wemm wrote: I'm a little confused as to which bugs are which that we're talking about now. Which is the one that you're trying to sell the info for? I'm not really trying to sell the fix; it's that I'm not willing to give it away when it's no benefit to do so; I'd need a bribe. Kind of like Kirk and Soft Updates. 8-). I was aware of all but the recent AMD problem, but it's pretty obvious in retrospect (Alfred had the idea of putting a fast network interface in an AGP slot some time back, and I looked at AGP a little back then). I can tell you how to cause the problem I'm talking about, off list, if you're willing to laugh along and keep quiet (I have dibs on the bribe 8-)). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
On Thu, 31 Jan 2002, Jason Evans wrote: On Wed, Jan 30, 2002 at 11:14:48PM +0100, Gérard Roudier wrote: Linux can be fixed, but the useless writes of the existing Athlons from the very fast cache to the relatively very slow memory cannot. And all Athlon users may well pay this penalty under any OS... unless we want to disable caching. :) Have you done benchmarks to show that the speculative writes are useless often enough to cause enough memory bus contention that overall performance is degraded, despite the speedups when the speculative writes are valid? I haven't done any benchmark of this sort, neither intend to do any since I haven't time for that. But I wrote in my email that my 2 Athlon systems worked fine and fast, just to indicate that for normal use I didn't see any performance problem at all. I suspect that AMD in fact performed such tests; otherwise they wouldn't have gone to the trouble of implementing speculative writes. The Athlon rewriting same value to cacheable memory under the knees of programmers looks a severe issue to me if it is true. Not only AGP memory can be affected. What about SMP, MMIO (if some cacheable mapping exists), etc...? In my opinion, OSes having some cacheable mapping to AGP memory is not the real problem. Just it has revealed the AMD issue. Gérard. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
On Thu, 31 Jan 2002, Terry Lambert wrote: Cameron, Frank wrote: From what was posted on the linux-kernel list the problem is the OS doing the wrong thing not the hardware. I originally asked the question (albeit not worded as clearly as I should have) because if Microsoft and Linux programmers made the same mistake, might FreeBSD have also. No. FreeBSD does not make active use of 4M pages for anything other than the initial kernel text and data, which is obvious, if you look at /sys/i386/machdep.c. For Linux and Windows, the obvious thing is to not map the memory into an aperture marked cacheable and in a 4M page; it's a coding problem with the use of 4M pages, when memory in them is allocated to AGP. This still doesn't get around the other bug, which happens if you use 4M pages certain obviously useful ways, without waving a dead chicken over certain things. 8-). This one is not $10,000 but $0, since it has already been suggested.:-) Gérard. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
On Thu, Jan 31, 2002 at 09:32:48PM +0100, Gérard Roudier wrote: On Thu, 31 Jan 2002, Jason Evans wrote: On Wed, Jan 30, 2002 at 11:14:48PM +0100, Gérard Roudier wrote: Linux can be fixed, but the useless writes of the existing Athlons from the very fast cache to the relatively very slow memory cannot. And all Athlon users may well pay this penalty under any OS... unless we want to disable caching. :) Have you done benchmarks to show that the speculative writes are useless often enough to cause enough memory bus contention that overall performance is degraded, despite the speedups when the speculative writes are valid? I haven't done any benchmark of this sort, neither intend to do any since I haven't time for that. But I wrote in my email that my 2 Athlon systems worked fine and fast, just to indicate that for normal use I didn't see any performance problem at all. I suspect that AMD in fact performed such tests; otherwise they wouldn't have gone to the trouble of implementing speculative writes. The Athlon rewriting same value to cacheable memory under the knees of programmers looks a severe issue to me if it is true. Not only AGP memory can be affected. What about SMP, MMIO (if some cacheable mapping exists), etc...? I am not familiar with the acronym MMIO is so I can't comment on that. In general though, having memoryspace used for memory-mapped I/O devices (including AGP) marked as cacheable is a bad idea unless you are very careful and know exactly what you are doing. For SMP it shouldn't be any problem. Multi-CPU systems normally run some cache-coherence protocol between themselves to make sure that things like this is not a problem. In my opinion, OSes having some cacheable mapping to AGP memory is not the real problem. Just it has revealed the AMD issue. It might be argued that there should be some cache-coherence protocol between the CPU and the AGP device. Not knowing how AGP is specified I don't know if this interaction between the CPU and AGP is a bug or just working as specified. I suspect it is the latter though. -- Insert your favourite quote here. Erik Trulsson [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
On Fri, 1 Feb 2002, Erik Trulsson wrote: On Thu, Jan 31, 2002 at 09:32:48PM +0100, Gérard Roudier wrote: On Thu, 31 Jan 2002, Jason Evans wrote: On Wed, Jan 30, 2002 at 11:14:48PM +0100, Gérard Roudier wrote: Linux can be fixed, but the useless writes of the existing Athlons from the very fast cache to the relatively very slow memory cannot. And all Athlon users may well pay this penalty under any OS... unless we want to disable caching. :) Have you done benchmarks to show that the speculative writes are useless often enough to cause enough memory bus contention that overall performance is degraded, despite the speedups when the speculative writes are valid? I haven't done any benchmark of this sort, neither intend to do any since I haven't time for that. But I wrote in my email that my 2 Athlon systems worked fine and fast, just to indicate that for normal use I didn't see any performance problem at all. I suspect that AMD in fact performed such tests; otherwise they wouldn't have gone to the trouble of implementing speculative writes. The Athlon rewriting same value to cacheable memory under the knees of programmers looks a severe issue to me if it is true. Not only AGP memory can be affected. What about SMP, MMIO (if some cacheable mapping exists), etc...? I am not familiar with the acronym MMIO is so I can't comment on that. In general though, having memoryspace used for memory-mapped I/O devices (including AGP) marked as cacheable is a bad idea unless you are very careful and know exactly what you are doing. Normally you just need a non-cachable mapping. Nothing should preclude to also have cachable mappings as long as programs donnot misuse _explicitely_ any of the existing mappings. For SMP it shouldn't be any problem. Multi-CPU systems normally run some cache-coherence protocol between themselves to make sure that things like this is not a problem. Theory looks perfect, reality is different. Just I am under the impression that Intel speak about their hardware errata a lot more clearly than AMD seem to do. In my opinion, OSes having some cacheable mapping to AGP memory is not the real problem. Just it has revealed the AMD issue. It might be argued that there should be some cache-coherence protocol between the CPU and the AGP device. Not knowing how AGP is specified I don't know if this interaction between the CPU and AGP is a bug or just working as specified. I suspect it is the latter though. AGP does not require cache to snoop AGP accesses to memory, but it perfectly allows implementations to ensure such coherency. As a result, AGP softwares must not rely on that. I guess that neither Windows 2000 nor Linux were/are relying on such coherency that is explicetely not required by AGP specifications. I never read that AGP specifications allow CPU to screw up AGP memory in the back of actually executed program instructions. Gérard. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Erik Trulsson wrote: The Athlon rewriting same value to cacheable memory under the knees of programmers looks a severe issue to me if it is true. Not only AGP memory can be affected. What about SMP, MMIO (if some cacheable mapping exists), etc...? I am not familiar with the acronym MMIO is so I can't comment on that. Memory Mapped I/O. 8-). In general though, having memoryspace used for memory-mapped I/O devices (including AGP) marked as cacheable is a bad idea unless you are very careful and know exactly what you are doing. What he said. 8-) 8-). For SMP it shouldn't be any problem. Multi-CPU systems normally run some cache-coherence protocol between themselves to make sure that things like this is not a problem. I think the problem is pages in which there are inter-CPU locks being set and cleared. Say you had a speculative write that would clear a lock, only you decide not to clear it because it doesn't happen. In my opinion, OSes having some cacheable mapping to AGP memory is not the real problem. Just it has revealed the AMD issue. It might be argued that there should be some cache-coherence protocol between the CPU and the AGP device. This is what Bruce and Peter suggested; Peter said that he was working on a rewrite of the pmap code and would look in that area. Not knowing how AGP is specified I don't know if this interaction between the CPU and AGP is a bug or just working as specified. I suspect it is the latter though. If it doesn't have to be correct, I can make it as fast as you want! The CPU is so fast, it can execute an infinite loop is 6 seconds! -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Kenneth Culver wrote: Actually FreeBSD does make use of them, but in a way that doesn't cause a problem. There's actually a seperate TLB bug, but FreeBSD doesn't trigger that one, either (Linux can tickle it, when there are certain specific circumstances met). $10,000, and I'll tell you how to fix it. 8-) 8-). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
There's actually a seperate TLB bug, but FreeBSD doesn't trigger that one, either (Linux can tickle it, when there are certain specific circumstances met). Well, I think I know what you're talking about, linux allocates agpgart memory without setting a non-cacheable bit, and then the agp card writes to that memory, but the cpu cached it already, which makes the cache wrong or something like that, and causes the crashes/hangs. I know this is a greatly simplified version of the real problem, but I think this is a linux bug not necesarily an amd bug. Ken To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
RE: AMD AGP Bug
From what was posted on the linux-kernel list the problem is the OS doing the wrong thing not the hardware. I originally asked the question (albeit not worded as clearly as I should have) because if Microsoft and Linux programmers made the same mistake, might FreeBSD have also. -Original Message- From: Kenneth Culver [mailto:[EMAIL PROTECTED]] Sent: Thursday, January 31, 2002 10:42 AM To: Terry Lambert Cc: David Malone; Cameron, Frank; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]' Subject: Re: AMD AGP Bug There's actually a seperate TLB bug, but FreeBSD doesn't trigger that one, either (Linux can tickle it, when there are certain specific circumstances met). Well, I think I know what you're talking about, linux allocates agpgart memory without setting a non-cacheable bit, and then the agp card writes to that memory, but the cpu cached it already, which makes the cache wrong or something like that, and causes the crashes/hangs. I know this is a greatly simplified version of the real problem, but I think this is a linux bug not necesarily an amd bug. Ken To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
RE: AMD AGP Bug
Yeah, that's what I saw on linux-kernel... Ken On Thu, 31 Jan 2002, Cameron, Frank wrote: From what was posted on the linux-kernel list the problem is the OS doing the wrong thing not the hardware. I originally asked the question (albeit not worded as clearly as I should have) because if Microsoft and Linux programmers made the same mistake, might FreeBSD have also. -Original Message- From: Kenneth Culver [mailto:[EMAIL PROTECTED]] Sent: Thursday, January 31, 2002 10:42 AM To: Terry Lambert Cc: David Malone; Cameron, Frank; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]' Subject: Re: AMD AGP Bug There's actually a seperate TLB bug, but FreeBSD doesn't trigger that one, either (Linux can tickle it, when there are certain specific circumstances met). Well, I think I know what you're talking about, linux allocates agpgart memory without setting a non-cacheable bit, and then the agp card writes to that memory, but the cpu cached it already, which makes the cache wrong or something like that, and causes the crashes/hangs. I know this is a greatly simplified version of the real problem, but I think this is a linux bug not necesarily an amd bug. Ken To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
RE: AMD AGP Bug
On Thu, 31 Jan 2002, Kenneth Culver wrote: Yeah, that's what I saw on linux-kernel... You probably didn't see the whole story or just did a too selective reading. You should re-read, in my opinion, since you may have just missed the important part. What I understood is that the Athlon allocates cache lines for speculative writes (if write may hit cachable) and such cache lines are flushed to memory even if the write does not happen. As a result, numerous useless writes to memory may happen under any OS. This does not cause visible issue for memory that is required to be cache coherent. Just AGP accesses are not required to be snooped by cache for performance concerns and as memory given to AGP is intended to contain data as textures that should not need tight synchronisation with CPU. Indeed Linux has cached mapping to AGP memory and this could be avoided. Nevertheless, only strange issues like the AMD AGP bug we are talking about could turn this into a real problem. Linux can be fixed, but the useless writes of the existing Athlons from the very fast cache to the relatively very slow memory cannot. And all Athlon users may well pay this penalty under any OS... unless we want to disable caching. :) Btw, I have 2 Athlon 1.2GHz machines that work just fine and fast for me. Gérard. Ken On Thu, 31 Jan 2002, Cameron, Frank wrote: From what was posted on the linux-kernel list the problem is the OS doing the wrong thing not the hardware. I originally asked the question (albeit not worded as clearly as I should have) because if Microsoft and Linux programmers made the same mistake, might FreeBSD have also. -Original Message- From: Kenneth Culver [mailto:[EMAIL PROTECTED]] Sent: Thursday, January 31, 2002 10:42 AM To: Terry Lambert Cc: David Malone; Cameron, Frank; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]' Subject: Re: AMD AGP Bug There's actually a seperate TLB bug, but FreeBSD doesn't trigger that one, either (Linux can tickle it, when there are certain specific circumstances met). Well, I think I know what you're talking about, linux allocates agpgart memory without setting a non-cacheable bit, and then the agp card writes to that memory, but the cpu cached it already, which makes the cache wrong or something like that, and causes the crashes/hangs. I know this is a greatly simplified version of the real problem, but I think this is a linux bug not necesarily an amd bug. Ken To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-bugs in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Kenneth Culver wrote: There's actually a seperate TLB bug, but FreeBSD doesn't trigger that one, either (Linux can tickle it, when there are certain specific circumstances met). Well, I think I know what you're talking about, linux allocates agpgart memory without setting a non-cacheable bit, and then the agp card writes to that memory, but the cpu cached it already, which makes the cache wrong or something like that, and causes the crashes/hangs. I know this is a greatly simplified version of the real problem, but I think this is a linux bug not necesarily an amd bug. No, you are describing the coherency bug in the chipset used for the AGP with AMD processors, which should enforce a coherency cycle, and cache invalidation for AGP based writes of the shared memory region. They are treating it as if it were non-cacheable dual ported SDRAM. There *really* is a bug in P4 and above and in late model AMD chips with 4M pages. It is very easy to reproduce with a few small kernel modifications (to actively use 4M pages) and 256M-512M of RAM. I'll characterize it further and provide the software workaround, for $10,000. It's about 16 machine instructions, if you write it out as code. Meanwhile, Linux and Windows can keep turning off the PSE, and losing 4-14% of their performance, depending on their application. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Cameron, Frank wrote: From what was posted on the linux-kernel list the problem is the OS doing the wrong thing not the hardware. I originally asked the question (albeit not worded as clearly as I should have) because if Microsoft and Linux programmers made the same mistake, might FreeBSD have also. No. FreeBSD does not make active use of 4M pages for anything other than the initial kernel text and data, which is obvious, if you look at /sys/i386/machdep.c. For Linux and Windows, the obvious thing is to not map the memory into an aperture marked cacheable and in a 4M page; it's a coding problem with the use of 4M pages, when memory in them is allocated to AGP. This still doesn't get around the other bug, which happens if you use 4M pages certain obviously useful ways, without waving a dead chicken over certain things. 8-). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
RE: AMD AGP Bug
I'm thinking you may have misunderstood, the people at amd themselves said this wasn't an amd bug... why would you write from cache to memory, I think it's the other way around, stuff getting written to cache from memory... (being retrieved by the cpu) I'll have to read the article again to be sure since it was a week ago when I read it. Ken On Wed, 30 Jan 2002, [ISO-8859-1] Gérard Roudier wrote: On Thu, 31 Jan 2002, Kenneth Culver wrote: Yeah, that's what I saw on linux-kernel... You probably didn't see the whole story or just did a too selective reading. You should re-read, in my opinion, since you may have just missed the important part. What I understood is that the Athlon allocates cache lines for speculative writes (if write may hit cachable) and such cache lines are flushed to memory even if the write does not happen. As a result, numerous useless writes to memory may happen under any OS. This does not cause visible issue for memory that is required to be cache coherent. Just AGP accesses are not required to be snooped by cache for performance concerns and as memory given to AGP is intended to contain data as textures that should not need tight synchronisation with CPU. Indeed Linux has cached mapping to AGP memory and this could be avoided. Nevertheless, only strange issues like the AMD AGP bug we are talking about could turn this into a real problem. Linux can be fixed, but the useless writes of the existing Athlons from the very fast cache to the relatively very slow memory cannot. And all Athlon users may well pay this penalty under any OS... unless we want to disable caching. :) Btw, I have 2 Athlon 1.2GHz machines that work just fine and fast for me. Gérard. Ken On Thu, 31 Jan 2002, Cameron, Frank wrote: From what was posted on the linux-kernel list the problem is the OS doing the wrong thing not the hardware. I originally asked the question (albeit not worded as clearly as I should have) because if Microsoft and Linux programmers made the same mistake, might FreeBSD have also. -Original Message- From: Kenneth Culver [mailto:[EMAIL PROTECTED]] Sent: Thursday, January 31, 2002 10:42 AM To: Terry Lambert Cc: David Malone; Cameron, Frank; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]' Subject: Re: AMD AGP Bug There's actually a seperate TLB bug, but FreeBSD doesn't trigger that one, either (Linux can tickle it, when there are certain specific circumstances met). Well, I think I know what you're talking about, linux allocates agpgart memory without setting a non-cacheable bit, and then the agp card writes to that memory, but the cpu cached it already, which makes the cache wrong or something like that, and causes the crashes/hangs. I know this is a greatly simplified version of the real problem, but I think this is a linux bug not necesarily an amd bug. Ken To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-bugs in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
On Wed, Jan 30, 2002 at 11:14:48PM +0100, Gérard Roudier wrote: Linux can be fixed, but the useless writes of the existing Athlons from the very fast cache to the relatively very slow memory cannot. And all Athlon users may well pay this penalty under any OS... unless we want to disable caching. :) Have you done benchmarks to show that the speculative writes are useless often enough to cause enough memory bus contention that overall performance is degraded, despite the speedups when the speculative writes are valid? I suspect that AMD in fact performed such tests; otherwise they wouldn't have gone to the trouble of implementing speculative writes. Jason To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
I was under the impression that they were writing into the cache not out of it... I really need to read that article again :-D Ken On Thu, 31 Jan 2002, Jason Evans wrote: On Wed, Jan 30, 2002 at 11:14:48PM +0100, Gérard Roudier wrote: Linux can be fixed, but the useless writes of the existing Athlons from the very fast cache to the relatively very slow memory cannot. And all Athlon users may well pay this penalty under any OS... unless we want to disable caching. :) Have you done benchmarks to show that the speculative writes are useless often enough to cause enough memory bus contention that overall performance is degraded, despite the speedups when the speculative writes are valid? I suspect that AMD in fact performed such tests; otherwise they wouldn't have gone to the trouble of implementing speculative writes. Jason To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Terry Lambert wrote:
Cameron, Frank wrote:
From what was posted on the linux-kernel list the problem is the OS
doing the wrong thing not the hardware. I originally asked the
question (albeit not worded as clearly as I should have) because if
Microsoft and Linux programmers made the same mistake, might
FreeBSD have also.
No. FreeBSD does not make active use of 4M pages for anything
other than the initial kernel text and data, which is obvious,
if you look at /sys/i386/machdep.c.
Actually, it is obvious if you actually do look at the pmap.c that we *do*
use 4MB pages for device mappings.
/*
* This code maps large physical mmap regions into the
* processor address space. Note that some shortcuts
* are taken, but the code works.
*/
if (pseflag (object-type == OBJT_DEVICE)
((addr (NBPDR - 1)) == 0) ((size (NBPDR - 1)) == 0)) {
...
for(i = 0; i npdes; i++) {
pmap-pm_pdir[ptepindex] =
ptepa | PG_U | PG_RW | PG_V | PG_PS;
ptepa += NBPDR;
ptepindex += 1;
}
...
Even doing a simple grep for the 4MB page flag (PG_PS) and using a
little initiative would have shown this.
Cheers,
-Peter
--
Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
All of this is for nothing if we don't go to the stars - JMS/B5
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Kenneth Culver wrote: Yeah, that's what I saw on linux-kernel... Ken We need to use the PAT cpu_features feature. This gives us 8 page attribute modes instead of simple no-cache / writethrough flags. We can (and must) control more carefully the speculative hardware prefetch, for example. I've been thinking about this with the pmap revamp that I'm working on. It may solve the athlon problems too. On Thu, 31 Jan 2002, Cameron, Frank wrote: From what was posted on the linux-kernel list the problem is the OS doing the wrong thing not the hardware. I originally asked the question (albeit not worded as clearly as I should have) because if Microsoft and Linux programmers made the same mistake, might FreeBSD have also. -Original Message- From: Kenneth Culver [mailto:[EMAIL PROTECTED]] Sent: Thursday, January 31, 2002 10:42 AM To: Terry Lambert Cc: David Malone; Cameron, Frank; '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]' Subject: Re: AMD AGP Bug There's actually a seperate TLB bug, but FreeBSD doesn't trigger that one, either (Linux can tickle it, when there are certain specific circumstances met). Well, I think I know what you're talking about, linux allocates agpgart memory without setting a non-cacheable bit, and then the agp card writes to that memory, but the cpu cached it already, which makes the cache wrong or something like that, and causes the crashes/hangs. I know this is a greatly simplified version of the real problem, but I think this is a linux bug not necesarily an amd bug. Ken To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-bugs in the body of the message Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] All of this is for nothing if we don't go to the stars - JMS/B5 To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Peter Wemm wrote: We need to use the PAT cpu_features feature. This gives us 8 page attribute modes instead of simple no-cache / writethrough flags. We can (and must) control more carefully the speculative hardware prefetch, for example. I've been thinking about this with the pmap revamp that I'm working on. It may solve the athlon problems too. This sounds cool. Do you have references to the page attribute stuff? The books I have here don't discuss it; the only thing I see are 3 bits (9,10,11) that are available in the PDE and PTE? -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Peter Wemm wrote: No. FreeBSD does not make active use of 4M pages for anything other than the initial kernel text and data, which is obvious, if you look at /sys/i386/machdep.c. Actually, it is obvious if you actually do look at the pmap.c that we *do* use 4MB pages for device mappings. The active use I was referring to was kernel space, and this sets PG_U, so there are not fault handler invocations on attempts to write the pages. This code also does not set the PG_G bit, like Linux does. This code has a requirement that the address mapping occur on a 4M boundary, and that it be an even multiple of 4M. My reading of the AGPGART kernel module indicates to me that this code is not triggered in normal use. Correct me if my interpretation here is wrong (I'd like to know *how*; the default aperture appears to be 1M, unless it is overridden, and the alignment isn't on the right boundary). In any case, let's say that it gets invoked on the AGP hardware, and establishes a 4M mapping on FreeBSD, as well. I think the Linux problem has more to do with having *both* 4K and 4M mappings to the same physical memory area at the same time. FreeBSD avoids doing this to active pages; the initial setup replaces the PDE and loses the 4K worth of 4K mapping tables, and the later code that you reference doesn't replace any existing entries. It's accidental, but still, it should mean that FreeBSD is not succeptible to the problem. The real hint in characterizing the problem is to ask why does it happen with 4M pages, but doesn't also happen with 4K pages?. This indicates that the problem is really a TLB problem, and therefore a real processor problem: the problem should happen no matter what, if it's a problem with writing to mapped memory use by the AGP not giving an invalidation signal to the CPU on writes, rather than a problem in the CPUs handling of 4M page invalidation itself, a problem that does not exist in the 4K invalidation case. Unfortuantely, I would need to upgrade both my FreeBSD and my XFree86 to try this with the AGP on my AMD box (it has a 0x7121 ID, and so can't GARTIOCINFO correctly) to be able to even try to repeat the problem, and Fry's does not appear to be selling FreeBSD CDROM sets any more, even with the books attached. 8-(. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
On Thu, Jan 31, 2002 at 07:22:25PM -0800, Terry Lambert wrote: Peter Wemm wrote: We need to use the PAT cpu_features feature. This gives us 8 page attribute modes instead of simple no-cache / writethrough flags. We can (and must) control more carefully the speculative hardware prefetch, for example. I've been thinking about this with the pmap revamp that I'm working on. It may solve the athlon problems too. This sounds cool. Do you have references to the page attribute stuff? The books I have here don't discuss it; the only thing I see are 3 bits (9,10,11) that are available in the PDE and PTE? Well, twice in this thread you've offered info for $1. I'm sure Peter can reciprocate with the info you seek for a similar price. -- Steve To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Steve Kargl wrote: This sounds cool. Do you have references to the page attribute stuff? The books I have here don't discuss it; the only thing I see are 3 bits (9,10,11) that are available in the PDE and PTE? Well, twice in this thread you've offered info for $1. I'm sure Peter can reciprocate with the info you seek for a similar price. 8-). The info I offered is unrelated to FreeBSD. It lets the Linux people and the Windows people turn the PSE back on. They can afford it where FreeBSD can't. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
AMD AGP Bug
Has this issue been addressed in FreeBSD: http://www.geocrawler.com/lists/3/Linux/35/175/7626960/ http://slashdot.org/article.pl?sid=02/01/24/1910227mode=thread To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
On Wed, Jan 30, 2002 at 12:13:07PM -0500, Cameron, Frank wrote: Has this issue been addressed in FreeBSD: http://www.geocrawler.com/lists/3/Linux/35/175/7626960/ http://slashdot.org/article.pl?sid=02/01/24/1910227mode=thread This is believed not to have any impact on FreeBSD because FreeBSD doesn't make much use of large pages. See Terry Lambert's post to freebsd-current on 21st Jan for more details. David. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
You should check the archives of the FreeBSD mailing lists before sending a message to 4 of the lists. This quiestion has been answered several times on the FreeBSD lists. The answer is that this isn't even really an AMD AGP bug, it's a bug in the way linux handles mapping it's AGP memory. FreeBSD isn't affected by this at all. Ken On Wed, 30 Jan 2002, Cameron, Frank wrote: Has this issue been addressed in FreeBSD: http://www.geocrawler.com/lists/3/Linux/35/175/7626960/ http://slashdot.org/article.pl?sid=02/01/24/1910227mode=thread To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
Re: AMD AGP Bug
Actually FreeBSD does make use of them, but in a way that doesn't cause a problem. Ken On Wed, 30 Jan 2002, David Malone wrote: On Wed, Jan 30, 2002 at 12:13:07PM -0500, Cameron, Frank wrote: Has this issue been addressed in FreeBSD: http://www.geocrawler.com/lists/3/Linux/35/175/7626960/ http://slashdot.org/article.pl?sid=02/01/24/1910227mode=thread This is believed not to have any impact on FreeBSD because FreeBSD doesn't make much use of large pages. See Terry Lambert's post to freebsd-current on 21st Jan for more details. David. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-current in the body of the message
