Re: How to visit physical memory above 4G?
On Sat, Aug 04, 2001 at 02:38:23AM -0300, Rik van Riel wrote: On Fri, 3 Aug 2001, Terry Lambert wrote: This is a trivial implementation. I'm not very impressed. Personally, I'm not interested in a huge user space, Maybe not you, but I bet the database and scientific computing people will be interested in having 64 GB memory support in this simple way. All I was interested so far was more address space for a single process and more physical memory for kernel use. Both I've found in ALPHA machines and am happy with it. Go and compare prices for ALPHA and i386 systems which support more than 4G of physical memory and don't forget that you put a lot of CPU power in managing the memory with i386. In the unix world there is less reason to stuck with a single architecture as in Windows. If I need more than one process to gain use of the memory I would consider using a second machine. Fully populating both the transmit and receive windows for 1M connections is 32G of RAM, right there... and it better be kernel RAM, or you're screwed. Well, you _could_ store this memory in files, which get mapped and unmapped by the same code the filesystem code uses to access file data in non-kernel-mapped RAM. Consider that you don't need such evilness on existing Machines such as ALPHA. Well AFAIK FreeBSD currently doesn't support ALPHAs with that amount of memory very well - but this can be considered more as a bug than a missing feature and can be fixed. -- B.Walter COSMO-Project http://www.cosmo-project.de [EMAIL PROTECTED] Usergroup [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
At 10:57 PM -0700 8/3/01, Terry Lambert wrote: Rik van Riel wrote: This is a trivial implementation. I'm not very impressed. Personally, I'm not interested in a huge user space, Maybe not you, but I bet the database and scientific computing people will be interested in having 64 GB memory support in this simple way. You mean 4G, of course, since the process address space remains limites to 32 bits... For what it's worth, we ran into some similar problem with a mainframe operating system I used to work on. We ended up creating something we called named address spaces for some user processes. Basically, it was just a quick swapping mechanism. In the context of IA-32, you could maybe have the first gigabyte of space as fixed, and the remaining three gigabytes as multiple (named) address spaces. Each named-address space could be between 1 and 3 gig, and you could have several of them. You'd make a system call to switch from one named-address space to a different one. It would not be practical for all user processes, but it would be useful for some of them. One ironic thing about this named-address space idea was that we had talked about it off-and-on for a few years, but we didn't actually *do* it until just as we were getting to the point where we could switch from 24-bit addressing to 31-bit addressing on our OS. We hit something where we just had to have the extra space right away (quicker than we could implement 31-bit addressing in userland processes). Once we decided to do this named-address space idea, it turned out it wasn't all that hard to implement. However, the current situation isn't quite the same as that one, and in the land of freebsd I'd think it would be better to concentrate on good support for the chips which do support 64-bit addresssing. I just thought it was worth pointing out that a process might well be limited to 32-bit addressing, and yet not be limited to 4-gig of usable memory space. -- Garance Alistair Drosehn= [EMAIL PROTECTED] Senior Systems Programmer or [EMAIL PROTECTED] Rensselaer Polytechnic Instituteor [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
mark tinguely wrote: Also, the PIII CAN'T natively support more than 4GB of ram. If a particular PIII motherboard supports this, then it's using some kind of wierd chipset that allows this to happen. 4GB is the limit with a 32 bit chip I believe; and the PIII is a 32-bit chip. Since the Pentium Pro processor, the Intel chipsets support a physical address extension (PAE) which has 4 extra addressing bits, and a third level of page table indirection called the page-directory-pointer-table base address field. Bit 5 of control register 4, and then it uses the top 27 bits of control register 3 to select a 32 byte aligned region in the lower 4G. It also changes the PSE bit to refer to 2M instead of 4M pages, so your would needto DISABLE_PSE, or the FreeBSD kernel would freak when it enabled the 4M page on the kernel itself. Then the high 4 bits are used to pick a pointer entry (which is effectively a software segment register select, for all practical purposes), giving you 64G of addressable space, in chunks of 4G at a time. Practically, you end up having to overlap this, which tends to cut you down to 32G. The addressing use 64 bits for a memory pointer and the additional page indirection add to the overhead. The stickler is the MMU is still 32 bits. This means the PAE must segment the 64GB space into 4GB segments or 4 1GB segments. The OS must manage which pages are viewable to the process at this time. Not only that: you reload CR3, and none of these pages are really global, so you can't set the PG_G bit, and so you get the full TLB shootdown on everything, so a segment switch ends up shooting _everything_ down. There is a third mode of addressing using 2MB pages (simular to the 4MB page addressing mode for the 32 bit addressing scheme) that will only give a process access to 4GB of memory (not segmentable to a larger space, but can address physical memory located above the 4GB address). Not really useful, unless you go back to a task gate, which itself will limit you to 1024 things; with code + data, you end up halving that to 512, minus overhead drops it to 510, so you end up with a limitation on number of processes. You could do all the switching manually, but it is very, very hard. Further, you can write off shared memory and shared libraries, and some types of IPC (e.g. descriptor passing), unless you want to rework everything. IMO, the resulting kernel would be so slow as to prevent the changes from being useful, due to their expense. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Julian Elischer wrote: On Thu, 2 Aug 2001, Rik van Riel wrote: On the really large machines, this can lead to the situation where even the page tables hardly fit into KVA. 4MB pages seem like the only solution ... There is no reason why we need to keep the kernel and the user process in the same 4GB map except for efficiency. There have been many UNIX machines in the past which put them in separate virtual spaces That was on machines where the CPU could actually address two separate spaces at once, right ? (like eg. m68k) The kernel would haev 4GB for itself and each process would have 4BG. System calls would be come more expensive as each would require a full page-table swap and a TLB flush. However it might be worth it for some people. Interrupt handling would also require a full page table swap and TLB flush. Considering that, I think the number of people for whom this will be worth it has probably dropped a bit ;) Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Terry Lambert wrote: Name an OS that supports this; more than likely, you will have to appeal to a purpose built embedded system. errr, linux? To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
John Baldwin wrote: Err. hang on. This has zero to do with segmentation. Zip, nada. PAE is completely in the paging side of things. No matter what fun games you play with segmentation, you still end up with a 32-bit linear address that gets handed off to the paging translations. PAE just allows you to use more backing store across multiple processes, but you are still stuck with a 4gb virtual address space for processes. (Including KVM) IMO, the 4 bit selector register is the moral equivalent of a segment register. Personally, I think it's much less useful to run the kernel out of KVA space, than it is to have more memory available to the kernel for things like mbufs, so I'm not really very interested in trying to raise the per process address space limits this way. You could actually get 4G for the kernel and 4G for processes using this, but you would only need two segments to make this happen; mapping the other stuff at the same time makes little sense: you just map a window on it to implement region overlays for user or kernel data paging. Given the compiler tools we have, this still limits you to using only 4G in a given VA space, unless you did something evil, like add HLOCK/HUNLOCK, etc.. But to directly answer your question: by rewriting much of the low core virtual memory and page mapping handling code to know about segmentation. No, to rewrite said code to handle a different type of page table structure. Virtual table structure/segements: same difference: I'm now wdoing in software what I bought hardware to get away from having to do in software. Given the vastly simplified page management in Linux, I could see how there wouldn't really be a big performance loss over the way Linux does things without this, so it might be OK, so long as there were no shared memory regions, semaphores, etc.. That really makes it pretty useless. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
BUT, don't the motherboards also have to support this? And isn't it only supported through some wierd segmentation thing? KEn On Thu, 2 Aug 2001, John Baldwin wrote: On 02-Aug-01 Kenneth Wayne Culver wrote: Also, the PIII CAN'T natively support more than 4GB of ram. If a particular PIII motherboard supports this, then it's using some kind of wierd chipset that allows this to happen. 4GB is the limit with a 32 bit chip I believe; and the PIII is a 32-bit chip. Ken Go look at some Intel docs. P6 chips since the Pentium Pro (yes, before Pentium II) have supported PAE which allows for a 36-bit physical address. -- John Baldwin [EMAIL PROTECTED] -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc Power Users Use the Power to Serve! - http://www.FreeBSD.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Rik van Riel wrote: [ ... 4G on 32 bit macines ... ] The short answer is you can't. The longer answer is that you end up having to window it using segmentation; Only if you want to use it all within one process. No. It still bites you if you want to do IPC, etc., since you can not guarantee the structures used for this are all within the non-segmented region of memory. You can have multiple 2 GB (that's the maximum process size in FreeBSD, right?) programs at the same time, happily using all physical memory. The default maximum size for FreeBSD is 3G. You can tune this up or down, with the limit being that the larger the user space, the smaller the KVA space, and vice versa. Only the FreeBSD memory management subsystem doesn't support it (yet?). It's not a question of supporting it, it's a question of whether or not it's a useful idea at all. This basically means that the memory is useless as a DMA target or source for disk controllers or gigabit ethernet cards, and is pretty useless for swap, ... So for limited uses in data intensive applications, it might be usable, And for those data intensive applications, it is very useful indeed... I have yet to see one person using it for anything. So far, it is nothing more than marketing fodder: I haven't seen one motherboard capable of more than 4G worth of SIMMs. But to directly answer your question: by rewriting much of the low core virtual memory and page mapping handling code to know about segmentation. Not just that. There is a more insidious problem with the FreeBSD VM code and support of huge machines. Not really. The part of handling the PAE extended page table format and mapping high memory pages in and out of KVA (kernel virtual address) memory to copy stuff is easy. Yes. Problem is that you'll have to fit all of FreeBSD's VM data structures in the 2GB of KVA. This just isn't going to fit with the size the data structures have today ... I currently run with 3G+ of KVA; it would be simple to invert this, but this leaves me a 1G user space window, with 3G available for kernel structures, etc.. It takes about 1G for all of the kernel support stuff for 4G, with an allowance for 1/8th million open network connections. So it's not unreasonable to think of putting 8G or 16G in a box, and being able to map it all. So in order to support huge memory machines right, you'd have to put a number of FreeBSD's VM data structures on a rather strict diet. Not really. There's always 4M pages. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Oh ok, I knew that regular PIII's only had 32 bits... but it's still obviously a pain in the butt to use above 4GB. Ken On Thu, 2 Aug 2001, Rik van Riel wrote: On Thu, 2 Aug 2001, Kenneth Wayne Culver wrote: Also, the PIII CAN'T natively support more than 4GB of ram. If a particular PIII motherboard supports this, then it's using some kind of wierd chipset that allows this to happen. 4GB is the limit with a 32 bit chip I believe; and the PIII is a 32-bit chip. The Xeon series have 32 bits of virtual address space and 36 bits of physical address space. Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
The costs involved in doing DMA to/from the memory region above 4G will be incredible, unless the address space is both exported, and known, to the PCI bus; even then, it could only work for 64 bit cards, since 32 bith cards will only be able to address the first 4G of physical memory. Actually, most server-class hardware these days will have no trouble with DMA above 4GB; even the 32-bit devices support DAC. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, mark tinguely wrote: The addressing use 64 bits for a memory pointer and the additional page indirection add to the overhead. The stickler is the MMU is still 32 bits. This means the PAE must segment the 64GB space into 4GB segments or 4 1GB segments. The OS must manage which pages are viewable to the process at this time. There's a better option. The application does this stuff itself, by mapping and unmapping SHM segments. This needs to be done anyway, since the application wouldn't be happy if the OS randomly remapped its address space to different pages ;)) There is a third mode of addressing using 2MB pages (simular to the 4MB page addressing mode for the 32 bit addressing scheme) that will only give a process access to 4GB of memory (not segmentable to a larger space, but can address physical memory located above the 4GB address). You can do that with 4kB pages too. Both the middle page directory and page table have 512 64-bit entries, so you can simply map 4kB pages from above the 4GB physical boundary... regards, Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Mike Smith wrote: Julian is on crack. DAC (Double Address Cycle) is a relatively recent addition to PCI that allows 32-bit cards with 64-bit savvy logic to talk to host memory using 64-bit target addresses. well day 1 was an exageration, but my 1995 PCI stuff already includes it. Crack? no.. optimism, maybe.. Older systems used a scatter-gather mechanism to present a virtualised view of the system's physical address space to the PCI bus (typically you'll see this in Alpha systems). To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Rik van Riel wrote: Only if you want to use it all within one process. No. It still bites you if you want to do IPC, etc., since you can not guarantee the structures used for this are all within the non-segmented region of memory. Wrong. Your process can have pages from all over the 64 GB mapped into its page tables. Try doing this with code pages in one segment, the stack in another, and the data being referenced in a third. It will not work. Each process has 3 GB of virtual memory mapped to any of the pages of the 64 GB of physical space. Like I said before, this is not useful. The only marginal use for such a thing is for implementing an L3 cache in software, or for implementing multiple virtual machines on one box. Only the FreeBSD memory management subsystem doesn't support it (yet?). It's not a question of supporting it, it's a question of whether or not it's a useful idea at all. I have yet to see one person using it for anything. So far, it is nothing more than marketing fodder: I haven't seen one motherboard capable of more than 4G worth of SIMMs. I've seen a bunch of the machines. They're rather popular with the database folks. Name an OS that supports this; more than likely, you will have to appeal to a purpose built embedded system. Problem is that you'll have to fit all of FreeBSD's VM data structures in the 2GB of KVA. This just isn't going to fit with the size the data structures have today ... So it's not unreasonable to think of putting 8G or 16G in a box, and being able to map it all. You can never map it all, since your virtual address space is limited to 4GB... Overlays. A technology from the dawn of time, I know, but so are segment registers. Basically the database folks are really keen on keeping their 3GB user addressable memory, so the kernel will remain limited to 1GB of KVA. They shouldn't care, since they are getting more memory. On the really large machines, this can lead to the situation where even the page tables hardly fit into KVA. 4MB pages seem like the only solution ... This is why we use 64 bit processors for really large machines, and we say that 36 bit address spaces are really pretty useless, and will be obsolete by the time the code is complete, because of ia64 being a cheaper and faster soloution. 8-p. 36 bits only gives you 2^4 * 4G, or 64G, anyway, and it is hardly worth the segment thrashing and instruction and data cache shootdowns to be able to handle it. You are better off throwing 16 machines at the problem: your major cost item is going to be memory, anyway, and getting 64G in one box is going to cost you significantly more than just putting together multiple boxes. If the locality is such that 2G per process is OK, you might as well be on seperate boxes with non-segmenet swapped memory. (well, there's also the mess of shared page tables, but nobody is keen on the locking issues those imply) It's much, much worse than that. Like I said before, you could do it pretty trivially, but the expense of doing it will be so high, relatively, that you might as well buy an Alpha or PA-RISC box and call it a day, if you need it now, or just wait for IA64, rather than throwing developement effort into something that will end up in a scrap heap before it gets reasonable performance, or perhaps before it's even deployed. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
No The space is linear in physical space and if you have PCI/64 capable devices they can access it all too. (In fact 64 bit addresses have been supported even in 32 bit wide PCI since day 1). On Thu, 2 Aug 2001, Kenneth Wayne Culver wrote: BUT, don't the motherboards also have to support this? And isn't it only supported through some wierd segmentation thing? KEn On Thu, 2 Aug 2001, John Baldwin wrote: On 02-Aug-01 Kenneth Wayne Culver wrote: Also, the PIII CAN'T natively support more than 4GB of ram. If a particular PIII motherboard supports this, then it's using some kind of wierd chipset that allows this to happen. 4GB is the limit with a 32 bit chip I believe; and the PIII is a 32-bit chip. Ken Go look at some Intel docs. P6 chips since the Pentium Pro (yes, before Pentium II) have supported PAE which allows for a 36-bit physical address. -- John Baldwin [EMAIL PROTECTED] -- http://www.FreeBSD.org/~jhb/ PGP Key: http://www.baldwin.cx/~john/pgpkey.asc Power Users Use the Power to Serve! - http://www.FreeBSD.org/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Also, the PIII CAN'T natively support more than 4GB of ram. If a particular PIII motherboard supports this, then it's using some kind of wierd chipset that allows this to happen. 4GB is the limit with a 32 bit chip I believe; and the PIII is a 32-bit chip. Ken On Thu, 2 Aug 2001, Rik van Riel wrote: On Wed, 1 Aug 2001, Terry Lambert wrote: craig wrote: I know PIII can support 64G physical memory. In FreeBSD how can I visit such range memory(4G-64G) ? The short answer is you can't. The longer answer is that you end up having to window it using segmentation; Only if you want to use it all within one process. You can have multiple 2 GB (that's the maximum process size in FreeBSD, right?) programs at the same time, happily using all physical memory. Only the FreeBSD memory management subsystem doesn't support it (yet?). This basically means that the memory is useless as a DMA target or source for disk controllers or gigabit ethernet cards, and is pretty useless for swap, ... So for limited uses in data intensive applications, it might be usable, And for those data intensive applications, it is very useful indeed... But to directly answer your question: by rewriting much of the low core virtual memory and page mapping handling code to know about segmentation. Not just that. There is a more insidious problem with the FreeBSD VM code and support of huge machines. The part of handling the PAE extended page table format and mapping high memory pages in and out of KVA (kernel virtual address) memory to copy stuff is easy. Problem is that you'll have to fit all of FreeBSD's VM data structures in the 2GB of KVA. This just isn't going to fit with the size the data structures have today ... So in order to support huge memory machines right, you'd have to put a number of FreeBSD's VM data structures on a rather strict diet. regards, Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Terry Lambert wrote: Rik van Riel wrote: Only if you want to use it all within one process. No. It still bites you if you want to do IPC, etc., since you can not guarantee the structures used for this are all within the non-segmented region of memory. Wrong. Your process can have pages from all over the 64 GB mapped into its page tables. Try doing this with code pages in one segment, the stack in another, and the data being referenced in a third. It will not work. Of course it cannot work. The segments you refer to will map to the linear 4GB space represented by the page tables. Those page tables can map, in their 4GB virtual space, pages from all over the 64GB physical space. I've seen a bunch of the machines. They're rather popular with the database folks. Name an OS that supports this; more than likely, you will have to appeal to a purpose built embedded system. Linux SCO Unix NT *cough* So it's not unreasonable to think of putting 8G or 16G in a box, and being able to map it all. You can never map it all, since your virtual address space is limited to 4GB... Overlays. A technology from the dawn of time, I know, but so are segment registers. s/You can never map it all/You can never map it all at once/ Happy now? ;) Basically the database folks are really keen on keeping their 3GB user addressable memory, so the kernel will remain limited to 1GB of KVA. They shouldn't care, since they are getting more memory. You're telling the people working with 50GB datasets that they shouldn't care about how many of their 512MB segments they can map in at once? ;) or just wait for IA64, rather than throwing developement effort into something that will end up in a scrap heap before it gets reasonable performance, or perhaps before it's even deployed. Ssssh, don't tell that to the folks who are using such machines today ;) cheers, Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
RE: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Charles Randall wrote: From: Terry Lambert [mailto:[EMAIL PROTECTED]] I have yet to see one person using it for anything. So far, it is nothing more than marketing fodder: I haven't seen one motherboard capable of more than 4G worth of SIMMs. The Dell PowerEdge 6450 supports 8 GB of RAM. http://www.dell.com/us/en/biz/products/model_pedge_pedge_6400.htm It's all a trick, they don't really exist. After all, why would people ever shell out the money for one of these useless beasts that won't be running until ia64 has gotten cheaper than these machines ? Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
No The space is linear in physical space and if you have PCI/64 capable devices they can access it all too. (In fact 64 bit addresses have been supported even in 32 bit wide PCI since day 1). OK, then what was that whole paging thing everyone was talking about, I thought that was partially done in hardware on the chipset of the motherboard... or was that completely in the operating system? Julian is on crack. DAC (Double Address Cycle) is a relatively recent addition to PCI that allows 32-bit cards with 64-bit savvy logic to talk to host memory using 64-bit target addresses. Older systems used a scatter-gather mechanism to present a virtualised view of the system's physical address space to the PCI bus (typically you'll see this in Alpha systems). -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Terry Lambert wrote: Original poster said he was working on it for Linux, which means it's not done, which means not Linux. It's been running for a while now, integrated in the 2.4 kernel. The way Linux manages to avoid the horrors you describe is by simply not letting the kernel use more than its 1GB (or 2GB) of KVA. No special tricks to make the kernel use more memory, let it go on a diet instead. FWIW, on the Linux has it, so FreeBSD should have it arguments: If all your friends jumped off a cliff... Fully agreed, but as long as you don't do any weird tricks trying to let the kernel itself use more memory, it's not at all like jumping from a cliff ... Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://distro.conectiva.com/ Send all your spam to [EMAIL PROTECTED] (spam digging piggy) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Terry Lambert wrote: Correct me if I'm wrong, and 64 bit PCI cards can in fact DMA at offsets above 4G, in the physical address space... They can. And for 32 bit PCI cards you simply use bounce buffers in the same way you handle ISA bounce buffers. It's ugly, but if you have a huge dataset it's better than getting the data from disk. Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://distro.conectiva.com/ Send all your spam to [EMAIL PROTECTED] (spam digging piggy) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Terry Lambert wrote: Rik van Riel wrote: Only the FreeBSD memory management subsystem doesn't support it (yet?). It's not a question of supporting it, it's a question of whether or not it's a useful idea at all. I have yet to see one person using it for anything. So far, it is nothing more than marketing fodder: I haven't seen one motherboard capable of more than 4G worth of SIMMs. Every box based on the Intel Saber board is capable of up to 16G. I've seen a bunch of the machines. They're rather popular with the database folks. Name an OS that supports this; more than likely, you will have to appeal to a purpose built embedded system. UnixWare (A.K.A. OpenUNIX). As far as I understand, Oracle maps in the pieces of SGA as it needs them, keeping the total mapped size under 3G. -SB To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Rik van Riel wrote: BUT, don't the motherboards also have to support this? And isn't it only supported through some wierd segmentation thing? Yes, the mainboard needs to support the memory. No, there is no weird segmentation thing, at least not visible from software. Last time I looked, the kernel was software... -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
RE: How to visit physical memory above 4G?
From: Terry Lambert [mailto:[EMAIL PROTECTED]] I have yet to see one person using it for anything. So far, it is nothing more than marketing fodder: I haven't seen one motherboard capable of more than 4G worth of SIMMs. The Dell PowerEdge 6450 supports 8 GB of RAM. http://www.dell.com/us/en/biz/products/model_pedge_pedge_6400.htm If I understand your comments in a few follow-up messages correctly you're saying that this effort may be better spent by working on an IA-64 port and making it support large memory configurations? Can you elaborate? -Charles To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Terry Lambert wrote: This basically means that the memory is useless as a DMA target or source for disk controllers or gigabit ethernet cards, and is pretty useless for swap, if you ever have to copy from one section to another (e.g. for IPC, SYSV shared memory, mmap'ed files, VM, or buffer cache, etc.). For the PCI cards supporting the 64-bit addresses you can. (Note that in PCI support of the 64-bit bus is independent of the 64-bit address space, the 64-bit addresses can be multiplexed over a physically 32-bit bus). -SB To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Julian Elischer wrote: No The space is linear in physical space and if you have PCI/64 capable devices they can access it all too. (In fact 64 bit addresses have been supported even in 32 bit wide PCI since day 1). It's been my experience that the TIGON cards take a 32 bit DMA target address, not a 64 bit DMA target address, and that the 54 bit width was only used for the data transfer, not for the address offset. Correct me if I'm wrong, and 64 bit PCI cards can in fact DMA at offsets above 4G, in the physical address space... -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Kenneth Wayne Culver wrote: Also, the PIII CAN'T natively support more than 4GB of ram. If a particular PIII motherboard supports this, then it's using some kind of wierd chipset that allows this to happen. 4GB is the limit with a 32 bit chip I believe; and the PIII is a 32-bit chip. The Xeon series have 32 bits of virtual address space and 36 bits of physical address space. Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Charles Randall wrote: From: Terry Lambert [mailto:[EMAIL PROTECTED]] I have yet to see one person using it for anything. So far, it is nothing more than marketing fodder: I haven't seen one motherboard capable of more than 4G worth of SIMMs. The Dell PowerEdge 6450 supports 8 GB of RAM. http://www.dell.com/us/en/biz/products/model_pedge_pedge_6400.htm If I understand your comments in a few follow-up messages correctly you're saying that this effort may be better spent by working on an IA-64 port and making it support large memory configurations? The IA64 intrinsically supports a physical address space of 2^64, so an address space of 2^36 vs. 2^32 is spectacularly unimpressive. Can you elaborate? Yeah, the overhead in doing this will up the CPU utilization to the point where it becomes fairly useless to do the swapping to and from above 4G, vs. just swapping normally. The costs involved in doing DMA to/from the memory region above 4G will be incredible, unless the address space is both exported, and known, to the PCI bus; even then, it could only work for 64 bit cards, since 32 bith cards will only be able to address the first 4G of physical memory. I can think of one or two uses for the memory, assuming the ability to DMA into and out of it with a 64 bit card, and the ability to shove a 1G or 2G window around in it so the kernel can get at the memory when it needs to, but the overhead seems to me to be high enough that you are better off buying a Sibytes card, running NetBSD on the MIPS processors on the thing, plugging in 16G of RAM, and calling your PC a control processor. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, 2 Aug 2001, Kenneth Wayne Culver wrote: Replying above an email because the curser is there is like shitting in your pants because your ass is there when you need to go to the toilet. BUT, don't the motherboards also have to support this? And isn't it only supported through some wierd segmentation thing? Yes, the mainboard needs to support the memory. No, there is no weird segmentation thing, at least not visible from software. regards, Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
Rik van Riel wrote: This is a trivial implementation. I'm not very impressed. Personally, I'm not interested in a huge user space, Maybe not you, but I bet the database and scientific computing people will be interested in having 64 GB memory support in this simple way. You mean 4G, of course, since the process address space remains limites to 32 bits... Fully populating both the transmit and receive windows for 1M connections is 32G of RAM, right there... and it better be kernel RAM, or you're screwed. Well, you _could_ store this memory in files, which get mapped and unmapped by the same code the filesystem code uses to access file data in non-kernel-mapped RAM. *runs like hell* That's the entire problem: it has to be performant, or I'm just not interested in it. Using the memory as a software L3 would make a lot more sense to me... a 3G user space is pretty useless, from my point of view, and I'd much rather spend the space on the kernel. Cutting that to 2G/2G might be OK, with 1G in the user used for mapping regions in and out. You are still limited to how much RAM you have, but at least you aren't shooting yourself in the foot trying to make it work. You still haven't told me what Linux does for 2x4G processes and a 1G kernel with only 8G of physical RAM. I rather suspect that as soon as your usage exceeds real memory, it all goes to hell very quickly, since your L1 and L2 caches are effectively disabled by the frequent reloading of CR3 and CR4 on context switches... -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Fri, 3 Aug 2001, Terry Lambert wrote: You still haven't told me what Linux does for 2x4G processes and a 1G kernel with only 8G of physical RAM. I rather suspect that as soon as your usage exceeds real memory, it all goes to hell very quickly, since your L1 and L2 caches are effectively disabled by the frequent reloading of CR3 and CR4 on context switches... Page tables can point to pages all over memory, there is no need to use stupid segment tricks for anything. regards, Rik -- Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://distro.conectiva.com/ Send all your spam to [EMAIL PROTECTED] (spam digging piggy) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
You should format your messages in ascii to send to this list. I know PIII can support 64G physical memory. In FreeBSD how can I visit such range memory(4G-64G) ? You can't. Those memory ranges are strictly off-limits to non-US citizens. -- ... every activity meets with opposition, everyone who acts has his rivals and unfortunately opponents also. But not because people want to be opponents, rather because the tasks and relationships force people to take different points of view. [Dr. Fritz Todt] V I C T O R Y N O T V E N G E A N C E To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Wed, 1 Aug 2001, Terry Lambert wrote: craig wrote: I know PIII can support 64G physical memory. In FreeBSD how can I visit such range memory(4G-64G) ? The short answer is you can't. The longer answer is that you end up having to window it using segmentation; Only if you want to use it all within one process. You can have multiple 2 GB (that's the maximum process size in FreeBSD, right?) programs at the same time, happily using all physical memory. Only the FreeBSD memory management subsystem doesn't support it (yet?). This basically means that the memory is useless as a DMA target or source for disk controllers or gigabit ethernet cards, and is pretty useless for swap, ... So for limited uses in data intensive applications, it might be usable, And for those data intensive applications, it is very useful indeed... But to directly answer your question: by rewriting much of the low core virtual memory and page mapping handling code to know about segmentation. Not just that. There is a more insidious problem with the FreeBSD VM code and support of huge machines. The part of handling the PAE extended page table format and mapping high memory pages in and out of KVA (kernel virtual address) memory to copy stuff is easy. Problem is that you'll have to fit all of FreeBSD's VM data structures in the 2GB of KVA. This just isn't going to fit with the size the data structures have today ... So in order to support huge memory machines right, you'd have to put a number of FreeBSD's VM data structures on a rather strict diet. regards, Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Wed, 1 Aug 2001, Mike Smith wrote: I know PIII can support 64G physical memory. In FreeBSD how can I visit such range memory(4G-64G) ? You can't. Those memory ranges are strictly off-limits to non-US citizens. And under the DMCA, US citizens aren't allowed to build or distribute PAE36 circumvention devices. Rik -- Executive summary of a recent Microsoft press release: we are concerned about the GNU General Public License (GPL) http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: How to visit physical memory above 4G?
On Thu, Aug 02, 2001 at 08:53:38AM +0800, craig wrote: I know PIII can support 64G physical memory. In FreeBSD how can I visit such range memory(4G-64G) ? You can't, right now. Kris PGP signature
Re: How to visit physical memory above 4G?
craig wrote: I know PIII can support 64G physical memory. In FreeBSD how can I visit such range memory(4G-64G) ? The short answer is you can't. The longer answer is that you end up having to window it using segmentation; if you are familiar with the 4k window on video memory in the TI 99/4A, or the bank select on the 6510 (e.g. the ability to select between 32K of RAM, and 32K of ROM, but not both at the same time, on the Commodore C-64 and the similar arrangement on the C-128, etc.), then you;ll have an idea of how the thing works... assuming you can find a motherboard that can handle it. This basically means that the memory is useless as a DMA target or source for disk controllers or gigabit ethernet cards, and is pretty useless for swap, if you ever have to copy from one section to another (e.g. for IPC, SYSV shared memory, mmap'ed files, VM, or buffer cache, etc.). So for limited uses in data intensive applications, it might be usable, but in general, it's nothing more than a hack so that they can claim to support more than 4G, for some extremely limited definition of support. But to directly answer your question: by rewriting much of the low core virtual memory and page mapping handling code to know about segmentation. Have fun doing this, since by the time you are done, you will probably be able to get IA64 machines for something less than the $7000/unit that you have to pay today, and they will likely have PCI/X, so you have enough bus bandwidth to actually make the RAM halfway usable. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
