On Aug 10, 2016, at 6:32 PM, Keith Medcalf <kmedc...@dessus.com> wrote: >> You must be talking about PAE, which is an unmitigated hack, in the >> dirtiest sense of that word > > It is not a hack. It is how things work. I do not see where you get the > idea that it is a hack.
Because I know how PAE works, and I have the technical competence to express an informed opinion about it. But if you don’t want to believe me, maybe you’ll believe Linus Torvalds: https://cl4ssic4l.wordpress.com/2011/05/24/linus-torvalds-about-pae/ > non-Windows have supported physical address limits beyond 4 GB as standard > since a very long time (Linux since 2009). Yes, via PAE. If you mean something other than PAE, please give a technical reference to what you are talking about. Like, maybe, a page in an Intel architecture reference manual. Even a Wikipedia link would do. >> As you hint, some OSes allow individual apps to allocate extra RAM via PAE >> — UnixWare was one such — but due to the way PAE works, it can never be >> more than 8 GiB per process at a given time. > > I hinted at no such thing. The original quoted paragraph said "more than 4 > GB of RAM". This has nothing to do with the per-process allocation which is > an artifact of how badly or ill-conceived the Operating System architecture > and the physical implementation of the V:R handling. Whether the machine and > OS can physically address more than 4 GB of physical RAM has nothing > whatsoever to do with the "bitedness" of the OS or the CPU, only the width of > the physical address bus and the translation tables and hardware. You’re describing PAE: https://en.wikipedia.org/wiki/Physical_Address_Extension PAE required extending an IA-32 processor’s normal 32-bit address bus to 36 bits, giving a maximum virtual address space of 64 GiB. PAE does not change the machine code instructions for accessing memory, since that would require recompiling everything to allow 36-bit addresses at the program level. This would require another incompatible Intel instruction set, as different from IA-32 as IA-64 is. If you look at the GCC manuals, you will not find a “PAE mode” flag for giving a binary with 36-bit addresses, because an IA-32 processor doesn’t offer that addressing mode. Such a flag would be on this page in the GCC manual: https://gcc.gnu.org/onlinedocs/gcc-6.1.0/gcc/x86-Options.html Notice that there is no PAE flag, and no -m36 flag. If you give -m64, you get IA-64 code, which won’t run on under a 32-bit kernel, even with PAE enabled. You also won’t find such a compiler flag for Visual C++: https://msdn.microsoft.com/en-us/library/19z1t1wy.aspx Though the OS kernel can use PAE to address more than 4 GiB via a 3-level TLB scheme — as opposed to the 2-level scheme Intel used before PAE — it doesn’t let a single program access more than 4 GiB at any given time. Since this whole thread is about giving a single program — sqlite3.exe — access to more RAM, PAE doesn’t solve the OP’s problem. You can install 32-bit Windows Server 2012 on a PAE-aware box with 16 GiB of RAM and run two instances of 32-bit sqlite3.exe on it, but they will only be able to chew up half the system’s RAM between themselves, no more. >> Linux in particular doesn’t let individual applications use PAE to access >> more than 3 GiB of VM space, with the standard 3/1 user/kernel split. >> Instead, if you have more than 4 GiB of RAM in the machine and are running >> a PAE kernel, it will let you have multiple programs *collectively* using >> more than 4 GiB of VM space. That’s not going the help the OP. > > And yet more of the same. You are much confused between "CPU accessing > physical RAM (the :R part)" and "processes accessing virtual RAM (the V: > part). Virtual memory still doesn’t solve the OP’s problem. You can take a PAE-supporting box with 4 GiB of physical RAM in it, install a PAE-aware OS on it, then configure that OS for 60 GiB of swap to give 64 GiB of total virtual memory space, but the individual programs running under that OS *still* won’t be able to address more than 4 GiB of virtual memory each, because the memory addressing instructions will continue to use 32-bit pointers. All virtual memory does in a situation like this is allow another program to come along and grab up to 4 GiB of virtual memory space itself. That doesn’t help the OP, who is running a single program — sqlite3.exe — and needs it to address more than 4 GiB of memory. It doesn’t matter whether you call it RAM, physical memory, or virtual memory, IA-32 simply doesn’t solve the OP’s problem, with or without PAE. >> Quoting Wikipedia, “…regular application software continues to use >> instructions with 32-bit addresses and (in a flat memory model) is limited >> to 4 gigabytes of virtual address space…no single regular application can >> access [all 64 GiB] simultaneously.” > > And your point is what exactly? We are not discussing per process access to > virtual memory limitations, but rather artificial physical memory access > limitations. I think you might be misunderstanding the quote. Low-level discussions about an OS’s memory addressing limitations are often couched in terms of “virtual memory,” which in that context doesn’t refer to swap space or anything like that. It means the machine’s total addressable memory, both swap and RAM together. If you take my example in the section immediately above, there is only 4 GiB of physical RAM, but 64 GiB of virtual memory. The first program you run on that system won’t be able to access 4 GiB of physical RAM because the OS kernel will have some of that tied up, but it *can* allocate 4 GiB of virtual memory, which may be composed partly of physical RAM and partly of swap space. If, while that first program is running, you run a second program on that machine, it will still get some physical RAM because the OS kernel will swap some of the first program’s pages out, so that both programs will end up with considerably less than 4 GiB of physical RAM, yet both could address 4 GiB of virtual memory at the same time. > You will notice that even Microsoft admits that they limit access to physical > RAM purely so they can make more money, and for no other reason That’s not true. According to Mark Russinovich, they tried enabling PAE on consumer OSes during development, and it caused a bunch of poorly-written hardware drivers to crash, so they gave up on it, rather than try to force their hardware partners to fix them. Microsoft had a much easier problem to solve for the Server class OSes because the scope of hardware is much smaller, and that hardware tends to have much better driver support. If this were simply a money grab, Microsoft could have enabled PAE only for Windows Pro or Enterprise editions, but they didn’t even do that. The open source OSes don’t have that problem because the projects write most of the drivers they use, so they’re not beholden to the hardware providers to fix their drivers for them, as Microsoft is. Much the same is true of OS X, which also shipped with a PAE-enabled kernel, back before they went 64-bit-only, many years ago now. >> I believe the situation is essentially the same on PAE-enabled versions of >> Windows as on Linux. > > Again, you are confused between per process addressing of Virtual Memory and > CPU access of Physical Memory. No, I am not confused. It just isn’t relevant to the topic of this thread that the CPU can access > 4 GiB if individual processes cannot. sqlite3.exe is, after all, a single process. Famously so. >> It is also the case that most machines that shipped with 32-bit Intel >> processors either didn’t have enough slots to allow > 4 GiB of RAM or >> didn’t have BIOS/EFI/chipset support for that much RAM if you did have the >> slots for it. And why should they have done? It just adds cost with a >> low chance that the user can make use of it, so that capability only >> showed up in high-dollar machines. > > Nor did cheap motherboards wire the bus master request pin properly to the > expansion bus even though the correct wiring was part of the "IBM compatible" > specification. Just because someone may choose to purchase low-end product > does not mean that the capabilities for bus mastering (or in the case of the > present conversation, accessing more than 4 GB of physical memory) does not > validate your argument. Okay, let’s say PAE or some other technology you are imagining is a solution to the OP’s problem. How is it then irrelevant if it turns out that a huge chunk of the installed base cannot use it? You can’t expect a lot of effort to go towards supporting technology that only a tiny slice of the SQLite user base can use. >> PAE’s time is long past. 64-bit is the proper solution today. > > What does the size of the data bus and registers have to do with the amount > of addressable memory, either physical or virtual? There are two-bit (bit > slice) processors on the market for the past 30 years which have > exceptionally huge addressable storage (far beyond 4 GB). …which has nothing to do with the way the IA-32 instruction set works. _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
