Re: vmmap: bad software everywhere
i'm sure you could fathom the idea that some people care more about streaming video on their browsers than address randomization, the same way some people care more about speedier local lookups to a stationary sync db than making sure a package has correct @want-lib by trashing the ftp server on every query some of these people may even call the alternative they're not using stupid what does that do? nothing On Sun, Jun 5, 2011 at 9:47 AM, Marc Espie es...@nerim.net wrote: On Sun, Jun 05, 2011 at 09:46:48AM -0400, Nico Kadel-Garcia wrote: On Fri, Jun 3, 2011 at 6:26 PM, Marc Espie es...@nerim.net wrote: On Fri, Jun 03, 2011 at 06:11:31PM -0400, Nico Kadel-Garcia wrote: On Tue, May 31, 2011 at 6:51 AM, Marc Espie es...@nerim.net wrote: How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to work, by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. Or you could just be engaging in an ad hominem attack without actually looking at their implementations and assuming they're not doing it right because they're not you or your favorite platform. But hey, we don't know anyone who'd do *that* in the OpenBSD community. Right? Wrong. An ad hominem attack would require me asserting all this for a fact, which is not what I'm doing. Notice the probably ? it makes all the difference in the world. No, I'm afraid it really doesn't require asserting the truth. To quote from Wikipedia, An ad hominem (Latin: to the man), short for argumentum ad hominem, is an attempt to link the truth of a claim to a negative characteristic or belief of the person advocating it It's what I just did to you, in turn. How's it feel? An example or two would have lent powerful credence to your claim. The fix for mono, which Marc Espie notes in this thread, is a very powerful such indicator. I tend to publish findings early, when I don't have THAT many built examples yet. There's also some teamwork, specifically, I don't personally oversee everything in OpenBSD. Nobody does. But we do notice trends, and do some design work based on that. You can call that ad hominem if you wish, do any kind of rhethoric. For me, putting a probably in front of a working hypothesis is enough to go forward. I expect the facts to be disputed, I don't care much for the rhethoric part o it... I would even venture this is a fundamental activity for us to go forward. If you lose yourself in gruntwork, you don't see the bigger picture. Sometimes, we do have the luxury of saying this is complete shit, it shouldn't work, and then we break bad software. On the other hand, secure by default, runs GENERIC is the other tenet of our culture - reproducible defaults, no need to tinker with configs to get things to work, and also, proceed cautiously, do not invent stupid APIS when we don't need to.
Re: vmmap: bad software everywhere
On Sat, Jun 4, 2011 at 10:39 AM, Marc Espie es...@nerim.net wrote: Well, the official fix for mono is in, from the mono team. Guess what ? Mono uses MAP_32BIT if it's available. From Linux's mmap manpage: MAP_32BIT (since Linux 2.4.20, 2.6) Put the mapping into the first 2 Gigabytes of the process address space. This flag is only supported on x86-64, for 64-bit programs. It was added to allow thread stacks to be allocated somewhere in the first 2GB of memory, so as to improve context-switch performance on some early 64-bit processors. Modern x86-64 processors no longer have this performance prob- lem, so use of this flag is not required on those systems. The MAP_32BIT flag is ignored when MAP_FIXED is set. From my point of view, it certainly looks like MAP_32BIT was only put there to address one specific issue in the lifetime of 64 bit platforms, but that some people got some nifty ideas about how to abuse it, and now it's probably there to stay, since software would break without it... Thanks. That's vastly better than simply impugning other people's code without reading it.
Re: vmmap: bad software everywhere
On Fri, Jun 3, 2011 at 6:26 PM, Marc Espie es...@nerim.net wrote: On Fri, Jun 03, 2011 at 06:11:31PM -0400, Nico Kadel-Garcia wrote: On Tue, May 31, 2011 at 6:51 AM, Marc Espie es...@nerim.net wrote: How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to work, by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. Or you could just be engaging in an ad hominem attack without actually looking at their implementations and assuming they're not doing it right because they're not you or your favorite platform. But hey, we don't know anyone who'd do *that* in the OpenBSD community. Right? Wrong. An ad hominem attack would require me asserting all this for a fact, which is not what I'm doing. Notice the probably ? it makes all the difference in the world. No, I'm afraid it really doesn't require asserting the truth. To quote from Wikipedia, An ad hominem (Latin: to the man), short for argumentum ad hominem, is an attempt to link the truth of a claim to a negative characteristic or belief of the person advocating it It's what I just did to you, in turn. How's it feel? An example or two would have lent powerful credence to your claim. The fix for mono, which Marc Espie notes in this thread, is a very powerful such indicator. Now, up to you. What explanation do you have for those problems not being that visible on other 64 bit platforms ? That is, other than their mmap not being very random and stuff tending to group together near the low end of the address space... I can personally comment on only a few: I see a mix. I've done some software porting from 32-bit to 64-bit in the last few years. (Including a fascinating open source project a few years ago, I know it works well in OpenBSD.) I agree that the poor randomization of memory allocation contribute profoundly to not noticing it in testing. (And how often do you *need* 2 GB of RAM for small applicatoins) For others, they simply set the compilation architecture to be 32-bit to avoid problems (which I've certainly done!) For others, they've relied extensively on the better supported 32-bit Java from Sun. and not dug into the code to fix the issue. (They were solviing a problem, not designing an architecture, and were happy to wait and try to fix it later, despite my personal advice.) The Java projects have been a chronic source of this kind of problem: the lack of stability of the early 64-bit implementations led a lot of Java programmers to insist on 32-bit toolkits, and they've never bothered to upgrade and support both. Java isn't supposed to be architecture dependent, but this belief is fundamentally mistaken when you have to pay attention to resource allocation and optimization. I'm doing other things this year: I'll be very curious if OpenJDK is good enough in 64-bit land to help improve performance and testing there and reduce this behavior. Waiting for a good explanation here. Do you think we make problems just to make other OSes look bad ? No, but the snarking is sometimes tedious. It goes both ways. No, we encounter problems, and we do our best to fix them. In my personal experience, most of the time, those problems are bugs in apparently portable software. In the case at hand, we already fixed several issues with some programs. All of them related to some very dirty assumptions about memory... *GOOD*. My last notable migration project involved well-written C. It partly needed shifting between the older use of int and nailing it down as int32_t or other well-defined structures as necessary, to avoid confusion when compiled on 64-bit. But it also needed considerable attention to inherent assumptions about its own, unique databases, and that kind of time and attention is expensive.
Re: vmmap: bad software everywhere
On Sun, Jun 05, 2011 at 09:46:48AM -0400, Nico Kadel-Garcia wrote: On Fri, Jun 3, 2011 at 6:26 PM, Marc Espie es...@nerim.net wrote: On Fri, Jun 03, 2011 at 06:11:31PM -0400, Nico Kadel-Garcia wrote: On Tue, May 31, 2011 at 6:51 AM, Marc Espie es...@nerim.net wrote: How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to work, by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. Or you could just be engaging in an ad hominem attack without actually looking at their implementations and assuming they're not doing it right because they're not you or your favorite platform. But hey, we don't know anyone who'd do *that* in the OpenBSD community. Right? Wrong. An ad hominem attack would require me asserting all this for a fact, which is not what I'm doing. Notice the probably ? it makes all the difference in the world. No, I'm afraid it really doesn't require asserting the truth. To quote from Wikipedia, An ad hominem (Latin: to the man), short for argumentum ad hominem, is an attempt to link the truth of a claim to a negative characteristic or belief of the person advocating it It's what I just did to you, in turn. How's it feel? An example or two would have lent powerful credence to your claim. The fix for mono, which Marc Espie notes in this thread, is a very powerful such indicator. I tend to publish findings early, when I don't have THAT many built examples yet. There's also some teamwork, specifically, I don't personally oversee everything in OpenBSD. Nobody does. But we do notice trends, and do some design work based on that. You can call that ad hominem if you wish, do any kind of rhethoric. For me, putting a probably in front of a working hypothesis is enough to go forward. I expect the facts to be disputed, I don't care much for the rhethoric part o it... I would even venture this is a fundamental activity for us to go forward. If you lose yourself in gruntwork, you don't see the bigger picture. Sometimes, we do have the luxury of saying this is complete shit, it shouldn't work, and then we break bad software. On the other hand, secure by default, runs GENERIC is the other tenet of our culture - reproducible defaults, no need to tinker with configs to get things to work, and also, proceed cautiously, do not invent stupid APIS when we don't need to.
Re: vmmap: bad software everywhere
On 4 June 2011 08:48, Amit Kulkarni amitk...@gmail.com wrote: How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to work, by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. Or you could just be engaging in an ad hominem attack without actually looking at their implementations and assuming they're not doing it right because they're not you or your favorite platform. But hey, we don't know anyone who'd do *that* in the OpenBSD community. Right? This is baiting. There might have been instances of attacks in the past, I don't know. But in this particular case, Marc is absolutely right. OpenBSD is late to the bigmem party but when they get there, they try and raise issues which benefit everybody. Tortoise / Hare. Could it be that all the hares are partying just before the finish line which none of them yet bothered to cross and none of them have noticed that the careful tortoise has cautiously made his way past their drunken fluffy arses and is crossing the line to take the win? : - )
Re: vmmap: bad software everywhere
Well, the official fix for mono is in, from the mono team. Guess what ? Mono uses MAP_32BIT if it's available. From Linux's mmap manpage: MAP_32BIT (since Linux 2.4.20, 2.6) Put the mapping into the first 2 Gigabytes of the process address space. This flag is only supported on x86-64, for 64-bit programs. It was added to allow thread stacks to be allocated somewhere in the first 2GB of memory, so as to improve context-switch performance on some early 64-bit processors. Modern x86-64 processors no longer have this performance prob- lem, so use of this flag is not required on those systems. The MAP_32BIT flag is ignored when MAP_FIXED is set. From my point of view, it certainly looks like MAP_32BIT was only put there to address one specific issue in the lifetime of 64 bit platforms, but that some people got some nifty ideas about how to abuse it, and now it's probably there to stay, since software would break without it... And yeah, if MAP_32BIT is not available, you end up in a jungle of #ifdefs... from what I see of the patch, it looks like the problem could affect any 64 bits arch without MAP_32BIT. We're probably just lucky to hit it first. Let's stress out how MAP_32BIT is not such a good idea, since you constrain the allocations into a much smaller address space, thus less random addresses, more opportunities for buffer overflow to be exploited. (but hey, we all know that buffer overflows don't ever happen under Linux, which is why they don't need no shoddy strlcpy). Differing opinions are welcome, but please, can you try to back it up with facts ?
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 6:51 AM, Marc Espie es...@nerim.net wrote: How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to work, by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. Or you could just be engaging in an ad hominem attack without actually looking at their implementations and assuming they're not doing it right because they're not you or your favorite platform. But hey, we don't know anyone who'd do *that* in the OpenBSD community. Right?
Re: vmmap: bad software everywhere
On Fri, Jun 03, 2011 at 06:11:31PM -0400, Nico Kadel-Garcia wrote: On Tue, May 31, 2011 at 6:51 AM, Marc Espie es...@nerim.net wrote: How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to work, by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. Or you could just be engaging in an ad hominem attack without actually looking at their implementations and assuming they're not doing it right because they're not you or your favorite platform. But hey, we don't know anyone who'd do *that* in the OpenBSD community. Right? Wrong. An ad hominem attack would require me asserting all this for a fact, which is not what I'm doing. Notice the probably ? it makes all the difference in the world. Now, up to you. What explanation do you have for those problems not being that visible on other 64 bit platforms ? That is, other than their mmap not being very random and stuff tending to group together near the low end of the address space... Waiting for a good explanation here. Do you think we make problems just to make other OSes look bad ? No, we encounter problems, and we do our best to fix them. In my personal experience, most of the time, those problems are bugs in apparently portable software. In the case at hand, we already fixed several issues with some programs. All of them related to some very dirty assumptions about memory...
Re: vmmap: bad software everywhere
How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to work, by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. Or you could just be engaging in an ad hominem attack without actually looking at their implementations and assuming they're not doing it right because they're not you or your favorite platform. But hey, we don't know anyone who'd do *that* in the OpenBSD community. Right? This is baiting. There might have been instances of attacks in the past, I don't know. But in this particular case, Marc is absolutely right. OpenBSD is late to the bigmem party but when they get there, they try and raise issues which benefit everybody.
Re: vmmap: bad software everywhere
On 2011-06-01 04.58, Damien Miller wrote: The recent trend of forking another process for a tab instead of a monolithic single process for the whole browser is a way of extending the time required to clean up this mess? Or there is no relation between them? I cannot look into the heads of the chrome devs. There's no technical reason why the tabs can't run in the same process. No technical reason if you exclude isolating mutually-distrusting data origins from each other. It is similar to the privilege separation we do in most OpenBSD network-facing daemons - it is pretty much the only way to do sandboxing on Unix. Yes, I was about to make that comment as well. Not only is there (perhaps) some security to be won that way, but also a more crash-resilient product. I often find myself having a couple of dozen browser windows, each with a dozen or more tabs, and whenever one of those 100+ sites does something that confuses the browser enough for it to crash, it brings down the whole house of cards with it. That is in fact the one advantage Chrome has that appeals to me - when it crashes it (most of the time) only brings down that one window/tab. (On the other hand, that it crashes in the first place is of course not a sign of a solid, secure, well made piece of software... oh well, that's modern software design for you.) Regards, /Benny -- internetlabbet.se / work: +46 8 551 124 80 / Words must Benny LC6fgren/ mobile: +46 70 718 11 90 / be weighed, / fax:+46 8 551 124 89/not counted. /email: benny -at- internetlabbet.se
Re: vmmap: bad software everywhere
@ Corey clinge...@gmail.com wrote (2011-06-01 04:39+0200): I mean, come on -- storing data in unused bits in a pointer? Even I know that's a bad idea. But really, there are user-space memory pools which align on 8 or 16 byte boundaries, so here you have at least three perfectly fine bits. That's at least sometimes much much better than growing a structure by that very size for one bit or two. Super object. Washing machine. (Ooh - upper bounds are beyond my scope.) -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments
Re: vmmap: bad software everywhere
On Wed, Jun 01, 2011 at 01:48:03PM +0200, Steffen Daode Nurpmeso wrote: @ Corey clinge...@gmail.com wrote (2011-06-01 04:39+0200): I mean, come on -- storing data in unused bits in a pointer? Even I know that's a bad idea. But really, there are user-space memory pools which align on 8 or 16 byte boundaries, so here you have at least three perfectly fine bits. That's at least sometimes much much better than growing a structure by that very size for one bit or two. Super object. Washing machine. (Ooh - upper bounds are beyond my scope.) Storing tag bits in the lower bits of a pointer can be ok indeed if you know things about alignment restrictions. Using higher bits because they *seem* unused will bite you sooner or later. -Otto
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 11:51 PM, bofh goodb...@gmail.com wrote: This is interesting. I would really appreciate it very much if you don't mind elaborating a bit more for a non-programmer? Thanks! The general idea which I believe is used by JS engines is to notice that javascript has two datatypes, floating point doubles and everything else (string, array, ...). A NaN is represented as the high 13 bits set to 1, meaning the bottom 51 can be used to store something else. Like a pointer to a string or array, plus some bits for type information and garbage collection. This makes your math code fast because it doesn't need to follow a pointer every time you add two numbers. It causes trouble because if you're lazy, you just assume all pointers (up to 64 bits) will fit in the limited space (maybe 44-48 bits), and then things break. Or you request the OS only give you low addresses (MAP_32BIT). Or you allocate a big block and manage it yourself, so you know all the pointers are close together. The technique is an old one. Usually, you'd store pointers, but reserve the low bit to mark integers. This has the effect of only giving you 31 bit integers. Javascript is based around doubles, so they did things in reverse and store doubles, but reserve special NaN patterns for pointers. I was first made aware of the NaN technique because it's used by luajit.
Re: vmmap: bad software everywhere
On 1 June 2011 00:13, Theo de Raadt dera...@cvs.openbsd.org wrote: On Tue, May 31, 2011 at 10:39 PM, Corey clinge...@gmail.com wrote: data in unused bits in a pointer? Even I know that's a bad idea. Is it really that important to run your Javascript 2% faster? The difference is quite a bit more than 2%. The technique is sound, imo, but it seems the implementations are lacking some safeguards. The implementations are forcing OS developers to remove safeguards from our kernels. google MAP_32BIT Go see what linux and other systems do with the hint argument to mmap(), regarding address space randomization. Go read some of these implementations to see how often they pass a non-NULL hint. I'm so glad we have more performance in the most dangerous applications I wonder if the MAP_32BIT flag was pushed cause it broke apps, or if they were really concerned about the performance implication of having high addresses in amd64. Can't decide which is worst.
Re: Obsolescence engineering (was: vmmap: bad software everywhere)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 01 Jun 2011 04:42:02 + annathemerm...@hush.com wrote: On 31 May 2011 19:51, Otto Moerbeek o...@drijf.net wrote: On Tue, May 31, 2011 at 07:23:46PM +, Christian Weisgerber wrote: Marc Espie es...@nerim.net wrote: Not surprisingly, a lot of software that claims to be 64 bits- ready isn't. This touches all web navigators, most jit engines, and probably lots more of software (our ports tree version of gnu-grep, for instance). I don't think a lot suffers from it, but some prominent cases do. Three problems have been mentioned: (1) Truncation of pointers to 32 bits. Our malloc(3) has returned addresses 4 GB for some time now on amd64 (and before that on other archs like alpha), so I don't expect any new fallout there. I seem to remember that we had a rash of ports fixes back when this first happened on amd64. (2) Tagged pointers. A tagged pointer is when you know that not all the bits in a pointer are used to generate an address and you squeeze some other data into the spare bits. This blocks newer versions of Firefox on sparc64. Mozilla's new JavaScript engine uses tagged pointers and those unused address bits on x86 are actually used on sparc64. (3) The expectation that, no matter what their absolute address, the relative offsets between all your pieces of data fit into 32 bits, i.e., all data is within a 4 GB window. That sounds like a bizarre requirement, but apparently some JIT engines are optimized to rely on this. These are the cases that break with new vmmap. The smart programmers solve number (3) by allocating 2G of memory in advance to store their jit compiled code, so their code can use 32 bit relative offsets. They say, hey, it's only virtual memory, so it doesn't take much resources. Often that is true and it seems a smart idea, but it has the consequence that you lose randomization and protected memory with page size granularity. Or you are forced to do all the memory mangement on your own, basically rewriting the memory management part of the OS in your browser. Suddenly the smart idea does not sound so smart anymore. -Otto But, hey, 64-bit desktop machines have only been around since 1993 or so, and I guess some of the Mozilla programmers weren't born yet when we watched oh-so-clever tagged pointer use blow up at, say, the Motorola 68000 to 68020 transition some 25 years ago. -- Christian naddy Weisgerber na...@mips.inka.de Great. Just absolutely fantastic. These people come up with more and more resource intensive ways of doing the same old computing tasks we've been able to do for a decade or more so that the rest of us have to buy newer, fancier, more expensive machines to do the same things we've been able to do for a decade or more. Of course, for a significant portion of the population, high performance computing means a computer I can access from the convenience of my home, rather than having to spend an hour walking to the library and an hour walking back just so I can sign up and wait an hour or two for the chance to use it for 30 minutes and then rush to do the important things, like fill out job applications for blue collar positions for companies who can't be bothered to take paper applications or check to see if I have any important business e-mail from people who are too annoying to send old-fashioned snail mail. For a lot of people, a computer is like a glorified communications device and typewriter. Except a whole lot more expensive. Hence the usefulness of old computers. When everyone else is rushing to get the latest and greatest, it's often possible to get a sufficiently aged computer for very cheap or even free. Of course, the big corporations don't make as much money if people do that. Which probably explains at least some of the bad software. If we make this new software resource intensive and inefficient enough, then people will have to buy newer, more expensive computers in order to run it. But the older software works just fine? Then we'll just have to stop releasing security patches for it. Good thing we didn't write solid, secure code to begin with. Now the hackers (or crackers, or whatever the correct term is) out there will force the laggards to upgrade to newer more expensive hardware than runs newer more expensive more inefficient software than we still support, and the computer industry goes on! Yay hackers! Well, I can understand that line from corporations looking to earn money, but it makes less sense to hear it from not-for-profits like Linux or Firefox. They say we should all upgrade our computers after three years, five years if we want to push it. What they seem to have missed is that it is a recession. A really bad recession. Goodbye art shows! Hello tent cities! Welcome to the most dangerous town in California: stop laying off cops! And that sort of thing In
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 12:51:27PM +0200, Marc Espie wrote: People not following development too closely may not be aware of it, but we've had a lot of fun with amd64 recently. Specifically, Ariane committed a new vmmap implementation that tends to actually use the 64 bits address space, in userland. She even has some more nasty diff that does its best to put allocations far apart in that address space. This sounds really cool. Thank you Ariane. Not surprisingly, a lot of software that claims to be 64 bits-ready isn't. This touches all web navigators, most jit engines, and probably lots more of software (our ports tree version of gnu-grep, for instance). If firefox or chrome can't fix their stuff in time what are our options for a graphical browser in the upcoming 5.0 release? To be honest I don't need anything fancy with javascript but just _some_ graphical browsers that displays pretty pictures would be nice. In that scenario can we find some old code and beef it up, or could we have our own openbsd web browser built from scratch? Not sure if it's OpenBSD's mission to include a browser but Google and Apple seem to think bundling a browser with their OS's is a must, even if they suck at it. I understand that we as a community have limited resources... and I don't think we should even think of reverting Arianes stuff nor put in that hack that Theo de Raadt asked us to google. Keep up the good work guys and gals, I'd help with coding but my X11 programming is very novice, let's see how time deals our cards eh? cut rest of quoted message Cheers, -peter
Re: vmmap: bad software everywhere
On Wed, 1 Jun 2011 17:49:47 +0200 Peter J. Philipp wrote: Not sure if it's OpenBSD's mission to include a browser but Google and Apple seem to think bundling a browser with their OS's is a must, even if they suck at it. OSS - OpenBSD Secure Surfer or something - Would certainly make the headlines for 5. What a dream but I'm sure they've got more than enough on their plate, atleast for a while (hoping they won't have so much shit to deal with in the future). Can't see it in base ever either.
Re: vmmap: bad software everywhere
Ariane van der Steldt wrote: On Tue, May 31, 2011 at 03:28:11PM -0500, Amit Kulkarni wrote: basically rewriting the memory management part of the OS in your browser. Do some browsers do this on OpenBSD? Googles v8 javascript engine has Pages, Spaces, Heaps and Pagination logic. It even has its own code to map files in. However, the managed to avoid the common pitfall of requiring all platforms to have 4 kB pages: they hardcoded to 8 kB instead. They actually managed to create their own VM inside the browser, running on top of our VM and being almost as complex. Chrome are the ones that use 32-bit relative pointers, I doubt I need to explain how that fails on 64-bit... But hey, it's fast. Webkit allocates 2 GB at startup. Within that area, it will do its own memory management for the JIT. They also use 32-bit relative addressing. Firefox uses the unused bits in your pointer to store some tags there. Because on 64-bit computers we have all these bits, but only 48 are used. So we can store 16 bit worth of data, for free! Ofcourse, your address randomization will be friendly enough to pick memory close to the base of your memory, so of those 48 bits, only 34 are really in use... Why do they need such a trick instead of simply storing tags in a associative array, where key is a pointer and value is a set of tags (or any other arbitrary data)? Lookup against properly aligned array is relatively fast. Am I missing something? Alexey
Re: vmmap: bad software everywhere
@ Otto Moerbeek o...@drijf.net wrote (2011-06-01 14:12+0200): Storing tag bits in the lower bits of a pointer can be ok indeed if you know things about alignment restrictions. Of course it all stands and falls with the quality of the memory allocator! If that sucks your canary's chirp beeps like a mouse in a huge arena! So no - no randomness here. And beside that it tends to get inscrutable when accessing the bits, even if one uses unions to avoid casts. -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments
Re: vmmap: bad software everywhere
@ Alexey Suslikov alexey.susli...@gmail.com wrote (2011-06-01 21:04+0200): Why do they need such a trick instead of simply storing tags in a associative array, where key is a pointer and value is a set of tags (or any other arbitrary data)? Lookup against properly aligned array is relatively fast. Am I missing something? May i answer this. One reason is of course space consumption. If you have 1 objects you need 1 hash nodes and i don't know how many indices the array shall have in this situation. And also, following a pointer chain is poison for CPU cache housekeeping. I.e. in that meaningless example // _area-owner-cbin_table[u.cbin-cbin_index] = u.cbin; you have to wait multiple times for memory to become available. And this is *really* expensive! -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments
vmmap: bad software everywhere
People not following development too closely may not be aware of it, but we've had a lot of fun with amd64 recently. Specifically, Ariane committed a new vmmap implementation that tends to actually use the 64 bits address space, in userland. She even has some more nasty diff that does its best to put allocations far apart in that address space. Not surprisingly, a lot of software that claims to be 64 bits-ready isn't. This touches all web navigators, most jit engines, and probably lots more of software (our ports tree version of gnu-grep, for instance). How comes nobody in other OSes noticed ? Well, people probably did, and tweaked their allocators to work, by using preferably the low address space, and having addresses that increase slowly, so that a lot of pointers are below 4GB, and a lot of pointer diffs are under 4GB. This is yet another example of the patheticness that is modern software development. Instead of going headfront and fixing the actual problems, most systems cope out and just sweep the problem under the carpet, hoping no-one will notice. So, a lot of developers are hard at work figuring the problems, getting the word upstream. In case you're wondering about the stakes, well: 1/ this software will break elsewhere eventually. It's just a question of processing enough data to break thru the 32 bits barrier consistenly. 2/ 64 bits is good for security. When you use the full address range and randomness, exploiting heap buffer overflows becomes much harder. Disclaimer: opinion and message my own, Theo, Ariane, Naddy, Robert will probably chime in, and correct stupid things I've said.
Re: vmmap: bad software everywhere
Marc Espie es...@nerim.net wrote: Not surprisingly, a lot of software that claims to be 64 bits-ready isn't. This touches all web navigators, most jit engines, and probably lots more of software (our ports tree version of gnu-grep, for instance). I don't think a lot suffers from it, but some prominent cases do. Three problems have been mentioned: (1) Truncation of pointers to 32 bits. Our malloc(3) has returned addresses 4 GB for some time now on amd64 (and before that on other archs like alpha), so I don't expect any new fallout there. I seem to remember that we had a rash of ports fixes back when this first happened on amd64. (2) Tagged pointers. A tagged pointer is when you know that not all the bits in a pointer are used to generate an address and you squeeze some other data into the spare bits. This blocks newer versions of Firefox on sparc64. Mozilla's new JavaScript engine uses tagged pointers and those unused address bits on x86 are actually used on sparc64. (3) The expectation that, no matter what their absolute address, the relative offsets between all your pieces of data fit into 32 bits, i.e., all data is within a 4 GB window. That sounds like a bizarre requirement, but apparently some JIT engines are optimized to rely on this. These are the cases that break with new vmmap. But, hey, 64-bit desktop machines have only been around since 1993 or so, and I guess some of the Mozilla programmers weren't born yet when we watched oh-so-clever tagged pointer use blow up at, say, the Motorola 68000 to 68020 transition some 25 years ago. -- Christian naddy Weisgerber na...@mips.inka.de
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 07:23:46PM +, Christian Weisgerber wrote: Marc Espie es...@nerim.net wrote: Not surprisingly, a lot of software that claims to be 64 bits-ready isn't. This touches all web navigators, most jit engines, and probably lots more of software (our ports tree version of gnu-grep, for instance). I don't think a lot suffers from it, but some prominent cases do. Three problems have been mentioned: (1) Truncation of pointers to 32 bits. Our malloc(3) has returned addresses 4 GB for some time now on amd64 (and before that on other archs like alpha), so I don't expect any new fallout there. I seem to remember that we had a rash of ports fixes back when this first happened on amd64. (2) Tagged pointers. A tagged pointer is when you know that not all the bits in a pointer are used to generate an address and you squeeze some other data into the spare bits. This blocks newer versions of Firefox on sparc64. Mozilla's new JavaScript engine uses tagged pointers and those unused address bits on x86 are actually used on sparc64. (3) The expectation that, no matter what their absolute address, the relative offsets between all your pieces of data fit into 32 bits, i.e., all data is within a 4 GB window. That sounds like a bizarre requirement, but apparently some JIT engines are optimized to rely on this. These are the cases that break with new vmmap. The smart programmers solve number (3) by allocating 2G of memory in advance to store their jit compiled code, so their code can use 32 bit relative offsets. They say, hey, it's only virtual memory, so it doesn't take much resources. Often that is true and it seems a smart idea, but it has the consequence that you lose randomization and protected memory with page size granularity. Or you are forced to do all the memory mangement on your own, basically rewriting the memory management part of the OS in your browser. Suddenly the smart idea does not sound so smart anymore. -Otto But, hey, 64-bit desktop machines have only been around since 1993 or so, and I guess some of the Mozilla programmers weren't born yet when we watched oh-so-clever tagged pointer use blow up at, say, the Motorola 68000 to 68020 transition some 25 years ago. -- Christian naddy Weisgerber na...@mips.inka.de
Re: vmmap: bad software everywhere
On Tue, 31 May 2011 21:51:40 +0200 Otto Moerbeek wrote: basically rewriting the memory management part of the OS in your browser. Do some browsers do this on OpenBSD?
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 08:58:29PM +, Kevin Chadwick wrote: On Tue, 31 May 2011 21:51:40 +0200 Otto Moerbeek wrote: basically rewriting the memory management part of the OS in your browser. Do some browsers do this on OpenBSD? webkit tries to do this. -Otto
Re: vmmap: bad software everywhere
basically rewriting the memory management part of the OS in your browser. Do some browsers do this on OpenBSD? Of course they do, otherwise they wouldn't run at all. google for MAP_32BIT. Once you've read enough to get sick to your stomach, please realize that this is just the tip of the iceberg.
Re: vmmap: bad software everywhere
basically rewriting the memory management part of the OS in your browser. Do some browsers do this on OpenBSD? Of course they do, otherwise they wouldn't run at all. google for MAP_32BIT. Once you've read enough to get sick to your stomach, please realize that this is just the tip of the iceberg. I am guessing there's nothing special about OpenBSD, they would probably be doing it everywhere. Then they lie about being truly 64 bit. The recent trend of forking another process for a tab instead of a monolithic single process for the whole browser is a way of extending the time required to clean up this mess? Or there is no relation between them? Wow... didn't know this.
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 03:28:11PM -0500, Amit Kulkarni wrote: basically rewriting the memory management part of the OS in your browser. Do some browsers do this on OpenBSD? Googles v8 javascript engine has Pages, Spaces, Heaps and Pagination logic. It even has its own code to map files in. However, the managed to avoid the common pitfall of requiring all platforms to have 4 kB pages: they hardcoded to 8 kB instead. They actually managed to create their own VM inside the browser, running on top of our VM and being almost as complex. Chrome are the ones that use 32-bit relative pointers, I doubt I need to explain how that fails on 64-bit... But hey, it's fast. Webkit allocates 2 GB at startup. Within that area, it will do its own memory management for the JIT. They also use 32-bit relative addressing. Firefox uses the unused bits in your pointer to store some tags there. Because on 64-bit computers we have all these bits, but only 48 are used. So we can store 16 bit worth of data, for free! Ofcourse, your address randomization will be friendly enough to pick memory close to the base of your memory, so of those 48 bits, only 34 are really in use... Oh, you wouldn't believe how starved for memory we are on those pesky 64-bit computers! Of course they do, otherwise they wouldn't run at all. google for MAP_32BIT. Once you've read enough to get sick to your stomach, please realize that this is just the tip of the iceberg. I am guessing there's nothing special about OpenBSD, they would probably be doing it everywhere. Then they lie about being truly 64 bit. When a developer writes a piece of code and that code breaks stuff, the developer gets blamed. Often rightfully so, but sometimes the code only exposes a bug elsewhere. Linux, windows, apple, they all make their memory management behave in certain ways to make these browsers work. When the OS stops the browser from working, people will complain until the OS stops breaking the browser. The recent trend of forking another process for a tab instead of a monolithic single process for the whole browser is a way of extending the time required to clean up this mess? Or there is no relation between them? I cannot look into the heads of the chrome devs. There's no technical reason why the tabs can't run in the same process. There's even no technical need to have them run in separate threads (although it does simplify some things). The separate processes do mean each JIT will have its personal 2GB, without competing with other tabs. So yeah, it could be related. Wow... didn't know this. I heard this somewhere: A man visits the doctor, saying if I do this, it hurts. To which the doctor replies then don't do that. We now have a situation where we say it hurts. The people behind mozilla, chrome v8, webkit (and so much more software) get to play doctor. Will they fix it or tell us not to do that? -- Ariane
Re: vmmap: bad software everywhere
Googles v8 javascript engine has Pages, Spaces, Heaps and Pagination logic. It even has its own code to map files in. However, the managed to avoid the common pitfall of requiring all platforms to have 4 kB pages: they hardcoded to 8 kB instead. And for those who don't know, OpenBSD has some 16 KB pagesize architectures.
Re: vmmap: bad software everywhere
On 05/31/2011 06:26 PM, Theo de Raadt wrote: Googles v8 javascript engine has Pages, Spaces, Heaps and Pagination logic. It even has its own code to map files in. However, the managed to avoid the common pitfall of requiring all platforms to have 4 kB pages: they hardcoded to 8 kB instead. And for those who don't know, OpenBSD has some 16 KB pagesize architectures. I am no low-level C hacker (I work in industrial automation). But what astounds me about this sort of thing is that the general programming community never seems to learn that taking these kinds of shortcuts eventually bites them, or their users, or the interns they trained before they retired, hard on the buttocks. Y2K anyone? I mean, come on -- storing data in unused bits in a pointer? Even I know that's a bad idea. Is it really that important to run your Javascript 2% faster? I don't use OpenBSD full-time as a desktop, because my principles have been compromised by crack like Flash and high-performance virtual machines. But this sort of thing reminds me why I do dual-boot into it to do my online banking (if but the bank ran it on their servers as well...) Corey
Re: vmmap: bad software everywhere
On Wed, 1 Jun 2011, Ariane van der Steldt wrote: The recent trend of forking another process for a tab instead of a monolithic single process for the whole browser is a way of extending the time required to clean up this mess? Or there is no relation between them? I cannot look into the heads of the chrome devs. There's no technical reason why the tabs can't run in the same process. No technical reason if you exclude isolating mutually-distrusting data origins from each other. It is similar to the privilege separation we do in most OpenBSD network-facing daemons - it is pretty much the only way to do sandboxing on Unix. -d
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 10:39 PM, Corey clinge...@gmail.com wrote: data in unused bits in a pointer? Even I know that's a bad idea. Is it really that important to run your Javascript 2% faster? The difference is quite a bit more than 2%. The technique is sound, imo, but it seems the implementations are lacking some safeguards.
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 10:39 PM, Corey clinge...@gmail.com wrote: data in unused bits in a pointer? Even I know that's a bad idea. Is it really that important to run your Javascript 2% faster? The difference is quite a bit more than 2%. The technique is sound, imo, but it seems the implementations are lacking some safeguards. The implementations are forcing OS developers to remove safeguards from our kernels. google MAP_32BIT Go see what linux and other systems do with the hint argument to mmap(), regarding address space randomization. Go read some of these implementations to see how often they pass a non-NULL hint. I'm so glad we have more performance in the most dangerous applications
Re: vmmap: bad software everywhere
On Tue, May 31, 2011 at 11:03 PM, Ted Unangst ted.unan...@gmail.com wrote: On Tue, May 31, 2011 at 10:39 PM, Corey clinge...@gmail.com wrote: data in unused bits in a pointer? Even I know that's a bad idea. Is it really that important to run your Javascript 2% faster? The difference is quite a bit more than 2%. The technique is sound, imo, but it seems the implementations are lacking some safeguards. This is interesting. I would really appreciate it very much if you don't mind elaborating a bit more for a non-programmer? Thanks! -- http://www.glumbert.com/media/shift http://www.youtube.com/watch?v=tGvHNNOLnCk This officer's men seem to follow him merely out of idle curiosity. -- Sandhurst officer cadet evaluation. Securing an environment of Windows platforms from abuse - external or internal - is akin to trying to install sprinklers in a fireworks factory where smoking on the job is permitted. -- Gene Spafford learn french: http://www.youtube.com/watch?v=30v_g83VHK4
Obsolescence engineering (was: vmmap: bad software everywhere)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 31 May 2011 19:51, Otto Moerbeek o...@drijf.net wrote: On Tue, May 31, 2011 at 07:23:46PM +, Christian Weisgerber wrote: Marc Espie es...@nerim.net wrote: Not surprisingly, a lot of software that claims to be 64 bits- ready isn't. This touches all web navigators, most jit engines, and probably lots more of software (our ports tree version of gnu-grep, for instance). I don't think a lot suffers from it, but some prominent cases do. Three problems have been mentioned: (1) Truncation of pointers to 32 bits. Our malloc(3) has returned addresses 4 GB for some time now on amd64 (and before that on other archs like alpha), so I don't expect any new fallout there. I seem to remember that we had a rash of ports fixes back when this first happened on amd64. (2) Tagged pointers. A tagged pointer is when you know that not all the bits in a pointer are used to generate an address and you squeeze some other data into the spare bits. This blocks newer versions of Firefox on sparc64. Mozilla's new JavaScript engine uses tagged pointers and those unused address bits on x86 are actually used on sparc64. (3) The expectation that, no matter what their absolute address, the relative offsets between all your pieces of data fit into 32 bits, i.e., all data is within a 4 GB window. That sounds like a bizarre requirement, but apparently some JIT engines are optimized to rely on this. These are the cases that break with new vmmap. The smart programmers solve number (3) by allocating 2G of memory in advance to store their jit compiled code, so their code can use 32 bit relative offsets. They say, hey, it's only virtual memory, so it doesn't take much resources. Often that is true and it seems a smart idea, but it has the consequence that you lose randomization and protected memory with page size granularity. Or you are forced to do all the memory mangement on your own, basically rewriting the memory management part of the OS in your browser. Suddenly the smart idea does not sound so smart anymore. -Otto But, hey, 64-bit desktop machines have only been around since 1993 or so, and I guess some of the Mozilla programmers weren't born yet when we watched oh-so-clever tagged pointer use blow up at, say, the Motorola 68000 to 68020 transition some 25 years ago. -- Christian naddy Weisgerber na...@mips.inka.de Great. Just absolutely fantastic. These people come up with more and more resource intensive ways of doing the same old computing tasks we've been able to do for a decade or more so that the rest of us have to buy newer, fancier, more expensive machines to do the same things we've been able to do for a decade or more. Of course, for a significant portion of the population, high performance computing means a computer I can access from the convenience of my home, rather than having to spend an hour walking to the library and an hour walking back just so I can sign up and wait an hour or two for the chance to use it for 30 minutes and then rush to do the important things, like fill out job applications for blue collar positions for companies who can't be bothered to take paper applications or check to see if I have any important business e-mail from people who are too annoying to send old-fashioned snail mail. For a lot of people, a computer is like a glorified communications device and typewriter. Except a whole lot more expensive. Hence the usefulness of old computers. When everyone else is rushing to get the latest and greatest, it's often possible to get a sufficiently aged computer for very cheap or even free. Of course, the big corporations don't make as much money if people do that. Which probably explains at least some of the bad software. If we make this new software resource intensive and inefficient enough, then people will have to buy newer, more expensive computers in order to run it. But the older software works just fine? Then we'll just have to stop releasing security patches for it. Good thing we didn't write solid, secure code to begin with. Now the hackers (or crackers, or whatever the correct term is) out there will force the laggards to upgrade to newer more expensive hardware than runs newer more expensive more inefficient software than we still support, and the computer industry goes on! Yay hackers! Well, I can understand that line from corporations looking to earn money, but it makes less sense to hear it from not-for-profits like Linux or Firefox. They say we should all upgrade our computers after three years, five years if we want to push it. What they seem to have missed is that it is a recession. A really bad recession. Goodbye art shows! Hello tent cities! Welcome to the most dangerous town in California: stop laying off cops! And that sort of thing In other words, lots of people have better things to do with their
