Re: how to python to use virtual memory?
On 26 June 2016 at 04:47, Ho Yeung Lee wrote: > what is the command or code to write to use virtual memory if i use extra > 20 GB from hard disk as memory, means from 70GB memory to 90GB memory > and left 10GB for file? > > Michael Torrie於 2016年6月25日星期六 UTC+8上午11時00分36秒寫道： >> On 06/24/2016 08:44 PM, Dennis Lee Bieber wrote: >> > I don't know how Linux handles swap disk -- Windows normally sets the >> > swap space to ~2X physical memory (for small RAM -- my 12GB system has a >> > 12GB swap and suggests 18GB). >> >> Linux typically uses a user-set swap partition. The old rule of thumb >> was to make the swap partition 2x the size of RAM. Now, though, for most >> installations with lots of RAM, 1:1 is often used. >> >> However, if the OP's program really requires 70 to 100 GB of space, >> relying on the virtual memory system to do this (64-bit only of course) >> is a mistake. The system will simply thrash itself to death on any OS >> at that level of over-commit. If he has that much data, he needs to >> employ techniques for working with the data directly on disk himself. I >> highly doubt these big data sets that large companies work rely simply >> on the OS to manage it! > > -- > https://mail.python.org/mailman/listinfo/python-list You really should avoid relying on the OS trick of pretending you have more ram than you actually do. this can easily cause several magnitudes worth of slowdown. if you succeed i bet the next post to this list will be titled "why is my program taking forever to run". in stead try to partition the work in to chunks and only keep the truly necessary working set loaded at a time. For example if you load data from an xml or json file it will likely include more data than you need to do your calculations. perhaps it is possible to throw some of it away or at least remove it for the time of the memory intensive step. there are ways to allow your program to use more memory ofc. but this may run in to OS limits (like the aforementioned 32 bit windows issue) and will likely incur heavy performance penalties. you should avoid it if at all possible. even if you do you will likely benefit from adapting your algorithm. -- https://mail.python.org/mailman/listinfo/python-list
Re: how to python to use virtual memory?
what is the command or code to write to use virtual memory if i use extra 20 GB from hard disk as memory, means from 70GB memory to 90GB memory and left 10GB for file? Michael Torrie於 2016年6月25日星期六 UTC+8上午11時00分36秒寫道： > On 06/24/2016 08:44 PM, Dennis Lee Bieber wrote: > > I don't know how Linux handles swap disk -- Windows normally sets the > > swap space to ~2X physical memory (for small RAM -- my 12GB system has a > > 12GB swap and suggests 18GB). > > Linux typically uses a user-set swap partition. The old rule of thumb > was to make the swap partition 2x the size of RAM. Now, though, for most > installations with lots of RAM, 1:1 is often used. > > However, if the OP's program really requires 70 to 100 GB of space, > relying on the virtual memory system to do this (64-bit only of course) > is a mistake. The system will simply thrash itself to death on any OS > at that level of over-commit. If he has that much data, he needs to > employ techniques for working with the data directly on disk himself. I > highly doubt these big data sets that large companies work rely simply > on the OS to manage it! -- https://mail.python.org/mailman/listinfo/python-list
Re: how to python to use virtual memory?
On 06/24/2016 08:44 PM, Dennis Lee Bieber wrote: > I don't know how Linux handles swap disk -- Windows normally sets the > swap space to ~2X physical memory (for small RAM -- my 12GB system has a > 12GB swap and suggests 18GB). Linux typically uses a user-set swap partition. The old rule of thumb was to make the swap partition 2x the size of RAM. Now, though, for most installations with lots of RAM, 1:1 is often used. However, if the OP's program really requires 70 to 100 GB of space, relying on the virtual memory system to do this (64-bit only of course) is a mistake. The system will simply thrash itself to death on any OS at that level of over-commit. If he has that much data, he needs to employ techniques for working with the data directly on disk himself. I highly doubt these big data sets that large companies work rely simply on the OS to manage it! -- https://mail.python.org/mailman/listinfo/python-list
how to python to use virtual memory?
after set resource hard and soft both out of real memory 70GB to 100GB and expecting the extra memory setting using virtual memory which is hard disk, the program still wrongly be killed itself after running in background side -- https://mail.python.org/mailman/listinfo/python-list
