Re: Disk based caching for Clojure app
We have been running builds on the same SSDs, doing intensive logging, ... for three years now. None deteriorated. Builds are mainly scrap write thousands of small files plus a few big ones (the targets). Write speed makes a huge difference for this kind of task. Aws allows to get VMs with SSDs of decent sizes and not to store only the OS. I think they would restrict usage if rewrites were a huge concern... Luc P. On Friday, March 6, 2015 at 3:16:09 PM UTC-5, Michael Blume wrote: Possibly stupid question: can you just pretend you have more memory than you do and let the operating system do the heavy lifting? As in, put the swap partition on the SSD and jack up the virtual memory in the OS config? Isn't it a bad idea to put swap on an SSD, because of the limited number of times an SSD byte can be rewritten before it sticks permanently? Thus making SSD more suited to storing stuff where speed counts, but which doesn't change very much, like the operating system kernel and modules and core libraries, plus your most often used applications? Then you get faster boot and app-startup times without constant writes to the SSD (just when something is upgraded). Of course, SSD being less well suited to frequently-written data would also militate against using it for a cache managed by the application, rather than by the OS ... -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Luc Prefontainelprefonta...@softaddicts.ca sent by ibisMail! -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk based caching for Clojure app
The whole SSD fails after X number of writes thing is pretty much a myth now that most drives implement pretty aggressive write leveling. These modern drives introduce a mapping layer between the physical disk locations and the location written to by the OS. This means that writing to KB 4242 on the disk actually is written to a different part of the disk each time it is written. Some recent tests report about 1PB of writes before failure ( http://techreport.com/review/27436/the-ssd-endurance-experiment-two-freaking-petabytes). Let's say your caching system generated and trashed 20GB of data a day, that would leave you with about 130 years before the drive failed. So yes, it's limited but not very limited. On Fri, Mar 6, 2015 at 3:53 PM, Luc Prefontaine lprefonta...@softaddicts.ca wrote: We have been running builds on the same SSDs, doing intensive logging, ... for three years now. None deteriorated. Builds are mainly scrap write thousands of small files plus a few big ones (the targets). Write speed makes a huge difference for this kind of task. Aws allows to get VMs with SSDs of decent sizes and not to store only the OS. I think they would restrict usage if rewrites were a huge concern... Luc P. On Friday, March 6, 2015 at 3:16:09 PM UTC-5, Michael Blume wrote: Possibly stupid question: can you just pretend you have more memory than you do and let the operating system do the heavy lifting? As in, put the swap partition on the SSD and jack up the virtual memory in the OS config? Isn't it a bad idea to put swap on an SSD, because of the limited number of times an SSD byte can be rewritten before it sticks permanently? Thus making SSD more suited to storing stuff where speed counts, but which doesn't change very much, like the operating system kernel and modules and core libraries, plus your most often used applications? Then you get faster boot and app-startup times without constant writes to the SSD (just when something is upgraded). Of course, SSD being less well suited to frequently-written data would also militate against using it for a cache managed by the application, rather than by the OS ... -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Luc Prefontainelprefonta...@softaddicts.ca sent by ibisMail! -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Disk based caching for Clojure app
Hello, I'm been thinking about an idea for a cache layer. It's driven by two trends. Most caches are in memory. They might have fancy additions like multi-machine, but they are in-memory. The fast memory access reduces back end load and improves overall performance. It also assumes you have memory to spare. The second trend is that bootstrapping a start up means minimizing costs. This means paying for one or two low memory machines with at most 2 GB of RAM. Linode, for example, would cost $40/month to have two 2GB nodes (app and db). The two trends don't really overlap well. I want the caching, but I can't yet afford it. Where they *could* overlap is the fact that Linode's drives are SSD. They are not as fast as memory, but probably faster than DB + Network. Is there any pure Clojure library that does this? I'd like to say, Keep 30 MB of Cache in Ram and 2 GB of SSD cache. I guess another way to ask it: is there anything like Java Caching System in pure Clojure? Thanks, JPD -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk based caching for Clojure app
You could build something on top memory mapped files. I did this to solve similar requirements with good effect. On 6 Mar 2015 18:55, JPatrick Davenport virmu...@gmail.com wrote: Hello, I'm been thinking about an idea for a cache layer. It's driven by two trends. Most caches are in memory. They might have fancy additions like multi-machine, but they are in-memory. The fast memory access reduces back end load and improves overall performance. It also assumes you have memory to spare. The second trend is that bootstrapping a start up means minimizing costs. This means paying for one or two low memory machines with at most 2 GB of RAM. Linode, for example, would cost $40/month to have two 2GB nodes (app and db). The two trends don't really overlap well. I want the caching, but I can't yet afford it. Where they *could* overlap is the fact that Linode's drives are SSD. They are not as fast as memory, but probably faster than DB + Network. Is there any pure Clojure library that does this? I'd like to say, Keep 30 MB of Cache in Ram and 2 GB of SSD cache. I guess another way to ask it: is there anything like Java Caching System in pure Clojure? Thanks, JPD -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk based caching for Clojure app
Possibly stupid question: can you just pretend you have more memory than you do and let the operating system do the heavy lifting? On Fri, Mar 6, 2015, 10:54 AM JPatrick Davenport virmu...@gmail.com wrote: Hello, I'm been thinking about an idea for a cache layer. It's driven by two trends. Most caches are in memory. They might have fancy additions like multi-machine, but they are in-memory. The fast memory access reduces back end load and improves overall performance. It also assumes you have memory to spare. The second trend is that bootstrapping a start up means minimizing costs. This means paying for one or two low memory machines with at most 2 GB of RAM. Linode, for example, would cost $40/month to have two 2GB nodes (app and db). The two trends don't really overlap well. I want the caching, but I can't yet afford it. Where they *could* overlap is the fact that Linode's drives are SSD. They are not as fast as memory, but probably faster than DB + Network. Is there any pure Clojure library that does this? I'd like to say, Keep 30 MB of Cache in Ram and 2 GB of SSD cache. I guess another way to ask it: is there anything like Java Caching System in pure Clojure? Thanks, JPD -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk based caching for Clojure app
I'm under the impression that, because of the hard limit on writes, OSes often already cache writes to SSDs, further limiting their usefulness in this kind of application. On Friday, March 6, 2015 at 4:10:54 PM UTC-5, Fluid Dynamics wrote: On Friday, March 6, 2015 at 3:16:09 PM UTC-5, Michael Blume wrote: Possibly stupid question: can you just pretend you have more memory than you do and let the operating system do the heavy lifting? As in, put the swap partition on the SSD and jack up the virtual memory in the OS config? Isn't it a bad idea to put swap on an SSD, because of the limited number of times an SSD byte can be rewritten before it sticks permanently? Thus making SSD more suited to storing stuff where speed counts, but which doesn't change very much, like the operating system kernel and modules and core libraries, plus your most often used applications? Then you get faster boot and app-startup times without constant writes to the SSD (just when something is upgraded). Of course, SSD being less well suited to frequently-written data would also militate against using it for a cache managed by the application, rather than by the OS ... -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Disk based caching for Clojure app
On Friday, March 6, 2015 at 3:16:09 PM UTC-5, Michael Blume wrote: Possibly stupid question: can you just pretend you have more memory than you do and let the operating system do the heavy lifting? As in, put the swap partition on the SSD and jack up the virtual memory in the OS config? Isn't it a bad idea to put swap on an SSD, because of the limited number of times an SSD byte can be rewritten before it sticks permanently? Thus making SSD more suited to storing stuff where speed counts, but which doesn't change very much, like the operating system kernel and modules and core libraries, plus your most often used applications? Then you get faster boot and app-startup times without constant writes to the SSD (just when something is upgraded). Of course, SSD being less well suited to frequently-written data would also militate against using it for a cache managed by the application, rather than by the OS ... -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
