RE: Running Accumulo straight from Memory
You guys are confirming what we predicted. This was a like to have from our customer and we wanted to see if anyone else had tried this. Thanks. Matt From: user-return-1336-MATTHEW.J.MOORE=saic@accumulo.apache.org [mailto:user-return-1336-MATTHEW.J.MOORE=saic@accumulo.apache.org] On Behalf Of Adam Fuchs Sent: Wednesday, September 12, 2012 5:28 PM To: user@accumulo.apache.org Subject: Re: Running Accumulo straight from Memory Yes, the effect of locality groups would be about the same in an in memory system. The only exception would be if you're not using locality groups and are fetching a particular column, the automatic seeking behavior of the column filtering iterator would be more efficient with in memory rfiles. Adam On Sep 12, 2012 5:20 PM, David Medinets david.medin...@gmail.com wrote: Why would locality groups be useful in an in-memory system? On Wed, Sep 12, 2012 at 4:53 PM, Adam Fuchs afu...@apache.org wrote: Even if you are just using memory, minor and major compactions are important to get compression, handle deletes, get sequential access (cache line efficiency), use iterators, and introduce locality groups.
RE: Running Accumulo straight from Memory
Adam, It does look like we are the first to try this. We are trying to keep everything in memory and as a result there is no minor compactions, and probably major compactions to make tables larger. We tried this on SSDs using a file system and we were not getting the processing speeds that we had wanted. Matt From: user-return-1330-MATTHEW.J.MOORE=saic@accumulo.apache.org [mailto:user-return-1330-MATTHEW.J.MOORE=saic@accumulo.apache.org] On Behalf Of Adam Fuchs Sent: Tuesday, September 11, 2012 5:30 PM To: user@accumulo.apache.org Subject: Re: Running Accumulo straight from Memory Matthew, I don't know of anyone who has done this, but I believe you could: 1. mount a RAM disk 2. point the hdfs core-site.xml fs.default.name http://fs.default.name/ property to file:/// 3. point the accumulo-site.xml instance.dfs.dir property to a directory on the RAM disk 4. disable the WAL for all tables by setting the accumulo-site.xml table.walog.enabled to false 5. initialize and start up accumulo as you regularly would and cross your fingers Of course, the you may lose data and this is not an officially supported configuration caveats apply. Out of curiosity, what would you be trying to accomplish with this configuration? Adam On Tue, Sep 11, 2012 at 12:02 PM, Moore, Matthew J. matthew.j.mo...@saic.com wrote: Has anyone run Accumulo on a single server straight from memory? Probably using something like a Fusion IO drive. We are trying to use it without using an SSD or any spinning discs. Matthew Moore Systems Engineer SAIC, ISBU Columbia, MD 410-312-2542
Running Accumulo straight from Memory
Has anyone run Accumulo on a single server straight from memory? Probably using something like a Fusion IO drive. We are trying to use it without using an SSD or any spinning discs. Matthew Moore Systems Engineer SAIC, ISBU Columbia, MD 410-312-2542
