"Differentiated Software Solutions Pvt. Ltd" wrote:
> 3. We have a problem rebuilding this database in the ram.... even say every
> 1000 requests.
What problem are you having with it?
> We tried using dbm and found it a good compromise solution.
> We found that it is about 8 times faster than postgres querying.
Some dbm implementations are faster than others. Depending on your data
size, you may want to try a couple of them.
> 4. Another surprising finding was.... we built a denormalised db on the
> Linux file system itself, by using the directory and file name as the key on
> which we wanted to search. We found that dbm was faster than this.
Did you end up with a large number of files in one directory? When
using the file system in this way, it's a common practice to hash the
key you're using and then split that across multiple directories to
prevent too many files from building up in one and slowing things down.
For example:
"my_key" --> "dHodeifehH" --> /usr/local/data/dH/odeifehH
Also, you could try using mmap for reading the files, or possibly the
Cache::Mmap module.
> We're carrying out more tests to see how scaleable is dbm.
If you're using read-only data, you can leave the dbm handles persistent
between connections. That will speed things up.
You could look at BerkeleyDB, which has a built-in shared memory buffer
and page-level locking.
You could also try IPC::MM, which offers a shared memory hash written in
C with a perl interface.
> Hope these findings are useful to others.
They are. Keep 'em coming.
- Perrin
