First, thanks for taking the time. Now, MongoDB is a cache engine? I think it's a DB?
I need a cache engine that can handle (some) objects bigger then 1MB. I don't have binary data. All I have is ascii elements. My final product is an aggregation of meta data of many binary code pieces. Just for context - my company has many products while each may have several hardware versions. Each version has it's own binary image. If you have one of our products and you need to upgrade the software you get the binary from on place (I have nothing to do with it) and a build meta data from another place. It's a scheme to have only the legitimate users able to upgrade. My application builds or assembles all the various products builds meta information and download it to the distribution cluster. Currently my application takes forever because everything needs to be read from the DB. Caching it will do wonders. I'm sure about that. Profiling my application I found that the total time spent on DB reading is hugh. Do you see any problem using memcached? Thanks, -Michael On Sep 20, 10:18 pm, Joseph Engo <[email protected]> wrote: > If you need larger than 1MB objects and you are only serving these objects > to 5 users at a time. It really sounds like memcache is the wrong tool for > your project. You might want to look into something like MongoDB which has > a larger object limit of 4MB. There are a number of key value stores that > can handle even larger size objects. > > Could you explain a little more on what type of binary data you > are manipulating ? > > > > On Mon, Sep 20, 2010 at 8:25 PM, MikeG. <[email protected]> wrote: > > Hi, > > > I'm starting a project in which I would like to have the entire DB in > > cache. > > The reason is that my transactions are reading large amount of data > > from the DB to generate > > a deliverable blob. > > > I have no concurrency issues. I will always have around 5 users max > > concurrently and most > > of the time it will be single user. SO I look at the context as > > virtually single user. > > > To avoid this massive DB reading (of large count of large chunks) I > > would like to have it > > permanently in memory. > > When any modification of a record happened it is not being written to > > the DB until the > > big blob final product is delivered and the local system goes idle. > > Only then modifications are written to the DB. Upon successful > > completion of DB update the local system sends a > > message to the recipient of the product to inform it that the DB is > > now in sync with the product > > at hand and it can be consumed. > > > Now, to hold 1TB in memory I need a cluster and not a small one. I > > have decided to use > > memory mapped files such that my RAM is virtual memory. Easy to get > > large file system of > > several TBs. > > > My question - is there any limit memcached has as far as cache size? > > > Also, memcached (the C implementation) has a 1MB record size limit. 1) > > What's the reason > > for that? 2) Can it be changed (with a hacked private version) 3) Does > > Jmemcached has the > > same limit? > > > Thanks, > > -Michael
