Bob Thanks, It is useful for a different approach in my system, where I need triggered pull replication from the master server. This is something like measurement system from multiple machines. Where each node can be "plugged" to the system. I did several test on my push approach and I am very happy about the results. I was able to run multiple nodes (10 for now) with push continuous replication to the master. All nodes has a local CouchDB server with a database with single document as a process log for this machine being replicated continuously on the master by push replication from the local CouchDB server. I furthermore was able to trigger after each big change a compaction of the master "status" db in a separated thread from the local machine. It work like a magic! No conflicts no problems at all. The size of the master database was always reasonable. I also tried the long-poll handler on the master database and it works very well. I put a simple couchapp to render the changes in a table form and it works as a real monitor over the whole cluster.
Nikolai On 14.03.2011, at 22:12, Robert Johnson wrote: > Nikolai > > You would have a rapidly growing database on your nodes as well because each > document update on each node would generate a new version. > > I offer this:- > > Have a document in your master database with the URLs of the nodes and have a > cron job to read that document and initiate a one time replication with each > node and compress your "master" database (you can use a shell script and > curl to do this). Now, you will need to either multi thread the replication > cron script or in some way test that the host is available before trying the > replication because if the remote host is not available your script will > stall for a good few seconds when trying to contact the remote host and if > you have a lot of hosts and a short(ish) cron cycle you can get cron job > collision. > Have the status collection job on each node compress the database on each > update. > > Bob > > On 14 Mar 2011, at 12:41, Nikolai Teofilov wrote: > >> Hi all, >> >> I need a help on a concept for a monitoring system based on CouchDB. I have >> master server with a database "status" and several, around 20 nodes machines >> (the number of machines will grow) performing some logging on a local >> instances of CouchDB. Each local machine (node) has a database "status" as >> well and logs realtime data in a single document named as the computer. The >> idea is to monitor all the data from all node machines in a single database >> on the master server. The final result should be a database that is a >> collection of documents named as the computer names of all node machines and >> each document reflect the current status of the node. Then it will be easy >> to have long-poll handler for monitoring and rendering the changes of all >> machines on the master server as a status panel... >> >> I made a local db "status" on each node machine and started a continuous >> replication to the master server "status" database. The problem is that some >> of the logging processes on the node machines are working over hours and can >> involve logs each second. This will reflect on the size of the master >> database. How to handle this correctly? I can compact the master target >> database from time to time remotely. Is this a good idea or is there any >> other way to make continuous replication with kind of auto compact feature? >> Would a compacting from time to time will affect somehow the other >> continuous replications? >> >> The second problem I have is once I delete (recreate) the source local >> "status" database the old instance of the continuous replication does not >> work for the new created database. I have a kind of workaround be canceling >> the old continuous replication and triggered it again. This work well but I >> am not sure if I can expect some other problems somehow. >> >> Any suggestions or thoughts? >> >> Thanks >> Nikolai >
