ok this is my setting, a tracker called colo4: listener_jobs = 35 delete_jobs = 0 replicate_jobs = 0 min_free_space = 25000 monitor_jobs = 1 mog_root = /var/mogilefs reaper_jobs = 1 debug = 3
another one called colo2: listener_jobs = 25 min_free_space = 25000 delete_jobs = 0 replicate_jobs = 0 monitor_jobs = 1 mog_root = /var/mogilefs reaper_jobs = 1 debug = 3 and the last one called colo3 queryworker_jobs = 0 listener_jobs = 0 delete_jobs = 0 replicate_jobs = N min_free_space = 25000 monitor_jobs = 1 mog_root = /var/mogilefs reaper_jobs = 1 debug = 3 I've just tested it in the morning non-busy time, I set N to 10 and let it run for 5 minutes. Then tried to inject file, I got: colo4:/home/www# perl script/inject.pl "tempf" "tempfile" t15 /tmp/aaa 1 MogileFS::Backend: tracker socket never became readable (local2:7001) when sending command: [create_close domain=tempf&fid=3462585&devid=16&path= http://192.168.11.4:7500/dev16/0/003/462/0003462585.fid&size=10000&key=t15 ] at /usr/local/share/perl/5.8.8/MogileFS/NewHTTPFile.pm line 335 0colo4:/home/www# perl script/inject.pl "tempf" "tempfile" t15 /tmp/aaa 1 MogileFS::Backend: tracker socket never became readable (local4:7001) when sending command: [create_close domain=tempf&fid=3462595&devid=18&path= http://192.168.11.6:7500/dev18/0/003/462/0003462595.fid&size=10000&key=t15 ] at /usr/local/share/perl/5.8.8/MogileFS/NewHTTPFile.pm line 335 By this time, the load avg of colo2 and colo4 were about 0.5, colo3 were 1.3. They hadn't reach a swap yet. Database hadn't reach the swap too, its avg was about 1.0 and I tested some queries on it, the database was still fast. There were no problem with get_paths, it's 100% success. So, what I'm trying to find out is why replication running on only colo3 affected other trackers on injection while the DB did not seemed to be overloaded. Also noticed that deletion has the similar symptom. And in the busy-time, if I enabled just one replication or deletion job, I usually see some failures on inject. In the past, I had tried to set up more tracker on other machine with the similar setting as colo4 and colo2, but it didn't help. btw, these trackers are running on it's own physical machine, database is dedicated. Each one of tracker machine also run mogstored for talking with tracker and a lighttpd for serving customers. -thank you kem 2008/6/25 dormando <[EMAIL PROTECTED]>: > Exactly how many jobs of each are you running? > > Some jobs don't scale as well as others. Running more of them would > increase database load... > > Are you properly monitoring your database server? Does it end up in > swap, are you high on IO usage already? > > The mogilefs clients don't have a very forgiving timeout, so if a > tracker is in swap it'll be unlikely to ever finish it works. However > even if IO is loaded on a machine, the trackers usually respond in > decent time... > > Are you running the trackers on the database? Out of CPU? Is your > database actually under any load? > > There are no transactions in use in mogilefs... So your theory isn't > likely. The amount of stuff in the queue also doesn't have relation to > load. At least not in any mogilefs service I maintain... I can insert > 20+ million fids into file_to_replicate and it'll work fine. It'll be > annoyed and that's a very stupid thing to do, but it won't overload > anything by nature of there being work to do. > > What version of mogilefs are you running? 2.17? Latest trunk? > > -Dormando > > Komtanoo Pinpimai wrote: > > Hello, > > > > My env is: > > 2 trackers running plenty of querywork,listener,monitor,reaper jobs. > > 1 tracker running only a few delete and replicate jobs. > > > > I guess, I didn't have this problem in the old version of > > MogileFS(version 1??). > > What's happening is when a hard drive has gone bad and I mark it as dead, > > it will create tons of replication jobs. If I have 5 simultaneous > > replication jobs in a tracker, > > it will be really hard to inject one file into the system(with tracker > > busy). Or I have only 1 replication job > > and it's in customer busy time, injecting file usually fails. > > > > What's annoying is no matter how many trackers or querywork/listener > > jobs I added, > > if there is a few simultaneous replicate jobs with tons of works in > > their queue, > > the whole trackers seem to be busy all the time. I also have a trying > > code, like trying to inject and sleep for 15 times, > > it still does not work very well in this situation. > > > > Since these trackers share the same database, I'm trying to guess, it > > must have something to do with transaction, > > for example, the replication or deletion jobs might create some > > transactions that lock some tables preventing > > other jobs from injecting files. And when there are tons of > > replication/deletion jobs in the line, the whole the system > > will turn into readonly mode. > > > > Have you ever experienced this and how do you deal with it ? Is there a > > way to tell mogilefs not to use transaction but still work _ pretty > > correct _? > > > > -- > > I'm going to stop checking email. > > Let's talk in my Hi5. > > -- I'm going to stop checking email. Let's talk in my Hi5.
