Hi Gibran, The scheduler does have some serious issues (I've seen what you are describing). This really became apparent in the release we are working on (1.2) where the new license analysis agent is 20x faster than the one you are using. I've been spending a lot of time on it but the solutions are pretty involved. Right now we are planning on the 1.3 release to be where we fix the scheduler (and other issues).
For now, 1) Are you only interested in license analysis? If so, only ever start the license agent, not all the agents. In other words, choose License Analysis but NOT MIME-type Analysis, Metadata Analysis, or Spec File Analysis. These other agents (especially pkgmettagetta does a LOT of database writes). 2) Since you already have these other agents running (you mention pkgmetagetta), go to the job queue display (Jobs -> Queue -> Details) and delete that job. This is probably the one causing the majority of your problems. 3) Check your Scheduler.conf to make sure it is not trying to run too many simultaneous processes. If you want someone else to look at it, send it to the list. Bob Gobeille On Dec 21, 2009, at 6:12 PM, Furosh One wrote: Hey guys and gal, I'm not sure if this is truly a problem but my collegue uploaded a rather large amount of data to be processed and it appears to have been uploaded and scheduled correctly, however it seems like the server has postgres pegged at around 100% for the majority of the time. I was reading and searching your archives to see if I could find anyone else who had the same issue and only found this: "Error while uploading" on Fri, 22 May 2009 08:56:39 -0700 After looking at some of the areas Bob suggested this guy check I thought I should look into each of those too. >1) your scheduler is in some loop >Do a top and see if fossology_scheduler is using lots of cpu time. This doesn't apply other than it's the Postgres process that is at 100% >2) you need to do a vacuum analyze on your database It seems like the DB is doing a lot being at 100%, could I do this while its at 100%. It has 2 CPU's so (1)=100% and (2)= around 20% utilization. >3) you are memory starved. We seem to be okay on memory, this server currently has 4GB RAM. >4) your postgresql statement timeout is set very low >In postgresql.conf the "statement_timeout" variable Our statement_timeout is #'d commented out, #statement_timeout = 0 Should I change this? >5) your disk performance is poor (perhaps due to other processes) This doesn't seem to be an issue at the moment for us. In addition, the agents seem to spit off a small process like pkgmetagetta or licinspect every once in a while but ends really quick. I also look at the Scheduler / Status and it's updating and showing running on localhost and agents, but never seems to get to the filter_license or bsam_engine parts. Anyway, these questions weren't to me directly but was trying to look for help in your archive and thought I'd send you some info to go on to see if you had any suggestions or recommendation. Please let me know what I could check/do to correct this. I don't know much about DB optimization or administration on the database. -Regards, Gibran <ATT00001..txt> _______________________________________________ fossology mailing list [email protected] http://fossology.org/mailman/listinfo/fossology
