Re: [FOSSology] Something weird with buckets/pkgagent agent with src.rpm
Thanks for reporting the initial errors and verifying the fixes. Mary -Original Message- From: fossology-boun...@fossology.org [mailto:fossology- boun...@fossology.org] On Behalf Of Raino Lintulampi Sent: Tuesday, December 07, 2010 12:50 AM To: Gobeille, Robert Cc: fossology@fossology.org Subject: Re: [FOSSology] Something weird with buckets/pkgagent agent with src.rpm Hi Bob, There were two issues: - some src.rpm packages were not unpacked correctly by ununpack. This is corrected in a new ununpack agent version, - there was an error in escape string handling in pkgagent which is also corrected in a newer versions. After making those changes to 1.2.1 and uploading distro again everything went OK. Thanks Raino ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Something weird with buckets/pkgagent agent with src.rpm
1 item/sec is not normal for pkgagent. I just ran an RHEL DVD with 2.5 M files. Pkgagent ran to completion in 1 minute (found 4,110 packages). Is the number of items in Job Queue: Package Scan the number of rpm packages? If so then there is something really wrong because the actual number of rpm packages is less than 2,000 packages but the Package Scan job queue shows over 200,000 items and still running (after the weekend!). Did you look at the fossology, postgres and apache log files for clues? fossology.log does not show any errors or debug infor. No errors in apache log files. However, postgres.log has thousands of lines of logs like: EST STATEMENT: SELECT parent_fk,folder_name FROM foldercontents INNER JOIN folder ON foldercontents.parent_fk = folder.folder_pk AND foldercontents.foldercontents_mode = 1 WHERE foldercontents.child_id = $1 LIMIT 1; EST ERROR: prepared statement DirGetNonArtifact_exported already exists EST STATEMENT: SELECT * FROM uploadtree LEFT JOIN pfile ON pfile_pk = pfile_fk WHERE parent = $1; EST ERROR: prepared statement FolderGetFromUpload_1_exported already exists EST STATEMENT: SELECT parent_fk,folder_name FROM foldercontents INNER JOIN folder ON foldercontents.parent_fk = folder.folder_pk AND foldercontents.foldercontents_mode = 2 WHERE foldercontents.child_id = $1 LIMIT 1; EST ERROR: prepared statement FolderGetFromUpload_2_exported already exists EST STATEMENT: SELECT parent_fk,folder_name FROM foldercontents INNER JOIN folder ON foldercontents.parent_fk = folder.folder_pk AND foldercontents.foldercontents_mode = 1 WHERE foldercontents.child_id = $1 LIMIT 1; EST ERROR: prepared statement FolderGetFromUpload_1_exported already exists As I reported in a separate mail, some rpm packages are not unpacked correctly by 1.2.1. I'm reloading the distro in a test server where I have the latest development version of fossology in order to see if there is any differences between it and 1.2.1. Do you have phpPgAdmin installed? It makes it easy to see what command postgres is working on (the database Processes link). This didn't give me much information. I'll check this later if I'll have the same problem with the svn version. br Raino Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Something weird with buckets/pkgagent agent with src.rpm
Hi Raino, Sorry for such a late response. I was taking a week off and not answering fossology (or HP) email. I'm glad the patched fixed the first problem. 1 item/sec is not normal for pkgagent. I just ran an RHEL DVD with 2.5 M files. Pkgagent ran to completion in 1 minute (found 4,110 packages). Did you look at the fossology, postgres and apache log files for clues? Do you have phpPgAdmin installed? It makes it easy to see what command postgres is working on (the database Processes link). Bob Gobeille On Nov 25, 2010, at 7:15 AM, Raino Lintulampi wrote: Hi, patch removed the error messages, so it solved the original problem. Execution of pkgagent is still slow (avg. 1 item/second) but I guess there is nothing that can be done with that. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Something weird with buckets/pkgagent agent with src.rpm
Hi Bob, I'm pretty sure that the problem is about escaped text, how backslash is handled. I patched EscapeString in 1.2.1 from newer version. However, for some reason the patched pkgagent cannot connect to database. The issue is that the original version is installed from deb package. I compiled the patched agent in the virtual environment from where I copied pkgagent. I have changed PREFIX PREFIX=/usr. I've tried to cross cheks location of all conf files, agents etc and there seem to be no differences between the virtual server and real server. But the new pkgagent cannot connect to database. So obviously there must be some differences how the original 1.2.1 deb package was compiled and how I compile patched 1.2.1 for debian. Can you upload 2010/11/17 Bob Gobeille bob.gobei...@hp.com: On Nov 17, 2010, at 7:58 AM, Raino Lintulampi wrote: doing that extremely slow. After two days pkgagent has handled about 40.000 items. Any idea why the situation is now different (it shouldn't be)? And why the execution of pkgagent is so slow? Hi Raino, I've seen this before but don't recall the fix. There has only been one fix to pkgagent since the 1.2.1 release. This was a fix for improperly escaped text being written to the database. If that's the problem, look in fossology.log and you will see errors from pkgagent. To diagnose: 1) look in fossology.log (/var/log/fossology/fossology.log) for errors. 2) look in the postgresql log file (/var/log/postgresql/postgresql-8.3-main.log - note you may have to replace the version number if you aren't running 8.3) Usually, those will give us a hint or show the error. Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Something weird with buckets/pkgagent agent with src.rpm
gmail sent the previous mail by itself ... I'm pretty sure that the problem is about escaped text, how backslash is handled. I patched EscapeString in 1.2.1 from newer version. However, for some reason the patched pkgagent cannot connect to database. The issue is that the original version is installed from deb package. I compiled the patched agent in the virtual environment from where I copied pkgagent. I have changed PREFIX to PREFIX=/usr which I think should be correct PREFIX. I've tried to cross cheks location of all conf files, agents etc and there seem to be no differences between the virtual server and real server. But the new pkgagent cannot connect to database. So obviously there must be some differences how the original 1.2.1 deb package was compiled and how I compile patched 1.2.1 for debian. What are the differences between makefiles between deb packages and the default Makefiles? What else could cause the db access problem? Thanks Raino 2010/11/17 Bob Gobeille bob.gobei...@hp.com: On Nov 17, 2010, at 7:58 AM, Raino Lintulampi wrote: doing that extremely slow. After two days pkgagent has handled about 40.000 items. Any idea why the situation is now different (it shouldn't be)? And why the execution of pkgagent is so slow? Hi Raino, I've seen this before but don't recall the fix. There has only been one fix to pkgagent since the 1.2.1 release. This was a fix for improperly escaped text being written to the database. If that's the problem, look in fossology.log and you will see errors from pkgagent. To diagnose: 1) look in fossology.log (/var/log/fossology/fossology.log) for errors. 2) look in the postgresql log file (/var/log/postgresql/postgresql-8.3-main.log - note you may have to replace the version number if you aren't running 8.3) Usually, those will give us a hint or show the error. Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
