Re: [FOSSology] Fossology
It looks like the scheduler is trying to start filter_license. This is a bsam agent that isn't supported in 1.4. So this tells me that your Scheduler.conf file needs to be updated. This might be the cause of the scheduler crashing (and the watchdog repeatedly restarting it). You can use mkschedconf utility to generate a new Scheduler.conf file. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] list licenses on search
Yes, if you don't see the licenses in the normal license browser, then no license were found. This might happen if the jar was not unpacked. If you have a public url for a jar with this behavior, could you send it to me? Thanks, Bob Gobeille On Aug 16, 2011, at 12:34 AM, bob.lan...@smals.be wrote: Hi Bob, I did another test. Your modification lists the licenses found, but not for jar-files. I suppose it's related to the bug that's causing the absence of licenses for jars in normal license browser view as well. regards Bob From:Bob Gobeille bob.gobei...@hp.com To:bob.lan...@smals.be bob.lan...@smals.be Cc:fossology@fossology.org fossology@fossology.org Date:12/08/2011 16:07 Subject:Re: list licenses on search Hi Bob, I see my quick hack has some limitations. I'll fix these in v 2.0, but for now, you have to click on search in the micro menu (the yellow bar when you are browsing a file), not the global Search in the top menu. And you have to Search for: All Files,not the default Containers only. Bob On Aug 12, 2011, at 12:51 AM, bob.lan...@smals.be wrote: Hi Bob, I'm using 1.4.1. Is it the search from the menu or the search when I'm in a certain package? When I do a search through the menu I do get errors in the apache log: [Thu Aug 11 10:00:28 2011] [error] [client 10.1.0.1] PHP Notice: Undefined index: licenses in /usr/share/fossology/www/common/common-dir.php on line 418, referer: http://url/repo/index.php?mod=search_file [Thu Aug 11 10:00:28 2011] [error] [client 10.1.0.1] PHP Notice: Undefined index: licenses in /usr/share/fossology/www/common/common-dir.php on line 411, referer: http://url/repo/index.php?mod=search_file [Fri Aug 12 08:39:44 2011] [error] [client 10.1.0.1] PHP Notice: Undefined variable: ContainerOnly in /usr/share/fossology/www/plugins/search-file.php on line 153, referer: http://url/repo/simpleIndex.php?mod=search_file When I'm in a package and do a search I do not get errors, but no licenses either. ATT1..jpg Overeenkomstig de bepalingen inzake de vertegenwoordiging van de vzw in haar statuten, kan enkel de gedelegeerde bestuurder, de algemeen directeur of zijn uitdrukkelijke lasthebber verbintenissen aangaan namens Smals. Indien dit bericht niet voor u bestemd is, verzoeken wij u dit onmiddellijk aan ons te melden en het bericht te vernietigen. Conformément aux dispositions relatives à la représentation de l'asbl dans ses statuts, seul l'administrateur délégué, le directeur général ou son mandataire exprès est habilité à souscrire des engagements au nom de Smals. Si ce message ne vous est pas destiné, nous vous prions de nous le signaler immédiatement et de détruire le message. According to the provisions regarding representation of the non profit association in its bylaws, only the chief executive officer, the general manager or his explicit agent can enter into engagements on behalf of Smals. If you are not the addressee of this message, we kindly ask you to signal this to us immediately and to delete the message. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] list licenses on search
Hi Bob, I see my quick hack has some limitations. I'll fix these in v 2.0, but for now, you have to click on search in the micro menu (the yellow bar when you are browsing a file), not the global Search in the top menu. And you have to Search for: All Files,not the default Containers only. Bob On Aug 12, 2011, at 12:51 AM, bob.lan...@smals.be wrote: Hi Bob, I'm using 1.4.1. Is it the search from the menu or the search when I'm in a certain package? When I do a search through the menu I do get errors in the apache log: [Thu Aug 11 10:00:28 2011] [error] [client 10.1.0.1] PHP Notice: Undefined index: licenses in /usr/share/fossology/www/common/common-dir.php on line 418, referer: http://url/repo/index.php?mod=search_file [Thu Aug 11 10:00:28 2011] [error] [client 10.1.0.1] PHP Notice: Undefined index: licenses in /usr/share/fossology/www/common/common-dir.php on line 411, referer: http://url/repo/index.php?mod=search_file [Fri Aug 12 08:39:44 2011] [error] [client 10.1.0.1] PHP Notice: Undefined variable: ContainerOnly in /usr/share/fossology/www/plugins/search-file.php on line 153, referer: http://url/repo/simpleIndex.php?mod=search_file When I'm in a package and do a search I do not get errors, but no licenses either. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] list licenses on search
Bob, can you give me the url of what you are trying to scan? Bob On Aug 12, 2011, at 8:07 AM, Gobeille, Robert wrote: Hi Bob, I see my quick hack has some limitations. I'll fix these in v 2.0, but for now, you have to click on search in the micro menu (the yellow bar when you are browsing a file), not the global Search in the top menu. And you have to Search for: All Files,not the default Containers only. Bob On Aug 12, 2011, at 12:51 AM, bob.lan...@smals.be wrote: Hi Bob, I'm using 1.4.1. Is it the search from the menu or the search when I'm in a certain package? When I do a search through the menu I do get errors in the apache log: [Thu Aug 11 10:00:28 2011] [error] [client 10.1.0.1] PHP Notice: Undefined index: licenses in /usr/share/fossology/www/common/common-dir.php on line 418, referer: http://url/repo/index.php?mod=search_file [Thu Aug 11 10:00:28 2011] [error] [client 10.1.0.1] PHP Notice: Undefined index: licenses in /usr/share/fossology/www/common/common-dir.php on line 411, referer: http://url/repo/index.php?mod=search_file [Fri Aug 12 08:39:44 2011] [error] [client 10.1.0.1] PHP Notice: Undefined variable: ContainerOnly in /usr/share/fossology/www/plugins/search-file.php on line 153, referer: http://url/repo/simpleIndex.php?mod=search_file When I'm in a package and do a search I do not get errors, but no licenses either. ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Nomos missing BSD-style license on v1.3.0
Yes, please submit the bug. I just verified the problem on 1.4.1 as well. Thanks Pete. Bob Gobeille On Aug 12, 2011, at 12:35 PM, Peter Jalajas, GigaLock Backup Services wrote: Hi all, Nomos doesn't seem to detect the BSD-style license, pasted below. I'm on FOSSology version 1.3.0 (code revision 4518). Should I submit a bug directly? Thanks, Pete Downloaded from: http://sourceforge.net/projects/keepass/files/KeePass%201.x/1.18/ Folder: Software Repository/ KeePass-1.18-Src/ KeePassLibCpp/ Crypto/ SHA2 Bucket Browser | Copyright/Email/URL | Browse | Nomos License | Nomos License List | Nomos License List Download | Search • View | Info • bsam License | License Tree | License Groups • Refresh Count Files License Name 4 ShowGPL (All 4 files contain the following BSD-style license text.) Folder: Software Repository/ KeePass-1.18-Src/ KeePassLibCpp/ Crypto/ SHA2/ TDefs.h Nomos One-Shot | One-Shot Copyright/Email/URL • Nomos License | View | Info | View Copyright/Email/Url • Hex | Text | Formatted • Reanalyze | bsam License | bsam One-Shot | Recompare • Refresh The Nomos license scanner found: GPL /* --- Copyright (c) 2003, Dr Brian Gladman, Worcester, UK. All rights reserved. LICENSE TERMS The free distribution and use of this software in both source and binary form is allowed (with or without changes) provided that: 1. distributions of this source code include the above copyright notice, this list of conditions and the following disclaimer; 2. distributions in binary form include the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other associated materials; 3. the copyright holder's name is not used to endorse products built using this software without specific written permission. ALTERNATIVELY, provided that this notice is retained in full, this product may be distributed under the terms of the GNU General Public License (GPL), in which case the provisions of the GPL apply INSTEAD OF those given above. DISCLAIMER This software is provided 'as is' with no explicit or implied warranties in respect of its properties, including, but not limited to, correctness and/or fitness for purpose. --- Issue 01/08/2005 ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] list licenses on search
Hi Bob, I should have mentioned that this code replaces files in 1.4.1. Is that what you are running? The error will be in the php log file. By default, I think that is the apache log. Bob On Aug 11, 2011, at 2:03 AM, bob.lan...@smals.be wrote: I followed your instructions but it doesn't seem to work. I restarted the apache, did a flush cache in the fossology interface. From what I can tell, the license is fetched but there's no screen output? ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] list licenses on search
Hi Bob,
I went ahead and made a few quick changes so that when you search, the search
results will display the file licenses. I'll release a cleaned up version of
this in v 2.0 but if you want to test it out I've attached the three modified
files. These go in directory WEBDIR. To see the real WEBDIR path, login to
fossology with admin privileges (user fossy is the default) then on the main
menu choose Help Debug Global Variables. It is probably
/usr/share/fossology/www.
Put the common- files in the common directory, and search.php in the
plugins directory. These will replace existing files so you might want to
move the old versions into another directory in case these not well tested
changes have bugs. You need to move the old files out, not just rename them
because otherwise you will get name collisions.
Bob Gobeille
?php
/***
Copyright (C) 2008 Hewlett-Packard Development Company, L.P.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
version 2 as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
***/
/
These are common functions used by analysis agents.
Analysis Agents should register themselves in the menu structure under the
top-level Agents menu.
Every analysis agent should have a function called AgentAdd() that takes
an Upload_pk and an optional array of dependent agents ids.
Every analysis agent should also have a function called AgentCheck($uploadpk)
that determines if the agent has already been scheduled.
This function should return:
0 = not scheduled
1 = scheduled
2 = completed
/
/*
* NOTE: neal says the name is wrong...should be Analysis...
* TODo: change the name and all users of it.
*/
/**
AgentCheckBoxMake
\brief Generate a checkbox list of available agents.
Only agents that are not already scheduled are added. If
$upload_pk == -1, then list all. User agent preferences will be
checked as long as the agent is not already scheduled.
@return string containing HTML-formatted checkbox list
*/
function AgentCheckBoxMake($upload_pk,$SkipAgent=NULL) {
global $Plugins;
global $DB;
$AgentList = menu_find(Agents,$Depth);
$V = ;
if (!empty($AgentList)) {
// get user agent preferences
$userName = $_SESSION['User'];
$SQL = SELECT user_name, user_agent_list FROM users WHERE
user_name='$userName';;
$uList = $DB-Action($SQL);
// Ulist should never be empty, if it is, something really wrong,
// like the user_agent_list column is missing.
if(empty($uList))
{
$text = _(Fatal! Query Failed getting user_agent_list for user);
return(h3 style='color:red'$text $UserName/h3);
}
$list = explode(',',$uList[0]['user_agent_list']);
foreach($AgentList as $AgentItem) {
$Agent = $Plugins[plugin_find_id($AgentItem-URI)];
if (empty($Agent)) {
continue;
}
if ($Agent-Name == $SkipAgent) {
continue;
}
if ($upload_pk != -1) {
$rc = $Agent-AgentCheck($upload_pk);
}
else {
$rc = 0;
}
if ($rc == 0) {
$Name = htmlentities($Agent-Name);
$Desc = htmlentities($AgentItem-Name);
// display user agent preferences
if(in_array($Name, $list))
{
$Selected = checked ;
}
else
{
$Selected = ;
}
$V .= input type='checkbox' name='Check_$Name' value='1' $Selected /$Descbr /\n;
}
}
}
return($V);
} // AgentCheckBoxMake()
/
AgentCheckBoxDo(): Assume someone called AgentCheckBoxMake() and
submitted the HTML form. Run AgentAdd() for each of the checked agents.
Because input comes from the user, validate that everything is
legitimate.
/
function AgentCheckBoxDo($upload_pk)
{
global $Plugins;
$AgentList = menu_find(Agents,$Depth);
$V = ;
if (!empty($AgentList)) {
foreach($AgentList as $AgentItem) {
/*
The URI below contains the agent name e.g agent_license, this is
not be confused with the Name attribute in the class, for example,
the Name attribute for agent_license is: Schedule License Analysis
*/
$Agent
Re: [FOSSology] Bucket question
On Aug 10, 2011, at 3:20 AM, bob.lan...@smals.be wrote:
Hi Bob,
it would indeed be nice to have a list of jars with the licenses for each
jar.
The way how (via search, buckets or other means) really depends on the amount
of work involved on your side.
I also noticed that in the license browser for files and folders the license
is shown underneath. For jars this is not the case.
That shouldn't be, but I see you are correct. I'll file a bug. Thanks.
When I tried the buckets I found an error in the docs
(http://fossology.org/buckets). In the table bucket_def, for bucket_filename
it should state that Data files are located in
$PROJECTSTATEDIR/bucketpools/{bucketpool_pk}. The bucketpools is missing.
Thanks. I just fixed it.
Bob Gobeille___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Ubuntu 11.04
Hi Dave, I don't use Ubuntu 11.04, but this is a php error. I'd check that your version of php has the PostgreSQL extensions. Bob Gobeille On Aug 10, 2011, at 3:28 PM, Dave McLoughlin wrote: Has anyone tried to install Fossology 1.4.0 on Ubuntu 11.04? I ran into some problems a few weeks ago with some database issues and haven’t had time to get back to it until this week. I just tried a fresh install from source and got the following at the end of fo-postinstall script: *** Checking database connectivity *** NOTE: Connect succeeded, database is working. *** Initializing database tables *** PHP Fatal error: Call to undefined function pg_connect() in /usr/local/share/fossology/www/common/common-db.php on line 51 ERROR: Unable to initialize. Mary L had recommended I check out my postgresql config, probably still the same issue. I’m just wondering if anyone has seen this and has been able to solve it. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] DBaccess(7): ERROR: more than one row returned by a subquery used as an expression
FWIW, this problem with the delete agent is addressed in V 2.0. There is another problem with the delete agent that is a bit more difficult. We store each file separately in the repository, so deleting say an iso with 2M files can take a while. Until/unless we go to a different file storage scheme, perhaps saving the files in large blocks, this problem will be with us. Bob Gobeille On Aug 3, 2011, at 10:57 AM, Laser, Mary wrote: -Original Message- Sent: Tuesday, August 02, 2011 8:22 PM Subject: [FOSSology] DBaccess(7): ERROR: more than one row returned by a subquery used as an expression This is VERY likely the cause. Please do delete the uploads completely and try again. The delete must run to a successful completion before you try and upload again. Otherwise, you run the risk of having 2 uploads in your DB with the same signature. Mary Likely due to trying to delete a prior Upload of the RedHat test file that threw a couple of FAILED agents. Deleting these Uploads and starting again is no big deal, I'm just familiarizing myself with FOSSology. Suggestions welcome! (I'm not afraid of the command line.) ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] License highlighting (Bob Gobeille)
On Jul 29, 2011, at 3:57 PM, Dragoslav Mitrinovic wrote: I'll add my vote for license match highlighting. This was an extremely useful feature in bsam, and for us it is the single most missed feature since the transition to nomos. In fact, while I like nomos for its speed and easier review of results, I wish bsam was not removed and was instead left as an alternative scanning agent. We are still using 1.3.0, where both agents co-exist side-by-side, and we sometimes run scans with both. When we scan with both agents, we rely primarily on nomos scan, but also check bsam results for few particular licenses of high interest to us. At times, bsam would find things that nomos missed, so it's a good complement to nomos. A problem we have with bsam is that we don't have a maintainer. Thats why we don't want to include it in our release. Since nomos is your primary scanner, why don't we work on it so that you don't need bsam? If you find files where nomos misses a license, could you send them to the list (or me or any developer)? Or better yet, go to http://bugs.linux-foundation.org/ and file a bug? One particular type of files that nomos was not stellar with is a class of debian copyright files. Those files tend to be long and list many licenses aggregated from many source files, which is in contrast to single source files with which nomos heuristics were primarily developed. I was going to file this bug for you, but most of the debian copyright files I see are short and only have a single license. Could you file a bug on a specific file? Another issue we ran into with nomos is scanning of native executables. For instance, if you scan GNU tar executable with bSAM and nomos, bSAM will report GPLv3 while nomos will stay silent. GNU tar has an embedded license statement (type tar --version to see it), and bSAM finds this string. Nomos on the other hand expects files to be in a form of a single 0-terminated string, and so regexp search on a binary file will typically terminate prematurely. This was fixed in 1.4.1 http://bugs.linux-foundation.org/show_bug.cgi?id=723 This is a longish post, so I'll summarize briefly: license match highlighting is super-important for us we would love to see bSAM reappear in its 1.3.0 form, even if no further development is planned for it I'd suggest running Unix strings(1) command on native executables prior to passing the data to nomos I think I've got these addressed above. By the way, license match highlighting was important enough for us that I've spent some time studying how nomos works and thinking about how it could be done. I have a very rough proof of concept thing that works about 90% of the time. (By works I mean it gives you some idea about where the license is found in the file). I'd be happy to share the ideas (and code if there is interest). This would be probably better suited for fossology-devel mailing list. Let me know if you are interested, and I could join the list and talk there... You have modified nomos code so the highlighting works 90% of the time? That's excellent. Yes, that's a fossology-devel subject. Doing it yourself is the best way to get what you want. ;-) Please share your code. You didn't mention about adding licenses to nomos. That's something I'd think nobody would be happy with. Thanks! Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] License highlighting (Bob Gobeille)
On Aug 1, 2011, at 10:03 AM, Dragoslav Mitrinovic wrote: Speaking of new licenses, one thing we miss from bSAM are phrases. Thanks to those phrases, bSAM was fairly good at detecting proprietary code, which is important for instance when you are vetting code to be released as OSS. Nomos signatures (STRINGS.in) are phrases.Unfortunately, as you know, it does require you to recompile. We added few new regexps to our version of nomos to catch proprietary code, but I'd need to clean it up before it could be contributed - there are too many false positives still. Feel free to file a bug/enhancement if you want to describe what phrases you want to catch. Make sure you attach sample files. What we do to test new license signatures is first run a baseline nomos from the command line on the 100MB RedHat.tar.gz in http://fossology.org/testing/testFiles/ Then add the license/phrase, rerun, then compare results. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] License highlighting
Hello Volker, The database table license_file is where nomos records what license matched in what file. There are columns for where in the file the match occurred, but they are not currently populated by nomos. Would anyone else like to vote on how important highlighting the license match is? Bob Gobeille On Jul 27, 2011, at 2:49 AM, Mader Volker (AA-DGP/ESD2) wrote: yes, this is exactly what would be most useful. I think it is very important to understand why a license was detected esp. for deciding on false positives. Certainly the signature match is the interesting information there. My plan: In combination with the tags feature to mark false positives or give approval to special file/license pair... it can be used to manage the findings more properly. So of course from my perspective this is quite important compared to most other items ;-) Does nomos store the match information in the DB already? Could we just use some query in the php to display it? ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] SPDX licences in FOSSology
Hi Daniel, FOSSology's license scanner, nomos, isn't SPDX license template compliant and it isn't our current intention to make it so. We only match a few signatures (regex's) found in context (rules). So when we say a file is BSD-3, that means that we are pretty confident that is the intended license, but we don't have the confidence that sentence matching like Ninka would give. We could write signatures and rules that implement the SPDX templates but at this time we don't believe that will give us significantly better results and will cost processing time, not to mention effort. Our signatures do take into consideration alternate spellings and extra words don't necessarily throw us off. Of course, this can be a good thing or a bad thing. Mary's email is trying to get a pulse from fossology users to see if we should give more priority to supporting the SPDX license naming guidelines, add licenses that are currently missing from fossology, and if anyone would like to work on this. An obvious question is why don't we use Ninka in fossology. There are multiple reasons, none of which preclude us from using it in the future, but the big one for us is that we don't have many non-hp contributors. So hp usually gets its way when setting priorities, and nomos is something they are used to and trust. More non-hp contributors would be a good thing for the project but we only have a few and we get surprisingly little feedback given the number of people I've met that use fossology. Bob Gobeille Hewlett Packard Open Source Program Office b...@fossology.org On Jul 22, 2011, at 6:06 PM, D M German wrote: let me start again, I think I made things more complicated they should be... and I made several mistakes below Let us start with a file that is clearly BSD-3. What do we output? - BSD-3 - spdxBSD-3. Probably spdx-BSD3. the problem is, sometimes a file looks to our tools like BSD-3, but it is not according to SPDX. Example: ./drivers/net/wimax/i2400m/usb-tx.c the file includes The sentence that does not match the SPDX version is: -- Neither the name of Intel Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. -- instead of the SPDX approved sentence: -- Neither the name of the ORGANIZATION nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. -- Notice the difference is the before ORGANIZATION. What should we output? - BSD-3? or unknown? In Ninka I defined a new license set with the prefix: spdxBSD-3 A file can be spdxBSD-3, but not BSD-3. I am not sure that is the best decision. --dmg dmg If you follow the SPDX discussion you know that I have been looking dmg into the implementation of SDPX. dmg one of challenges of new SPDX licenses is that they are subsets of dmg what we call licenses. dmg i.e. more strict in their text. dmg For instance, in Ninka we allow some variability in some by dmg inclusion licenses. Like dmg British vs American spellings Iin BSD. SPDX does not. dmg So the challenge I had was to allow the more strict SPDX detection, dmg while allowing the dmg usual detection for those licenses. In other words: dmg I presume something similar would happen to Fossology, where the SPDX licenses dmg would have to be scanned first, and if not-matched, then the more dmg traditional licenses dmg scanned. dmg So say file contains a British spelling version of a couple of words dmg in the BSD-3 clauses. dmg Currently Ninka would output: BSD-3 dmg With SPDX support, what do we output? dmg BSD-3 and spdx-BSD-3? dmg or only BSD-3? dmg alternatively, our license detection could be configured: dmg - for default licenses (output BSD-3) dmg - for spdx support: unknown licence (attach to Appendix) dmg SPDX is going to gives some headaches :) dmg --dmg dmg For a more concrete example: what would this license be: dmg Here is an example of the BSD3 (non-spdx): dmg ./drivers/net/wimax/i2400m/usb-tx.c dmg The sentence that does not match the SPDX version is: dmg Neither the name of Intel Corporation nor the names of its dmg contributors may be used to endorse or promote products derived from dmg this software without specific prior written permission. dmg instead of: dmg Neither the name of the ORGANIZATION nor the names of its dmg contributors may be used to endorse or promote products derived from dmg this software without specific prior written permission. dmg Here
Re: [FOSSology] Deb install on Ubuntu
Hi Mary, How is a user supposed to know this? Was there instructions in the apt-get install? Bob Gobeille On Jul 7, 2011, at 10:20 AM, Laser, Mary wrote: File locations are listed here: http://fossology.org/file_locations The debian install instructions are here: http://fossology.org/debian_install Please refer to the INSTALL file for Apache Config. http://fossology.svn.sourceforge.net/viewvc/fossology/tags/1.4.0~rc7/INSTALL?revision=4199view=markup Mary From: fossology-boun...@fossology.org [mailto:fossology-boun...@fossology.org] On Behalf Of Dave McLoughlin Sent: Thursday, July 07, 2011 10:06 AM To: fossology@fossology.org Subject: [FOSSology] Deb install on Ubuntu Up til now, I’ve only installed fossology from source. Yesterday I went to do a clean install of the latest release 1.4.1 on Ubuntu 11.04. So I downloaded fossology_1.4.1_all.deb (http://fossology.org/debian/ubuntu/maverick/fossology_1.4.1_all.deb). I double clicked on the package and it appears to have installed something. How do I know what it installed, where it installed it and what I need to do next to get fosslogy running? Is there a README associated with the deb package? I checked top –u fossy and it shows that it is running. But when I go to http://localhost/repo/, it appears Apache is not configured. So, it appears there are some additional steps I need to take to complete the install, I’m just not sure what they are. Any assistance would be appreciated. -- Dave McLoughlin | OpenLogic ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Move Folder Contents
Hi Ray, Sorry, I was away for a couple of days. Yes, you can do a bulk move of folder contents with sql: 1) Find the folder primary keys for the source and destination folders with the user interface. For example, when the url says folder=123, then 123 is the folder_pk. For this example, let's say your source folder key is 10 and your destination folder key is 20. 2) Move ALL the uploads the source folder to the destination folder: update foldercontents set parent_fk=20 where parent_fk=10 and foldercontents_mode=2; Bob Gobeille On Jun 23, 2011, at 9:43 AM, Westphal, Raymond W wrote: Thanks Mary. Is there a bulk method? Something like a sql update to a table indicating the new folder? Thanks, Ray W. From: Laser, Mary [mailto:mary.la...@hp.com] Sent: Thursday, June 23, 2011 10:35 AM To: Westphal, Raymond W; 'fossology@fossology.org' Subject: RE: Move Folder Contents Oops! You’re right. I should have said “Organize - Uploads - Move” From: Westphal, Raymond W [mailto:raymond.w.westp...@ehi.com] Sent: Thursday, June 23, 2011 9:34 AM To: Laser, Mary; 'fossology@fossology.org' Subject: RE: Move Folder Contents Oh - I thought that moved the entire folder. Thanks, Ray W. From: Laser, Mary [mailto:mary.la...@hp.com] Sent: Thursday, June 23, 2011 10:29 AM To: Westphal, Raymond W; 'fossology@fossology.org' Subject: RE: Move Folder Contents Yes, use the “Organize - Folders - Move” menu. Mary From: fossology-boun...@fossology.org [mailto:fossology-boun...@fossology.org] On Behalf OfWestphal, Raymond W Sent: Thursday, June 23, 2011 5:30 AM To: 'fossology@fossology.org' Subject: [FOSSology] Move Folder Contents Importance: High Good Morning Everyone. My users would like to know if folder contents can be moved into another folder. They recently created another folder in order to organize and prepare for cleanup of the repository. They now want the remaining uploads to be located in that single new folder. Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Clean Out Repository
Hi Ray, It looks like you found an error in fossjobs, so I filed a bug on your behalf: http://bugs.linux-foundation.org/show_bug.cgi?id=819 You can run delagent without the scheduler from the command line: 0) Find all your upload_pk's to delete (from previous emails) 1) Find out where your agents are installed. In the user interface login as an administrator and click on Help Debug Global Variables The variable AGENTDIR shows you the agent install location. For illustration, let's say AGENTDIR is /usr/lib/fossology/agents. 2) Run delagent from the command line. For example: /usr/lib/fossology/agents/delagent -U 1234 Where 1234 is your upload_pk. You can put that in a script to delete all the uploads. Don't run delagent in parallel or with other agents running in the scheduler. Because of the file sharing that fossology does, you could run into concurrency problems if you do that. If you want to see all the delagent options, then run delagent --help Bob Gobeille On Jun 15, 2011, at 8:28 AM, Westphal, Raymond W wrote: Thanks for the response Bob. I'm sorry Bob; I don't understand how to use the delete agent. When I list the available default agents as fossy or root ... fossrepo]$ fossjobs -a The available agents are: agent_bucket agent_copyright agent_mimetype agent_nomos agent_pkgagent agent_unpack agent_specagent agent_license PHP Notice: Undefined index: U in /usr/bin/fossjobs on line 167 Is this list complete? Thanks, Ray W. -Original Message- From: Bob Gobeille [mailto:bob.gobei...@hp.com] Sent: Tuesday, June 14, 2011 12:27 PM To: Westphal, Raymond W Cc: fossology@fossology.org Subject: Re: [FOSSology] Clean Out Repository Hi Ray, To find the uploads over a year old: select upload_pk from upload where upload_ts (now() - interval '1 year') order by upload_ts You can then take that list and schedule the delete agent on each one of the upload_pk's with the fossjobs command (man fossjobs). Unfortunately, you can't simply delete the upload from the database because that won't delete the files from the repository, and because cascade isn't turned on for most of the foreign keys. So, at least for now, you have to use the delete agent. Bob Gobeille On Jun 14, 2011, at 6:29 AM, Westphal, Raymond W wrote: Hello Everyone. I'm trying to find a way to automatically cleanup the repository. The users will allow me to delete uploads older than 1 year. I created the following query from watching the database while I selected the Organize/Uploads/Delete Uploaded File menu option. SELECT upload_pk, upload_desc, upload_ts, ufile_name FROM foldercontents,uploadtree,upload WHERE foldercontents.parent_fk = '4' AND upload_ts (select current_date -365) AND foldercontents.foldercontents_mode = 2 AND foldercontents.child_id = upload.upload_pk AND uploadtree.upload_fk = upload.upload_pk AND uploadtree.parent IS NULL; ORDER BY upload_ts; Can I simply delete from the upload table where upload_ts (select current_date -365) ? Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Clean Out Repository
Hi Ray, To find the uploads over a year old: select upload_pk from upload where upload_ts (now() - interval '1 year') order by upload_ts You can then take that list and schedule the delete agent on each one of the upload_pk's with the fossjobs command (man fossjobs). Unfortunately, you can't simply delete the upload from the database because that won't delete the files from the repository, and because cascade isn't turned on for most of the foreign keys. So, at least for now, you have to use the delete agent. Bob Gobeille On Jun 14, 2011, at 6:29 AM, Westphal, Raymond W wrote: Hello Everyone. I'm trying to find a way to automatically cleanup the repository. The users will allow me to delete uploads older than 1 year. I created the following query from watching the database while I selected the Organize/Uploads/Delete Uploaded File menu option. SELECT upload_pk, upload_desc, upload_ts, ufile_name FROM foldercontents,uploadtree,upload WHERE foldercontents.parent_fk = '4' AND upload_ts (select current_date -365) AND foldercontents.foldercontents_mode = 2 AND foldercontents.child_id = upload.upload_pk AND uploadtree.upload_fk = upload.upload_pk AND uploadtree.parent IS NULL; ORDER BY upload_ts; Can I simply delete from the upload table where upload_ts (select current_date -365) ? Thanks, Ray W. CONFIDENTIALITY NOTICE: This e-mail and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed and may contain confidential and privileged information protected by law. If you received this e-mail in error, any review, use, dissemination, distribution, or copying of the e-mail is strictly prohibited. Please notify the sender immediately by return e-mail and delete all copies from your system. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Cygwin Analysis causing scheduler restarts
Hi Ray, I just grabbed the source from the cygwin cvs, tared and uploaded it. On the system I used it took 7 minutes. http://repo.fossology.org/simpleIndex.php?mod=nomoslicenseshow=detailupload=148item=50608165 We have seen the scheduler problem you describe before. I think, as long as you aren't hitting some system resource, stopping the scheduler and restarting it might take care of the problem (btw, v2.0 has a new scheduler). Bob On May 23, 2011, at 6:31 AM, Westphal, Raymond W wrote: On Friday my customer uploaded Cygwin for analysis. I think it was a zip file. The job ran from 11:23am until I deleted it at around 8am on Sunday. I had to delete it because it caused Fossology scheduler to keep restarting. And that was causing the load average on the server to slowly climb. Has anyone had any experience with analyzing Cygwin? ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] License Tree
Hello Patrick, On May 13, 2011, at 10:25 AM, Moreau Patrick wrote: The licence tree function is no more avaliable. I send an email to Bruno Cornec and he answered that it was replaced by License List. We are not fully satisfied by the Licence List. Indeed, we appreciated to be able to double-clicked and to access directly to the licence text in order to read it. When you say that you want to read the license text, I think you mean that you want to view the file that you uploaded. That is you want to view the file that contains a license reference. This is opposed to view the text of the actual license (e.g. the full GPL license text). Is that right? This is still possible to do that but it required to look for the file ans it takes more time. It could become difficult to perform analysis for important SW. It sounds like you are using the license list as a license browser. Why don't you use the license browser instead? Do you prefer the format of all the files listed on a single page? The purpose of the License List is to give people a list of all file paths and their licenses so they can do processing outside of fossology. So for that, the links were unwanted. Putting them in would be easy but I'd like to hear from more people who use this function before deciding what to do. It would really help to know why you use this instead of the license browser. If we need the links in the License List, I'd be tempted to create a new web page where you choose what you want in the list (like links or not), and if you want to view or download it. I think that would satisfy the most people. How much does the slowness of License List (or License Tree) bother you? We have also one question. When it is written for example LGPL_v2.1+, we understand that the licence is not fully compliant. We don't know: Why do you say that it is not compliant? To me compliance means that you are complying with the license terms. LGPL, out of context, doesn't mean you are in or out of compliance. I think I need more context around what you are trying to do. - if all the licence texts are the same in this bucket What is the definition of your bucket? or - if you put all the licence texts that are not fully compliant in it and, in this case, we would have different differences that we would have to detect. Are you referring to the GPL demo bucket? If so, any license that has GPL in it is put in that bucket. So LGPL, GPLv2, GPLv3, Affero GPL, ... all go in it. That bucket was only a sample to show how to create a simple bucket. If you are really using this bucket, you might consider other buckets to be more useful. For example, a copyleft bucket, or a source required bucket. I've been wanting to create those buckets for a while but haven't found the time to figure out what should be in them. Thank you for support. Best regards Patrick PS: the new feature called Copyright/Email/URL Browser is very nice! Thank you. :-) Bob Gobeille___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] php configuration question
Hi Michael, You can turn it off. Bob Gobeille On Apr 7, 2011, at 10:06 AM, Fay Michael T wrote: Does Fossology need to have expose_php=On in the php.ini file? There is a current php vulnerability related to this that I would like to shut off. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Job fails at license (fossology 1.3.0)
Hi Kazuo, Thanks for letting me know that it worked. Don't blame yourself. In the next version (1.4) that option for the deprecated license scan won't even be there. Bob Gobeille On Mar 23, 2011, at 7:42 AM, ?? ?? wrote: Hi Bob I uploaded file with options you told, Copyright/Email/URL Analysis MIME-type Analysis (Determine mimetype of every file. Not needed for licenses or buckets) Nomos License Analysis Package Analysis (Parse package headers) I could complete license scan successfully. Thank you for your support and I apologize for my fault. Kazuo. 2011/3/21 Gobeille, Robert bob.gobei...@hp.com Hi Kazuo, I should have spotted this from your first email. You are running the wrong license scanner. What you are doing should work, but it has been replaced by a faster, more accurate scanner. When you upload a file you are given the options: Bucket Analysis Copyright/Email/URL Analysis MIME-type Analysis (Determine mimetype of every file. Not needed for licenses or buckets) Nomos License Analysis Package Analysis (Parse package headers) Spec File Analysis (deprecated. Replaced by Package analysis.) bSAM License Analysis (deprecated) It looks like you are choosing bSAM. Use Nomos License Analysis instead. In v 1.4 that bSAM option has been removed. Bob On Mar 20, 2011, at 5:57 AM, ?? ?? wrote: I can't run license with any file. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Job fails at license (fossology 1.3.0)
Hi Kazuo, This is good. You got the expected response. Unfortunately, I still don't see how someone could get this particular error. Can you run a license scan on anything? Or does it only fail on this one (apache) file? Can you reproduce this by deleting the job and resubmitting it? Bob Gobeille On Mar 16, 2011, at 11:44 PM, ?? ?? wrote: Hi Bob. I sent following command by pgAdmin3 SELECT DISTINCT(Akey), A,Size FROM license_7 INNNER JOIN agent_lic_status ON agent_lic_status.pfile_fk = Akey WHERE agent_lic_status.inrepository IS TRUE AND agent_lic_status.processed IS NOT TRUE ORDER BY Size DESC LIMIT 5000; fossology database responded 881 records. Column 'akey' contains 5digit numerical characters. Column 'a' contains 75digit numerical and upper alphabet characters separated by dot. Column 'size' contains 3digit numerical characters. Any errors was not reported. regards. 2011/3/17 Bob Gobeille bob.gobei...@hp.com Hi Kazuo, This error confuses me since it says you are violating a unique key constraint in a select. Normally you would only see an error like this when you are trying to do an insert or update. I was hoping to reproduce the problem, but could not: http://repo.fossology.org/?mod=nomoslicenseshow=detailupload=145item=47977519 Do you know how to use psql or phppgadmin or pgadmin3 or some other way to send the database a command? If so, could you run this command and tell me what response you get? SELECT DISTINCT(Akey), A,Size FROM license_7 INNNER JOIN agent_lic_status ON agent_lic_status.pfile_fk = Akey WHERE agent_lic_status.inrepository IS TRUE AND agent_lic_status.processed IS NOT TRUE ORDER BY Size DESC LIMIT 5000; Thanks, Bob Gobeille On Mar 16, 2011, at 12:57 AM, ?? ?? wrote: Hello. I'm Kazuo Tsunemine. It might be reported similar problem posted by Mr.Bob Grobeile already. When I have tried to scan 'apache_1.3.41.tar.gz2', fossology failed running license job with following message in /var/log/fossology. DEBUG[13]: ERROR:DBaccess(7) ERROR: DEBUG[13]: license_7 WARNING: In thread 13: SQL had duplicate constraint in sqlagent: 'SELECT DISTINCT(pfile_pk) as Akey, pfile_sha1 || '.' || pfile_md5 || '.' || pfile_size as A, pfile_size As Size INTO license_7 FROM uploadtree,pfile WHERE upload_fk = '7' AND pfile_fk IS NOT NULL AND (ufile_mode (129)) = 0 AND pfilefk = pfile_pk ORDER BY Size DESC;' DEBUG[13]: WARNING: SQL had duplicate constraint in sqlagent: 'SELECT DISTINCT(pfile_pk) as Akey, pfile_sha1 || '.' || pfile_md5 || '.' || pfile_size as A, pfile_size As Size INTO license_7 FROM uploadtree,pfile WHERE upload_fk = '7' AND pfile_fk IS NOT NULL AND (ufile_mode (129)) = 0 AND pfilefk = pfile_pk ORDER BY Size DESC;' And Job Queue jq_args is following message. SELECT DISTINCT(Akey), A,Size FROM license_7 INNNER JOIN agent_lic_status ON agent_lic_status.pfile_fk = Akey WHERE agent_lic_status.inrepository IS TRUE AND agent_lic_status.processed IS NOT TRUE ORDER BY Size DESC LIMIT 5000; And Job Queue jq_endtext is following message. Failed: Failed: Agent terminated prematurely I have installed fossology1.3.0(RPM) to CentOS5. I'm sorry by poor English. Kazuo. ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Job fails at license (fossology 1.3.0)
Hi Kazuo, This error confuses me since it says you are violating a unique key constraint in a select. Normally you would only see an error like this when you are trying to do an insert or update. I was hoping to reproduce the problem, but could not: http://repo.fossology.org/?mod=nomoslicenseshow=detailupload=145item=47977519 Do you know how to use psql or phppgadmin or pgadmin3 or some other way to send the database a command? If so, could you run this command and tell me what response you get? SELECT DISTINCT(Akey), A,Size FROM license_7 INNNER JOIN agent_lic_status ON agent_lic_status.pfile_fk = Akey WHERE agent_lic_status.inrepository IS TRUE AND agent_lic_status.processed IS NOT TRUE ORDER BY Size DESC LIMIT 5000; Thanks, Bob Gobeille On Mar 16, 2011, at 12:57 AM, ?? ?? wrote: Hello. I'm Kazuo Tsunemine. It might be reported similar problem posted by Mr.Bob Grobeile already. When I have tried to scan 'apache_1.3.41.tar.gz2', fossology failed running license job with following message in /var/log/fossology. DEBUG[13]: ERROR:DBaccess(7) ERROR: DEBUG[13]: license_7 WARNING: In thread 13: SQL had duplicate constraint in sqlagent: 'SELECT DISTINCT(pfile_pk) as Akey, pfile_sha1 || '.' || pfile_md5 || '.' || pfile_size as A, pfile_size As Size INTO license_7 FROM uploadtree,pfile WHERE upload_fk = '7' AND pfile_fk IS NOT NULL AND (ufile_mode (129)) = 0 AND pfilefk = pfile_pk ORDER BY Size DESC;' DEBUG[13]: WARNING: SQL had duplicate constraint in sqlagent: 'SELECT DISTINCT(pfile_pk) as Akey, pfile_sha1 || '.' || pfile_md5 || '.' || pfile_size as A, pfile_size As Size INTO license_7 FROM uploadtree,pfile WHERE upload_fk = '7' AND pfile_fk IS NOT NULL AND (ufile_mode (129)) = 0 AND pfilefk = pfile_pk ORDER BY Size DESC;' And Job Queue jq_args is following message. SELECT DISTINCT(Akey), A,Size FROM license_7 INNNER JOIN agent_lic_status ON agent_lic_status.pfile_fk = Akey WHERE agent_lic_status.inrepository IS TRUE AND agent_lic_status.processed IS NOT TRUE ORDER BY Size DESC LIMIT 5000; And Job Queue jq_endtext is following message. Failed: Failed: Agent terminated prematurely I have installed fossology1.3.0(RPM) to CentOS5. I'm sorry by poor English. Kazuo. ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Session time out
Hi Dave, How about if I remove the time out check?I don't think that level of security is really needed for fossology. Does anyone reading this have an opinion on the time out? Bob Gobeille On Feb 17, 2011, at 7:55 AM, Dave McLoughlin wrote: Fossology logs me out after a few minutes. Is there a way to configure it so it doesn’t time out and just leaves me logged in? ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Session time out
I'd like to hear from other users as well. In the mean time, do you want me to send you a version without the timeout? If so, let's take this offline. Bob Gobeille On Feb 17, 2011, at 10:43 AM, Dave McLoughlin wrote: That would work for me. Thanks Dave On 2/17/11 8:41 AM, Bob Gobeille bob.gobei...@hp.com wrote: Hi Dave, How about if I remove the time out check?I don't think that level of security is really needed for fossology. Does anyone reading this have an opinion on the time out? On Feb 17, 2011, at 7:55 AM, Dave McLoughlin wrote: Fossology logs me out after a few minutes. Is there a way to configure it so it doesn’t time out and just leaves me logged in? ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Session time out
Hi Drago, For the way you use fossology, would you prefer removing it or making it configurable? Bob Gobeille On Feb 17, 2011, at 12:15 PM, Dragoslav Mitrinovic wrote: I am also in favor of removing the timeout, or at least increasing it substantially (the default value) and making it easily configurable. Best regards, -Drago On Thu, Feb 17, 2011 at 11:58 AM, Bob Gobeille bob.gobei...@hp.com wrote: I'd like to hear from other users as well. In the mean time, do you want me to send you a version without the timeout? If so, let's take this offline. Bob Gobeille On Feb 17, 2011, at 10:43 AM, Dave McLoughlin wrote: That would work for me. Thanks Dave On 2/17/11 8:41 AM, Bob Gobeille bob.gobei...@hp.com wrote: Hi Dave, How about if I remove the time out check?I don't think that level of security is really needed for fossology. Does anyone reading this have an opinion on the time out? On Feb 17, 2011, at 7:55 AM, Dave McLoughlin wrote: Fossology logs me out after a few minutes. Is there a way to configure it so it doesn’t time out and just leaves me logged in? ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] dutch language file
This is totally awesome Jeroen! Bob Gobeille On Feb 15, 2011, at 3:13 AM, Jeroen Baten wrote: Hi, It took me 1,5 days of hard work, but here is the language file for the nl_NL locale. greets, Jeroen Baten messages.poATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Weird scan of fossology-1.3.0.tar.gz fails Package Scan
This was supposed to be fixed on January 13. http://bugs.linux-foundation.org/show_bug.cgi?id=596 1.3.0 was released January 19. Mary, did this fix get into 1.3.0? Bob Gobeille On Feb 11, 2011, at 1:47 AM, Jeroen Baten wrote: Don't know if this helps, but this is a relevant piece of the log: 2011-02-10 13:42:27 scheduler[716] : Child[15] 'agent=pkgagent host=localhost ' state=SPAWNED(4) @ Thu Feb 10 13:42:27 2011 2011-02-10 13:42:27 scheduler[716] : Child[15] 'agent=pkgagent host=localhost ' state=READY(5) @ Thu Feb 10 13:42:27 2011 2011-02-10 13:42:27 scheduler[716] : Child[6] 'agent=buckets host=localhost ' state=FREE(1) @ Thu Feb 10 13:42:27 2011 2011-02-10 13:45:52 scheduler[716] : ERROR: Child[15] died prematurely (was state RUNNING, signal was 11) 2011-02-10 13:45:52 scheduler[716] : Thread 15: 2011-02-10 13:45:52 scheduler[716] : PID: 1736 2011-02-10 13:45:52 scheduler[716] : Pipes: in=8-9 / out=13-12 2011-02-10 13:45:52 scheduler[716] : Attr: 'agent=pkgagent host=localhost ' 2011-02-10 13:45:52 scheduler[716] : Command: '/usr/local/lib/fossology/agents/pkgagent' 2011-02-10 13:45:52 scheduler[716] : Parm: 'pfile_pk=87202 pfilename=BE9651C3D92B8A76FD3D0C03FF5597411E8FF860.A0FD8E3774FED62ABEB6C2CDA3362862.2639 mimetype=application/x-rpm ' 2011-02-10 13:45:52 scheduler[716] : Heartbeat: Thu Feb 10 13:45:42 2011 2011-02-10 13:45:52 scheduler[716] : State: Thu Feb 10 13:45:42 2011 2011-02-10 13:45:52 scheduler[716] : Status: 6 (RUNNING) 2011-02-10 13:45:52 scheduler[716] : Spawn: 1 at Thu Feb 10 13:42:27 2011 2011-02-10 13:45:52 scheduler[716] : DB: 2011-02-10 13:45:52 scheduler[716] : IsDB: 2 2011-02-10 13:45:52 scheduler[716] : DBJobKey: 128 2011-02-10 13:45:52 scheduler[716] : DBMSQrow: 1 2011-02-10 13:45:52 scheduler[716] : DBagent: 29 2011-02-10 13:45:52 scheduler[716] : Child[15] 'agent=pkgagent host=localhost ' state=FREEING(2) @ Thu Feb 10 13:45:52 2011 2011-02-10 13:45:52 scheduler[716] : Child[15] 'agent=pkgagent host=localhost ' state=FREE(1) @ Thu Feb 10 13:45:52 2011 On Thu, 2011-02-10 at 11:37 -0700, Bob Gobeille wrote: Hi Jeroen, The best place to see why an agent failed is in the fossology log file (/var/log/fossology/fossology.log). If you look there and see a spawning too fast error, that is a known (major) bug. See: http://bugs.linux-foundation.org/show_bug.cgi?id=636 If it is something different than what is in the bug report, please let us know. Thanks, Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Wondering which are OSI approved licenses
Everything is a work in progress. ;-) But buckets are used today. The only work going on with buckets is a new tool that lets you compare buckets between files. Bob Gobeille On Feb 11, 2011, at 1:45 AM, Jeroen Baten wrote: Hi, I found the page on buckets and see that it is a 'work in progess'. No problem. I will see if I can help. Thanks for the heads-up. kind regards, On Thu, 2011-02-10 at 20:36 +0200, Raino Lintulampi wrote: Hi Jeroen, I think buckets are what you are looking for. With buckets you can classify licenses as you want f.ex. for OSI approved, and with colors. br Raino ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Copyright listed by file
On Feb 9, 2011, at 8:10 PM, Dave McLoughlin wrote: I noticed that with the new Nomos license analysi there is an option get the file list associated with the licenses. How would I do that for the copyright/email/url info. Can I write a sql query and get that info? Hi Dave, That's a good idea. Raino recently made a great suggestion that we put a filter in the copyright report (much like the new comparison tool) to limit the results to just those files where no license was found. I could add links to the copyright page to display or download the raw list (just like we do for the license scan). It's easy to generate the list from an sql procedure, but not a query due to how files data is reused between uploads. Is this something you can wait for or would it be a great help for you to have it now? Bob Gobeille___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Weird scan of fossology-1.3.0.tar.gz fails Package Scan
Hi Jeroen, The best place to see why an agent failed is in the fossology log file (/var/log/fossology/fossology.log). If you look there and see a spawning too fast error, that is a known (major) bug. See: http://bugs.linux-foundation.org/show_bug.cgi?id=636 If it is something different than what is in the bug report, please let us know. Thanks, Bob Gobeille On Feb 10, 2011, at 5:49 AM, Jeroen Baten wrote: This is the info I found. Maybe it helps you people. Kind regards, Jeroen Baten jq_pk 128 jq_job_fk 108 job_name Package Scan jq_type pkgagent job_priority 0 jq_args SELECT pfile_pk as pfile_pk, pfile_sha1 || '.' || pfile_md5 || '.' || pfile_size AS pfilename, mimetype_name AS mimetype FROM uploadtree INNER JOIN pfile ON upload_fk = '7' INNER JOIN mimetype ON (mimetype_pk = '1' OR mimetype_pk = '2' OR mimetype_pk = '3') AND uploadtree.pfile_fk = pfile_pk AND pfile.pfile_mimetypefk = mimetype.mimetype_pk AND pfile_pk NOT IN (SELECT pkg_rpm.pfile_fk FROM pkg_rpm) AND pfile_pk NOT IN (SELECT pkg_deb.pfile_fk FROM pkg_deb) LIMIT 5000; jq_runonpfile pfile_pk jq_starttime 2011-02-10 13:45:42.098366+01 jq_endtime2011-02-10 13:45:52.166725+01 jq_end_bits 2 jq_endtextFailed: Failed: Agent terminated prematurely jq_elapsedtime00:02:44 jq_processedtime 00:00:01 jq_itemsprocessed 29 job_submitter job_queued2011-02-10 13:40:11.799265+01 job_email_notify fossy job_upload_fk 7 depends on40 required by 127, 129 ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Fwd: compile gives error. Where to look?
From: Jeroen Baten Date: February 9, 2011 12:11:22 AM MST To: Gobeille, Robert bob.gobei...@hp.com Subject: Re: [FOSSology] compile gives error. Where to look? Hello Bob, You were right. Your solution fixed it. Had another error later on with postinstall: *** Initializing database tables *** PHP Fatal error: Call to undefined function posix_getgrnam() in /usr/local/bin/fossinit on line 34 ERROR: Unable to initialize. Figured it was a php module dependancy on posix. fixed it. It is in php-process on fc14. thanks for your help! kind regards, On Tue, 2011-02-08 at 09:02 -0700, Bob Gobeille wrote: Hi Jeroen, The easiest Fedora install is to use an rpm rather than building from source: http://fossology.org/installing_fossology_with_rpm_s That said, I personally have no experience with the rpm's. The error you see looks like a mistake in the makefile. libfosspython isn't used in 1.3. It was used by our previous copyright agent. We don't need it for the new copyright agent. I haven't tested this but you should be able to: 1) cd fossology-1.3.0/devel/ 2) rm -rf libfosspython 3) edit Makefile - on line 6 you will see: DIRS = libfossdb libfossrepo libfossagent libfosspython Change it to: DIRS = libfossdb libfossrepo libfossagent Then rerun make. Bob Gobeille On Feb 8, 2011, at 8:15 AM, Jeroen Baten wrote: Hello, The compile of 1.3.0 on fedora core 13 fails with an error (duh). But where should I look for the solution to this? kind regards, Jeroen Baten [jbaten@htn-ws-3092 fossology-1.3.0]$ make make -C devel make[1]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel' make -C libfossdb make[2]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossdb' gcc -c libfossdb.c -DFOSSDB_CONF='/usr/local/etc/fossology/Db.conf' -I`pg_config --includedir` -fPIC -DSVN_REV=\1.3.0, exported\ -g -O2 -Wall ar cr libfossdb.a libfossdb.o gcc dbtest.c -lpq -I`pg_config --includedir` -I../../devel/libfossdb -L../../devel/libfossdb -lfossdb -I`pg_config --includedir` -o dbtest gcc dbq.c -lpq -I`pg_config --includedir` -I../../devel/libfossdb -L../../devel/libfossdb -lfossdb -I`pg_config --includedir` -o dbq gcc dbcheck.c -lpq -I`pg_config --includedir` -I../../devel/libfossdb -L../../devel/libfossdb -lfossdb -I`pg_config --includedir` -o dbcheck make[2]: Leaving directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossdb' make -C libfossrepo make[2]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossrepo' gcc -c libfossrepo.c -DFOSSREPO_CONF='/usr/local/etc/fossology' -DFOSSGROUP='fossy' -fPIC -DSVN_REV=\1.3.0, exported\ -g -O2 -Wall ar cr libfossrepo.a libfossrepo.o gcc reppath.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o reppath gcc repwrite.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repwrite gcc repcat.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repcat gcc repexist.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repexist gcc repcopyin.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repcopyin gcc repremove.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repremove gcc rephost.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o rephost gcc repmmapcheck.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repmmapcheck make[2]: Leaving directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossrepo' make -C libfossagent make[2]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossagent' gcc -c libfossagent.c -I`pg_config --includedir` -I../../devel/libfossdb -L../../devel/libfossdb -lfossdb -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -fPIC -DSVN_REV=\1.3.0, exported \ -g -O2 -Wall ar cr libfossagent.a libfossagent.o make[2]: Leaving directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossagent' make -C libfosspython make[2]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfosspython' python setup.py build_ext --inplace running build_ext pyrexc libfosspython.pyx -- libfosspython.c /home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfosspython/libfosspython.pyx:156:4: __new__ method of extension type will change semantics in a future version of Pyrex. Use __cinit__ instead. building 'libfosspython' extension creating build creating build/temp.linux-x86_64-2.7 gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp
Re: [FOSSology] spontaneous user feedback on 1.3.0
Hi Jeroen, We always like feedback. See embedded comments. Bob Gobeille On Feb 9, 2011, at 3:38 AM, Jeroen Baten wrote: primary feedback: I love this tool! How can I help? We have a developer's mailing list fossology-de...@fossology.org. Give us enough information so we can get an idea of your competencies, then we can make suggestions. If you have a particular area of interest, let us know. Typically what we do is have people send us patches. Then when we are all comfortable with your judgement and ability, we would give you commit access. There are so many things to do on this project, take a look at http://fossology.org/task_list. Those are features, but we also have docs, install testing, feature testing, language translations, ... question: someone told me about 'maven site' that show licensing information. I have the feeling Fossology is better but I lack the arguments. Can you help? I think maven records licenses in their project object but doesn't figure out what they are. I really don't know much about it. Features wanted: For every release we prioritize what we should do. The priority is very heavily based on what users tell us would make them more productive. Keep that in mind in the discussion that follows. Things that aren't a priority now, could become so if we hear from enough users. Sometimes enough users can be one. === -Guestimations in the job details screen. Great that I am analyzing a gazillion files, but when will it end approximately? Do you have some ideas on how to do this? I do but it's not high on the priorities. -Did I already mention guestimations? :-) -Why am I seeing the contents of a binary in the 'browse' screen? Maybe a warning 'are your sure' beforehand? We don't show a warning because it's reasonable that you want to see the binary. That's why we have 3 options for how it is displayed (hex, text, formatted). If you click View on a binary, you will see the binary. But we have had errors where a binary was not unpacked. In that case you expect to be able to drill down through it to see it's component files, but cannot. All you can do is view the binary. -in job screen links to short help on text in 2nd column would be great for newbie. What is this step doing? Are you looking at the agent name column (unpack, adj2nest, pkgagent, fo_notify, nomos, ...)? -in job screen: links in Queued Scheduled Running Finished Blocked Failed to show only those type jobs. The Queued, Scheduled, ... colored bar aren't links. That is just a legend for the color codes. We never thought about making them links, but I can see the appeal. We are working on a new scheduler for V2.0. We should revisit this with 2.0. -My first mistake was uploading a binary file. Don't know how to prevent that one though. What type of binary file? Was it an archive (bz2, gzip, tar, iso,...) or a single file binary like a .exe? The former should have been unpacked, the latter not. -upload direct from subversion and also download sources of dependencies (maven libs and stuf) Upload from subversion is on http://fossology.org/task_list. I suspect this one will bubble up in priority fairly fast if we get more user requests. -hudson plugin Mark (fossology developer on this list) was working on that. He could fill us in on the status. -nagios check: job with id X only found known open source licenses means status green. found unknown license: staus orange. Found known closed source license: status red. This is a new one. There is a proposal for a REST API on http://fossology.org/task_list. However, the proposal is languishing due to lack of response. Errors encountered in php and query code: === -I don't know it they are big, but here's my syslog: I dedoubled stuff but maybe missed 1 or 2. Feb 9 09:20:26 htn-ws-3092 httpd: PHP Notice: Undefined index: All of these Undefined Index notices can be ignored. You have your php logging level turned up very high to see these. That said, I have the same log level and we try to fix them when we see them in any code we are touching. We just haven't caught them all. Feb 9 09:35:58 htn-ws-3092 postgres[29401]: [2-1] 2011-02-09 09:35:58 CET ::1 4d524b2d.72d9ERROR: prepared statement FolderGetFromUpload_1_exported already exists I've been meaning to fix these duplicate prepared statement errors for a long time. You can safely ignore them. They are really notices not errors, but they are annoying and can clog up the postgres log file. Thanks for the feedback, and we would be happy for you to work on fossology with us. The nice thing about working on an open source project like this is that you can fix all the things that annoy you. ;-) Bob Gobeille ___ fossology
Re: [FOSSology] compile gives error. Where to look?
Hi Jeroen, The easiest Fedora install is to use an rpm rather than building from source: http://fossology.org/installing_fossology_with_rpm_s That said, I personally have no experience with the rpm's. The error you see looks like a mistake in the makefile. libfosspython isn't used in 1.3. It was used by our previous copyright agent. We don't need it for the new copyright agent. I haven't tested this but you should be able to: 1) cd fossology-1.3.0/devel/ 2) rm -rf libfosspython 3) edit Makefile - on line 6 you will see: DIRS = libfossdb libfossrepo libfossagent libfosspython Change it to: DIRS = libfossdb libfossrepo libfossagent Then rerun make. Bob Gobeille On Feb 8, 2011, at 8:15 AM, Jeroen Baten wrote: Hello, The compile of 1.3.0 on fedora core 13 fails with an error (duh). But where should I look for the solution to this? kind regards, Jeroen Baten [jbaten@htn-ws-3092 fossology-1.3.0]$ make make -C devel make[1]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel' make -C libfossdb make[2]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossdb' gcc -c libfossdb.c -DFOSSDB_CONF='/usr/local/etc/fossology/Db.conf' -I`pg_config --includedir` -fPIC -DSVN_REV=\1.3.0, exported\ -g -O2 -Wall ar cr libfossdb.a libfossdb.o gcc dbtest.c -lpq -I`pg_config --includedir` -I../../devel/libfossdb -L../../devel/libfossdb -lfossdb -I`pg_config --includedir` -o dbtest gcc dbq.c -lpq -I`pg_config --includedir` -I../../devel/libfossdb -L../../devel/libfossdb -lfossdb -I`pg_config --includedir` -o dbq gcc dbcheck.c -lpq -I`pg_config --includedir` -I../../devel/libfossdb -L../../devel/libfossdb -lfossdb -I`pg_config --includedir` -o dbcheck make[2]: Leaving directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossdb' make -C libfossrepo make[2]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossrepo' gcc -c libfossrepo.c -DFOSSREPO_CONF='/usr/local/etc/fossology' -DFOSSGROUP='fossy' -fPIC -DSVN_REV=\1.3.0, exported\ -g -O2 -Wall ar cr libfossrepo.a libfossrepo.o gcc reppath.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o reppath gcc repwrite.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repwrite gcc repcat.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repcat gcc repexist.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repexist gcc repcopyin.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repcopyin gcc repremove.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repremove gcc rephost.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o rephost gcc repmmapcheck.c -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -o repmmapcheck make[2]: Leaving directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossrepo' make -C libfossagent make[2]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossagent' gcc -c libfossagent.c -I`pg_config --includedir` -I../../devel/libfossdb -L../../devel/libfossdb -lfossdb -I../../devel/libfossrepo -L../../devel/libfossrepo -lfossrepo -fPIC -DSVN_REV=\1.3.0, exported \ -g -O2 -Wall ar cr libfossagent.a libfossagent.o make[2]: Leaving directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfossagent' make -C libfosspython make[2]: Entering directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfosspython' python setup.py build_ext --inplace running build_ext pyrexc libfosspython.pyx -- libfosspython.c /home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfosspython/libfosspython.pyx:156:4: __new__ method of extension type will change semantics in a future version of Pyrex. Use __cinit__ instead. building 'libfosspython' extension creating build creating build/temp.linux-x86_64-2.7 gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/postgresql -I/usr/include/python2.7 -c libfosspython.c -o build/temp.linux-x86_64-2.7/libfosspython.o libfosspython.c:1:2: error: #error Do not use this file, it is the result of a failed Pyrex compilation. error: command 'gcc' failed with exit status 1 make[2]: *** [libfosspython.so] Error 1 make[2]: Leaving directory `/home/local/ONVZ/jbaten/src/java/license-check/fossology-1.3.0/devel/libfosspython' make[1]: *** [build-libfosspython] Error 2 make[1]: Leaving directory `/home/local/ONVZ/jbaten/src/java/license
Re: [FOSSology] Announcing FOSSology 1.3.0 Release (Laser, Mary)
On Jan 19, 2011, at 3:41 PM, Dave McLoughlin wrote: I believe I saw something in the mailing list about losing backwards compatibility with 1.2 when 1.3 is released. Is that true? Could you be referring to the planned obsolescence of bsam license data? Or the forwards/backward database compatibility (that taggart answered already)? Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Announcing FOSSology 1.3.0 Release (Laser, Mary)
This is correct. Since we get better and faster results with nomos, and bSAM is difficult to maintain, we no longer want to support it. We don't want to migrate your data because it is potentially very time consuming to rescan all your uploads. This would be especially wasteful if you are no longer interested in all of your previously scanned data. We have been putting out messages about this for quite a while but have receive no feedback until your email arrived. All the bsam code is still in 1.3 but it is no longer supported. In the next release we would like to remove that code, which will orphan your bsam data. Suggestions? Bob Gobeille On Jan 20, 2011, at 12:20 PM, Dave McLoughlin wrote: I found it. Here’s what I was referring to: From Mary Laser to the fossology mailing list on 11/12/10: “Furthermore, there is NO upgrade path from bSAM results to nomos results. Therefore, you MUST rerun license analysis on your uploads using the nomos analyzer introduced in 1.2.0.” ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] at SCALE Feb 25-25 Los Angeles
Hi Everyone, I plan on being at the Southern California Linux Expo Feb 25-27 http://www.socallinuxexpo.org/scale9x/ So of anyone wants to get together there and talk about fossology, let me know. I'll be in the HP booth, but we can get together in the evening if you would like. Thanks, Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Reg: Updates to Fossology repository
Hi Pravin, I can answer your question on updating licenses and copyrights. How to add a license is documented on http://fossology.org/nomos. However, adding a license is a code fork. So if this is a license that is not specific to you, it would be good to get it into the fossology distribution. We have had people who aren't committers send us patches. I've also added licenses on request.I could add a method to simplify this, but so far there hasn't been much demand. Alex, who wrote the copyright scanner is going to write up how to update it. Currently, the copyright scanner uses a very small dictionary to identify copyrights and other developer touch points. The dictionary looks like this: copyright (c) written modified patched maintained contributed copy; #169; #xa9; author So one thing you can do is extend this dictionary. However, we could add a second method that uses regex's so that you could write very specific rules. Would that be helpful or does extending the dictionary work for you? If you can share on this list (or send me privately if necessary) the copyrights that are being missed, we can get a better idea of the problem you see and what the solution is. Thanks, Bob Gobeille b...@fossology.org On Dec 27, 2010, at 2:58 AM, Ma, Dong (Vincent, ES-Best-Shore-Services-China-BJ) wrote: Hi Pravin, Sorry for the late reply as most of fossology team member in vacation now. Regarding your request, I can try to help with this: Request 1: 1. Your installation is from source or packages? a. If from source, you need to check out new release code, compile and install with install instruction from source. b. If from packages, the new packages will support upgrade. 2. Both from source/packages, the upgrade process will not remove your old date in the repository and database. Request 2: For now we didn’t support edit the analysis results, if needed you should update from database, this will be our further feature. Hope that help some. Thanks, Vincent From: fossology-boun...@fossology.org [mailto:fossology-boun...@fossology.org] On Behalf Of Pravin Jagatheesan Sent: Friday, December 24, 2010 5:02 PM To: fossology@fossology.org Subject: [FOSSology] Reg: Updates to Fossology repository Hi, I have installed the latest version of fossology 1.2.1 Appreciate if someone can help me with the following 1. How to update the repository when there are new releases from fossology 2. How do I update a license or copyright in my local repository as I have some application specific copyrights Thanks, Pravin CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS*** ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Something weird with buckets/pkgagent agent with src.rpm
Hi Raino, Sorry for such a late response. I was taking a week off and not answering fossology (or HP) email. I'm glad the patched fixed the first problem. 1 item/sec is not normal for pkgagent. I just ran an RHEL DVD with 2.5 M files. Pkgagent ran to completion in 1 minute (found 4,110 packages). Did you look at the fossology, postgres and apache log files for clues? Do you have phpPgAdmin installed? It makes it easy to see what command postgres is working on (the database Processes link). Bob Gobeille On Nov 25, 2010, at 7:15 AM, Raino Lintulampi wrote: Hi, patch removed the error messages, so it solved the original problem. Execution of pkgagent is still slow (avg. 1 item/second) but I guess there is nothing that can be done with that. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] new licenses
I have a little time to work on the license scanner. If anyone has bugs they haven't reported, or would like me to add new licenses, please let me know asap. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Install of 1.2.1
On Nov 15, 2010, at 11:02 AM, Dave McLoughlin wrote:2) After the install I'm see the following in my browser when I navigate to http://localhost/repo/: Warning: file_get_contents(/usr/local/etc/fossology/Db.conf) [function.file-get-contents]: failed to open stream: Permission denied in /usr/local/share/fossology/www/plugins/core-db.php on line 91 Warning: pg_pconnect() [function.pg-pconnect]: Unable to connect to PostgreSQL server: FATAL: database "www-data" does not exist in /usr/local/share/fossology/www/plugins/core-db.php on line 91 Warning: pg_last_error(): supplied argument is not a valid PostgreSQL link resource in /usr/local/share/fossology/www/plugins/core-db.php on line 136 The user/group is set to root on those files. Do they need to fossy/fossy?Your web browser needs to have read access to them to the .conf files. Typically one would put www-data (or whatever your web server runs as) in the fossy group. Then set your permissions:-rw-rw 1 root fossy 62 2010-10-21 16:15 /usr/local/etc/fossology/Db.confI've attached the INSTALL doc if you want to know the details.All the above "Warnings" (which should have been handled as fatal) are because your web server doesn't have read access to the files in /usr/local/etc/fossology. fo_postinstall should have taken care of this.Someone else will have to answer you about why you get the install errors. Did you try:apt-get install fossology (http://fossology.org/ubuntu)? INSTALL Description: Binary data Bob Gobeille___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Can I automate download of a nomos license list?
On Oct 18, 2010, at 8:36 AM, Mike Kinghan wrote:Hi folks,I'm trying to get an automated way to download the nomos license list for a previously analysed upload. I thought something like this would return the license list:wget --user=fossy --password=fossy "http://localhost/foss/?mod=license-listshow=detailupload=1item=1"
But it doesn't. The output I get is:Module unavailable or your login session timed out. P /Click a href=''http://repo.fossology.org/?mod=license-listshow=detailupload=58item=25317859'You will have to parse the web page output to get the list.You may be getting a session timed out because generating this list is slow. Try it with a very small number of files first.Here are a couple of random bits that might help:1) if you don't want to provide the user/pass for a plugin, you can set the plugin to not need authentication. You can do this by changing ui-nomos-license-list.php, change "$DBaccess = PLUGIN_DB_READ;" to "$DBaccess = PLUGIN_DB_NONE;"2) The timeout is controlled in your php.ini file3) Use a custom script (see attached) so you can get the output just as you like it:The url for this script will be something like:http://repo.fossology.org/?mod=custom_liclistupload=58item=25317859Put it in your ui/plugins directory.Bob Gobeille?php
/***
Example program to download license list.
*/
global $GlobalReady;
if (!isset($GlobalReady)) {
exit;
}
define(TITLE_ui_custom_liclist, _(Nomos License List));
class custom_liclist extends FO_Plugin {
var $Name = custom_liclist;
var $Title = TITLE_ui_custom_liclist;
var $Version = 1.0;
var $Dependency = array( db);
var $DBaccess = PLUGIN_DB_NONE;
var $LoginFlag = 0;
var $NoHTML = 1;
function Output()
{
global $PG_CONN, $DB;
if (!$PG_CONN) { $dbok = $DB-db_init(); if (!$dbok) echo _(NO DB connection); }
if ($this-State != PLUGIN_STATE_READY) return (0);
// I could easily look up the upload_pk from the uploadtree_p, or vice versa.
// but for now require both
$uploadtree_pk = GetParm(item, PARM_INTEGER);
if (empty($uploadtree_pk))
{
echo Missing uploadtree_pk (aka item);
return;
}
$upload_pk = GetParm(upload, PARM_INTEGER);
if (empty($upload_pk))
{
echo Missing upload_pk (aka upload);
return;
}
/* get last nomos agent_pk that has data for this upload */
$Agent_name = nomos;
$AgentRec = AgentARSList(nomos_ars, $upload_pk, 1);
$agent_pk = $AgentRec[0][agent_fk];
if ($AgentRec === false)
{
echo _(No data available);
return;
}
/* get the top of tree */
$sql = SELECT upload_fk, lft, rgt from uploadtree where uploadtree_pk='$uploadtree_pk';
$result = pg_query($PG_CONN, $sql);
DBCheckResult($result, $sql, __FILE__, __LINE__);
$toprow = pg_fetch_assoc($result);
pg_free_result($result);
/* loop through all the records in this tree */
$sql = select uploadtree_pk, ufile_name, lft, rgt from uploadtree
where upload_fk='$toprow[upload_fk]'
and lft'$toprow[lft]' and rgt'$toprow[rgt]'
and ((ufile_mode (128)) = 0);
$outerresult = pg_query($PG_CONN, $sql);
DBCheckResult($outerresult, $sql, __FILE__, __LINE__);
/* Select each uploadtree row in this tree, write out text:
* filepath : license list
* e.g. Pound-2.4.tgz/Pound-2.4/svc.c: GPL_v3+, Indemnity
*/
while ($row = pg_fetch_assoc($outerresult))
{
$filepatharray = Dir2Path($row['uploadtree_pk']);
$filepath = ;
foreach($filepatharray as $uploadtreeRow)
{
if (!empty($filepath)) $filepath .= /;
$filepath .= $uploadtreeRow['ufile_name'];
}
echo $filepath . : . GetFileLicenses_string($agent_pk, 0, $row['uploadtree_pk']) . br\n ;
}
pg_free_result($outerresult);
return;
}
};
$NewPlugin = new custom_liclist;
$NewPlugin-Initialize();
?
___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Fossology Install
Hi Martin, There will be a way in v 1.3 (http://fossology.org/difference_tool). I just started working on this last Friday. In 1.2 you can Browse, then click on Nomos License List Download in the micromenu (yellow bar). Do this for both versions and diff the output. But beware, making this list is very slow. It will work to get a license list of a package (or maybe 200 packages), but it is too slow to complete if you are doing it on say a Fedora DVD. Bob Gobeille On Sep 20, 2010, at 7:48 AM, Martin von Willebrand wrote: By the way, is there currently a way to see the differences between two scans? Say I just had version 2.7.0 scanned and used 3 hours to reviewing the results and picking up and reporting problematic files. Then I want to scan version 2.7.1. (This will probably happen very fast, since only the changed files are scanned.) Is there any way to just review the results of the changed files (or changed results). ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] errors after upgrading to 1.2
It looks like your upgrade to 1.2 didn't completely work. The error you just sent says the bucketpool table doesn't exist. Creating it is part of the upgrade process. What OS are you using and how did you do the upgrade? Bob On Sep 20, 2010, at 1:29 PM, Fay Michael T wrote: I went to edit the users and got this error: File: /usr/share/fossology/www/common/common-buckets.php, Line number: 52 ERROR: relation bucketpool does not exist select * from bucketpool where active='Y' #0 debugbacktrace() called at [/usr/share/fossology/www/common/common-ui.php:117] #1 DBCheckResult(, select * from bucketpool where active='Y', /usr/share/fossology/www/common/common-buckets.php, 52) called at [/usr/share/fossology/www/common/common-buckets.php:52] #2 SelectBucketPool() called at [/usr/share/fossology/www/plugins/user-edit-self.php:274] #3 user_edit_self-Output() called at [/usr/share/fossology/www/index.php:44] I’m logged in as fossy…But this confuses me as my test was obviously able to log in… From: Bob Gobeille [mailto:bob.gobei...@hp.com] Sent: Monday, September 20, 2010 2:17 PM To: Fay Michael T Cc: 'fossology@fossology.org' Subject: Re: [FOSSology] errors after upgrading to 1.2 Hi Michael, I suspect this user does not have any default agents. Update your account settings and make sure at least one of the default agents is checked. Let me know if this fixes it. If it does I'll file a bug. Bob Gobeille On Sep 20, 2010, at 12:42 PM, Fay Michael T wrote: Hi, We are getting errors after having upgraded to v1.2. I have included screen shots of the errors: Uploading Fatal error after clicking upload.JPG Uploading Fatal error.JPG Does anyone have any ideas here? Thanks, Michael ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Extracting data from Fossology
Hello Jean-Paul, In the mini menu (menu in the yellow bar), click on Nomos License List Download. That will download a file in the format: file path : license1, license2, ... This can take several minutes to generate (depending on your system performance and the number of files in the container you are getting the license list for). So to see what this does, browse to a directory that doesn't have many files in it (i.e. 1000 files), then click on Nomos License List to see the list displayed in your browser, or Nomos License List Download to download the same list in the format above as a text files. Bob Gobeille On Sep 14, 2010, at 8:17 AM, Jean-Paul Broussard wrote: Hello, I would like extract automatically the results of Fossology to Open Office Calc for example. I'm using the Fossology version 1.2.0 rc2 (Ubuntu Hardy 8.04). Is it possible? If yes, can you help to do this. Is there a plugging or other? Best regards Jean-Paul -- This message and any attachments herein are confidential, intended solely for the addressees and are SoftAtHome’s ownership. Any unauthorized use or dissemination is prohibited. If you are not the intended addressee of this message, please cancel it immediately and inform the sender. ATT1..txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Uploading files over 1MB
Hi Leslie, Your php configuration may be the limitation here. Look in your php.ini file (mine is located in /etc/php5/apache2/) and check your max upload size: ; Maximum allowed size for uploaded files. upload_max_filesize = 4094M Bob Gobeille On Aug 19, 2010, at 2:19 PM, Haynes, Leslie D wrote: We have a new installation of Fossology 1.2 on RHEL 5.4. When we try to upload files over 1 MB the connection seems to timeout. Smaller files successfully upload. Any ideas/help is appreciated. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Fossology Install
Gurbax, I think the best thing to do now is provide the info taggart requested What distribution (name and version) are you using and what apt source are you using to get the packages? You shouldn't be having all these issues so with the above info we can at least test the release/disto combination. Bob Gobeille On Aug 17, 2010, at 3:01 PM, Deol, Gurbax wrote: Mary, Spent too much time trouble shooting it. Since it was a new machine I am setting up, I have un-installed and re-installed Ubuntu OS on my machine. At this point, there is no trace of Fossology. What is the best way to install Fossology without running into issues? Thanks for your support in advance. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Fwd: list of nomos licenses
Thanks Raino. I hope you don't mind me forwarding this to the public mailing list. I've added a nomos link in the developer docs (http://fossology.org/developer_documentation). On the nomos page I put a link to your license list: http://fossology.org/nomos_liclist Thanks so much! Bob Gobeille Begin forwarded message: From: Raino Lintulampi Date: August 16, 2010 3:07:14 PM MDT To: b...@fossology.org b...@fossology.org Subject: list of nomos licenses Hi Bob, one problem which I faced when creating buckets was that I couldn't figure out what where the license names nomos was using. And furthermore what are the licenses which nomos identifies. Nomos is very good checker but the information about the known licenses is implicit. So your reply on how to add a new nomos license gave me a hint where the real information could be foudn and so I grep the following information from the parse.c file, see attachment. If the information is accurate enough and you think it could be useful, be free to use it in fossology documentation. br Raino ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Scheduler running slow messages...
Hi Gibran, On Jul 21, 2010, at 12:23 PM, Furosh One wrote: In addition, one of our engineers has also created some reporting tools and he's not sure of the impact the upgrade may have on our report generator, whether it uses the scanner or not. We basically provide reports of our releases that generate all license info per folder/scan we perform on our releases. Yes, this will be a change, but it may be easier in 1.2. It's a change because 1.2 uses different tables to store license results, but it's much easier because all the license results are in a single table. Also in 1.2 you can click on a link to get a file of all the file pathnames and their licenses. That file is easily parsable. Now, I'm wondering if anyone has seen any issues with updatedb somehow getting in the way of the scheduler at times? Other than disk contention I don't know why updatedb would have anything to do with our software. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Scheduler running slow messages...
This can happen with some very fast agents like the mimetype agent (which you don't need to run unless you are doing your own db queries. Is this what is running? Bob Gobeille On Jul 19, 2010, at 4:54 PM, Furosh One wrote: I'm not sure what could be causing this, do you know what I can check or verify which may be causing the scheduler to run slow? WARNING: scheduler is running slow. It took 302 seconds to check agent status: Mon Jul 19 15:34:56 2010 WARNING: scheduler is running slow. It took 182 seconds to check agent status: Mon Jul 19 15:37:58 2010 WARNING: scheduler is running slow. It took 323 seconds to check agent status: Mon Jul 19 15:44:17 2010 This is a pretty good server with 16 CPUs, 24 Gb memory, and 1 TB of storage space... It's a single-fossology machine as opposed to a mult-setup. -FuRoSh ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Scheduler running slow messages...
On Jul 20, 2010, at 1:41 PM, Furosh One wrote: On Tue, Jul 20, 2010 at 10:22 AM, Bob Gobeille bob.gobei...@hp.com wrote: This can happen with some very fast agents like the mimetype agent (which you don't need to run unless you are doing your own db queries. Is this what is running? Bob Gobeille Thanks for the response Bob, Actually the mimetype agent was not according to the person who ran the last 2 jobs. I am seeing problems with database Errors and the scheduler Exiting repeatedly after looking in: /var/lib/pgsql/data/pg_log/* /var/log/fossology/fossology.log I have shut down and restarted the postgresql and fossology services. Also ran DB check, Vacuum/Analyze on db. I also checked 'yum update' (since I installed from fossology repo on my RHEL5 server) and it found a newer fossology, but not sure if I should upgrade it.I'm guessing I probably should since I'm having these weird issues... I would update to 1.2 if I were you. ;-) You can see the major chages on http://fossology.org That was good that you did a vac/analyze because that can cause the same problem. If you really want to track this down, it would be good to know what agent was running. Also to track these down, look in your postresql log file, if you have it set to log queries that take to long, that's a clue. Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Warning: delete agent
Hello fossologists, I just wanted to send out a warning about delagent (in v 1.1 and all previous versions): Organize Uploads Delete Uploaded file When you delete an uploaded file, DO NOT process any other jobs at the same time. If you simultaneously upload a file that contains duplicates of files that you are deleting, you can leave the database with holes for those files. For example, if you were to delete upload myfiles.tar and simultaneously upload a file myotherfiles.tar and both tar files contain widget.c, then it is possible that the delete removes widget.c after it is unpacked from myotherfiles.tar. This can lead to erroneous results. In this example, the licenses found in myotherfiles.tar could be missing the the licenses found in widget.c. This problem is very dependent on the timing of the delete with other agents and is only a problem if there are files in common. But to be safe, queue up delagent all by itself. Don't queue something else until delagent has finished. So how do you know when del agent finishes? This is harder than it sounds. Del agent removes itself from the job queue pretty early in its process execution. It does this because the job table contains a reference to the upload being deleted which must be removed before the actual files can be removed. So you can't use Jobs Queue to tell you when delagent is done. It will disappear before the agent finishes. The safest indicator is to use Admin Agent Status. Ten minutes after delagent has finished, this status will show delagent as FREE. I'm considering refactoring the scheduler, which will fix these problems (and speed up the agents a great deal - early indicator on the new nomos agent shows a 5X speedup by the agent). But don't count on this in 1.2. Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] FOSSology - Quick questions
Hi George, On Oct 29, 2009, at 11:57 AM, George A Pace wrote: 1. Binary File processing - It looks like FOSSology is reading them - I just wanted to verify that this is indeed true - and what changes are coming in future versions Anything with a binary type, we run through strings (1) and then use that output to scan for licenses. 2. Timeframe to the next FOSSology version (I guess version 1.2) - Is this looking more like a 1st half 2010 ? Yes on 1st half of 2010. We really don't have an estimate, but my guess is 4 months. Emphasis is on guess. 3. Having FOSSology use binary and/or source signatures during the analysis phase - I know we had a conversation on this a while back, but how far away is FOSSology from being able to use a library of existing signatures when doing comparisons .. Are you talking about discovery - finding out what packages are used, or code clone detection? I think we talked about clone detection before. There is lots of great research on code clone detection. One of the best tools is CCFinderX which is going to be open sourced soon. It is not really suitable for incremental detection like what we need. We need to compare a relatively small set of code to a large library of code to find code clones. CCFinderX, along with most other academic research (DECKARD comes to mind) finds clones within a single set. However, these could be modified. DECKARD (UC Davis) is also not freely available, but probably could be made so. My plan was a bit different though. My plan was to sponsor some research to get us a state of the art solution, for this type of detection, that we could plug into fossology. I put out an academic RFP and got many excellent proposals. At this time (several months ago) we had money in our budget to sponsor this and I awarded the gift grant ($20,000 USD). Unfortunately, this coincided with the market bust and we were not allowed to send the money. Do you have $20,000 (or $5k, $10k, $15k) you could donate to a University? You mentioned binary scanning. I also received a proposal from a couple of experts in binary license scans. Unfortunately, even though they have a great deal to benefit by having this in fossology, they would like to be paid for their efforts. Again, we have no money for this. So I'd dearly love to implement a state of the art clone detector and a super binary scanner, but it's just Mark and I writing code and it's hard to look past the next version (http://fossology.org/task_list). This is why I was looking for help from our academic friends and experts. If anyone has expertise, time and/or money who would like to support one or both these agents (or any other agent), please contact me. Bob Gobeille b...@fossology.org___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Licterm table, Phrase meaning and number of licenses for a file
Jonathan, What would help you? What do you need to do? Thanks, Bob Gobeille b...@fossology.org On Aug 31, 2009, at 8:56 AM, Laser, Mary wrote: Hi Jonathan, There is a discussion about License Phrases and interpreting license analysis in the fossology project documentation: http://fossology.org/how_to_interpret_the_license_analysis_report#license_phrases . Please take a look and see if this answers your questions. Thanks, Mary From: fossology-boun...@fossology.org [mailto:fossology-boun...@fossology.org ] On Behalf Of Jonathan Parès Sent: Friday, August 28, 2009 8:31 AM To: fossology@fossology.org Subject: [FOSSology] Licterm table, Phrase meaning and number of licenses for a file Dear all, I would like to know what is the exact meaning of the Phrase item inside the licterm table. It seems to correspond to the files where the license is described by a simple sentence. I am right ? Then I have noticed in the database (the licterm_name table) that a file can be associated to several licenses with different rates of matching (0, 1, 2), I am right too ? If yes it doesn't help me in my work ... Thanks for your replies, Jonathan -- Jonathan Parès, Open Source Software engineer, Qualipso project, O-Engine/GMRC, Open Source Competence Center (China), Guangzhou China. ATT1.txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] technical questions FOSSology
On Aug 19, 2009, at 7:20 AM, Armijn Hemel wrote: snip Just a plan full text search will find function names, text in printf's, and strings in binaries because everything looks like a string (in my idea of a simple initial implementation). In other words, the search won't distinguish between a function name and a string found elsewhere. They would all be treated as strings without any other meaning. For a first version of what I would need it for that would suffice. I Great. I think this would be a good step 1. can imagine that a future version would scan a C file according to a grammar and build an abstract syntax tree from it, where you could have more structure and grab things like function names, possibly including the order they appear in in the file. Absolutely. But there are other sources for this, like koders.com. What is the advantage to having this in fossology? It is very well possible that this does not have to be done in the database, but can be done 'on the fly'. Most C files I'd look at would not be extremely big. Sure creating an AST for a single file can be fast, and done on the fly. But wouldn't you then want to compare the symbols, etc in that file with a large body of code from the repo? This one to many similarity matching means all that data for the repo needs to be in the DB. You wouldn't want to create AST's (or PDG's or whatever) on the fly for the entire repo if your repo is big. Yes. I'm still not sure why it would be limited to English only, since that sounds a bit odd. If you mean ASCII, then it definitely is weird, because there are enough strings that are in other languages/character sets (the AUTHOR field for Linux kernel modules for example). There are language specific rules for natural language processing. This doesn't apply to pulling out proper names, symbols, classes, etc. I brought it up because your example was: This software is derived from the GNU GPL XviD codec and I thought it would be good to also find variations quickly (which require language specific stemming, and thesaurus): This project derives from the gnu gpl XVID codec and I was looking for a quick, easily implemented solution - like postgresql full text search. So I don't think proper names like in AUTHOR would be a problem. Non english phrases would simply not take advantage of language specific built-in features (stop words, stemming, etc). Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] technical questions FOSSology
On Aug 18, 2009, at 9:56 AM, Armijn Hemel wrote: dear FOSSology people, I have some technical questions regarding FOSSology. It basically comes down to the following: Is there a mechanism that given a string I can let FOSSology search a knowledgebase, and it can come up with a list of possible packages where this string can be found in? An example, say I want to search for the following string: This software is derived from the GNU GPL XviD codec and I have populated the FOSSology database with (at least) a copy of XviD sources, will it be possible that it will let me know in which file this can be found? If not, is this functionality planned? There is a way to do this in 1.1 (latest stable release) by defining license terms. From the top menu it is in Organize License Manage Terms. However, this is being deprecated in the next release. In the next release, it will be easy to add your own license and you could simply add This software is derived from the GNU GPL XviD codec as a license. This will be easy to do in 1.2. Another feature that I've been wanting to get to is an ad hoc string search but I'm not sure how useful it really is. Since the results won't be stored in the db. So do you want ad hoc string searches or an easy way to add a phrase, like you quoted, as a new license to look for? Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] technical questions FOSSology
On Aug 18, 2009, at 12:58 PM, Armijn Hemel wrote: Are you only interested in English strings? Well, I would be interested in random UTF-8 strings that are likely to appear in source code :-) These could be: * function names * printf/fprintf statements * other strings that are in source code and that have been copied into a binary I don't see why that should be restricted to just one language. It would be a lot harder to implement than plain UTF-8 searches. So, basically it comes down to full text searches. I'm not sure how well PostgreSQL can handle these (I think it can). The dictionary, stemmer and stopwords are language specific. Postgres can handle all of this. Just a plan full text search will find function names, text in printf's, and strings in binaries because everything looks like a string (in my idea of a simple initial implementation). In other words, the search won't distinguish between a function name and a string found elsewhere. They would all be treated as strings without any other meaning. My thought for the first implementation was something simple: 1) scan each file loaded into the repository and create a full text index based on english strings. 2) user can then do ad-hoc search for string. So searching on Xvid would find files containing functions named xvid, printf strings with XviD, XVID in language files, ... The search would return a list of files and some bytes around the string (something like grep). This simple implementation is much like a fast grep with stemming, stopwords, and thesaurus. Of course, lots of things could make this better. For example, dumping symbol tables and saving them as symbols in the string index (this would be easy for an agent to do). Or parsing code (programming language specific) to extract out function names, etc. That way you could limit your search to function names, class names, symbol names, etc. The next step would probably be to load in files of strings as symbols, text, function names, etc. and use that as a reference file, like we do for licenses. The search would find files based on that whole lot of criteria and return a ranked list of hits. Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Debugging fo_nomos Agent From Scheduler
On Aug 13, 2009, at 7:37 AM, Benson, Channing D wrote: Greetings, I've added (and checked in) some diagnostic code to the fo_nomos agent to help figure out why nothing seems to happen when it is invoked from the ui. When fo_nomos starts up, it touches the file /tmp/nomos_start. When it's run from the scheduler, it also touches /tmp/nomos_sched_run. What I'm seeing is that when you do a fossology-scheduler -t, those files do indeed get touched. But when you do a file upload and check the Schedule Nomos License Analysis checkbox, those files do not get touched, which indicates that even though it shows up in the job queue, the fo_nomos program is not getting run. Anyone have any clues as to why this is the case? Chan, Let's talk on IRC (#fossology) about this. Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Handling duplicate files from multiple uploads....
On Aug 13, 2009, at 10:45 AM, thomas.j.mur...@kodak.com wrote: We would like our product teams to submit their source code for open source analysis at periodic times throughout the development cycle (at least once per year for active projects). Most likely they will create a .tar for their source code directory and upload to our fossology server. We would like to avoid re-analyzing and reporting files that have already been reported from earlier runs. Can you recommended a method for handling this situation without burdening the product development teams too much (i.e. so they don't have to keep track of the files and somehow filter them for subsequent runs)? Hi Tom, I just wrote (yesterday) the sql to do this in v 1.2. Take two uploads - report the license changes. Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Handling duplicate files from multiple uploads....
I just added the pkg and archive diffs (numbers 9, 10): http://fossology.org/task_list I need to do a mockup for the file diffs. If you have something in particular in mind, please mock it up your self and send it to me. Bob Gobeille b...@fossology.org On Aug 13, 2009, at 11:18 AM, Laser, Mary wrote: Tom, Here is a mockup of how it will be reported http://fossology.org/~bobg/1.2reqs/pkghistory.html . Is this useful? Mary From: thomas.j.mur...@kodak.com [mailto:thomas.j.mur...@kodak.com] Sent: Thursday, August 13, 2009 11:15 AM To: Gobeille, Robert; Laser, Mary Cc: fossology@fossology.org Subject: Re: [FOSSology] Handling duplicate files from multiple uploads Bob and Mary, I was aware that reanalysis was not performed on duplicate files --- but how do we present the data so that the user isn't tracking down the same files that have been 'flagged' with a license. It looks like v 1.2 will indicate if there are license changes from one upload of the same file(s) to the next? I must have missed that in the features list how will it be reported? or is there a mockup screenshot of this capability? --- Basically, I just want the user to be able to zero in on the changes from the last upload relative to detected licenses. thanks Tom Murray ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Meeting agenda for 8-13-09
On Aug 13, 2009, at 7:14 AM, Ma, Dong (vinc...@gdcc-bj-most) wrote: First, I share my status of the Backup/restore task: 1. I have updated the Data backup and restore scope on our wiki page, and team you can review it and to see if you have more comments and questions For those needing the URL: http://fossology.org/task:define_test_implement_full_backup_recovery_process_for_rfo 2. These days I also do some investigate the code changes about to implement only gold files backup and get some questions. First, I list my consider about the code should changes: a. the unpack agent may should be some code changes to fit the backup solution Ununpack requires a switch so that it can unpack to a repository but not update the db. b. the UI code about browse the files which the unpack files not in the repository Any place a file is retrieved from the repository needs to check to make sure the file exists. If it does not, check to make sure the gold file exists. If it does, possibly ask the user if they want to recover from the gold, and if so, queue up a job to do the ununpack. c. I find filter_license and licinspect agent will need the unpack files All agents that read repository files need unpacked files, not just filter_license and licinspect. d. investigate how to distinguish the gold files and unpack files in the Database Could you be more specific? Do you mean given an upload_pk, find the file path to the gold file? If so: select pfile_pk from pfile,uploadtree where pfile_fk=pfile_pk and upload_fk=$Upload_pk and parent is null; once you have the pfile_pk, use RepPath() in common-repo.php to convert that into a complete file path. Second, I got two questions: Question a: What impact is on license analysis when unpacked files are not recovered? License analysis requires unpacked files. Question b: I cannot visit the phppgadmin web page of our production system, how I can get the entrance of phppgadmin? I want to investigate the database related to the repository files. If you need access to this specific system, let me know in a separate email. In general, to do investigations like this you should use your own fossology system. Or if your system does not use multiple agent machines, you might want to use another of our development systems and not a production system. This is my draft Backup/Restore plan: 1. team review the scope, if no more questions I'll do the design of the task 2. continue investigate the code changes of only gold files solution, and before this weekend give team a update about my investigate, and team help to review if I lost something 3. Design the Data backup and restore task and estimate the time(early of next week) 4. implement the code changes and backup strategy Please help to review this plan and any questions feel free send me a email. I'll take a look to meeting notes if I feel better tomorrow. Sounds good. I hope you are feeling better. Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] split this list?
When we started fossology we created this single general purpose mailing list for user questions, support, developers, ... A lot of the traffic lately has been developer oriented and an occasional support/help request has gone unanswered. Is it time to be thinking about splitting this into separate developer and support/help lists? Thanks, Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Problem wile uploading a winzip file
On Aug 12, 2009, at 1:04 PM, Laser, Mary wrote: -Original Message- From: fossology-boun...@fossology.org [mailto:fossology-boun...@fossology.org] On Behalf Of Abraham Bennis Sent: Wednesday, August 12, 2009 12:12 PM To: fossology@fossology.org Subject: [FOSSology] Problem wile uploading a winzip file I am having a problem while uploading a winzip file using the option Upload from Server . Please let me know how to resolve this problem. Thanks, Bennis Hi Bennis, I had a similar problem yesterday and it turned out my Scheduler.conf file was wrong. (If you have installed fossology 1.1 from a deb package, this file will reside in /etc/fossology/ Scheduler.conf.) Please send me your log file and your Scheduler.conf file. Also, what version of fossology are you running and what distro. Thanks, Mary To clarify this a bit - the problem that Mary is talking about refers to the question of what server does upload from server use? Sometimes people that use multiple agent machines put multiple host entries for fosscp-agent in Scheduler.conf. That's bad since Upload from server uses whatever server it is running on. However, if you are only running on a single machine, this doesn't apply to your case. If you have multiple agent machines, please look at your Scheduler.conf lines that refer to fosscp-agent. Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] unpack size/time
On Aug 11, 2009, at 11:27 AM, Dan Stangel wrote: One small clarification on the disk space: You said that we have additional space on the repo. We have unused space in the repository, but we do not have unallocated space in the SAN storage. In order to implement a full repo backup as you described, we would need to either shrink the allocated repo size by 1/2, or add twice as many disks as we currently have. I was talking about using the unused space in the mounted volumes (aka the repo). Blabby rsync's it's 91 GB onto bashful (/srv/fossology/repository/ bashful/blabby), bashful rsync's it's 93 GB onto blabby. Happy and sneezy do the same. Shifty and dopey do the same. You end up with a duplicate repo in the storage we are currently have mounted. So this is all using the same SAN storage? What is the point of the repo backup then? I'm guessing that we are using a raid level to support a single or double drive failure. Is that the case? What failure mode are you trying to protect from? Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Survey - for next version of FOSSology
On Aug 10, 2009, at 12:20 PM, Matt Taggart wrote: Are you saying you would rather move it to a separate package than remove it from the main package? I'm saying it should remain the the upstream fossology tarball, but for Debian I can put it in a separate package that won't be required (the fossology-agents package will only Recommends instead of Depends). BTW one other thing that needs to be fixed for this to happen is that I'm not sure the selftest agent is able to deal with 'missing' agents. Yes, selftest does need to be changed for this. There may be some other code that has a concept of default agents which would also need to change. In the future, selftest should read the db to find the agents to test. For now, it's easy to remove pkgmetagetta from selftest (so there would be no test for it). I'd still like to know if anyone is using pkgmetagetta. Are you saying that we should keep it because someone somewhere might use the data even though fossology doesn't? Yes. FOSSology is still sort of a one-trick-pony, but I think the metadata agent is something that fits into the bigger picture. I agree. We did it mostly as an example to show our potential. But now I'm thinking that we shouldn't have released it until we actually use the data. Right now it's a cost with no benefit unless people are doing direct db queries or are counting on a UI to use it in the future. The only feedback I've gotten so far is from folks who do not know what this agent does. So far we have the following options: 1) Do nothing. Leave as is. 2) Make pkgmetagetta a separate package (requires minor code changes and pkgmetagetta will never do selftest even if installed) 3) Leave packaging alone, but make same minor code changes as in 2, so that pkgmetagetta is NEVER run. I don't feel strongly about this but am asking for feedback to find out if users would like a change. Votes taggart: option 1 or 2 bobg: option 1, 2, or 3 ;-) Bob Gobeille b...@fossology.org ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Combining databases from several servers
On Jul 27, 2009, at 11:40 AM, thomas.j.mur...@kodak.com wrote: I believe your 3rd suggestion is what I'm leaning toward --- I have asked each product team to do a pg_dump to archive the database would be nice to then 'import/combine' them into a central database Hi Tom, I was hoping to find a postgresql migration utility that allows me to do the import/combine. So far, no luck. The only suggestion that has come up is to change our primary/foreign keys to prefix a server ID. There are different ways we could implement this, but code/ schema changes are needed. If you haven't added any licenses or agents, and you only want license data (not data from pkgmetagetta or specagent), and you throw away all the job queue history. I think the only keys you need to update are for folders, pfile, upload, uploadtree, and users. Some possibilities: Is this something you (or any programmer) are interested in doing yourself? You could hire HP Services, or a fossology developer, or anyone to make the changes. You could punt and simply reanalyze the code on your master server. You could manually tweek the primary keys (need to set the top 3 bits to make unique keys for your 7 servers). Then a normal import might work. You could write a program to rewrite the primary keys. You could wait and we can try and get this into v 1.3 You do realize that in addition to the db, you have to copy over your repository? That's a simple rsync but maybe a lot of data. Any other ideas? Bob___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Combining databases from several servers
On Jul 27, 2009, at 9:31 AM, thomas.j.mur...@kodak.com wrote: Hello all, Our company is attempting to scan/analyze all of our product software. Many locations throughout the world, some product teams are reluctant to use our central Fossology server so they have setup their own, So we have approximately 7 separate servers collecting OSS info. I'd like to aggregate all the various server data into one database so that it can be viewed via the Fossology UI (or other means).. any ideas on how to do this? Hello Thomas, There are three things to keep track of: 1) UI (this step is unnecessary if you do 3. below) the UI can handle a split repository, but not a split database. You could change each UI plugin to query the multiple databases and aggregate the results. 2) Split repository If each repository could be nfs mounted to each other, then Hosts.conf could specify them all. If any file in the DB can't be found in host/ repository 1, then it would look in host/repository 2, ... 3) Database (this is unnecessary if you do 1. above) Each site could share the same (replicated, or synched) database. The problem is how to combine the multiple databases without breaking referential integrity. I've asked the postgresql list for some help with this because it seems crazy that this hasn't already been addressed. Bob___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Unable to browse the repository and view the uploaded files
On Jul 20, 2009, at 12:27 AM, Lin Ramachandran wrote: When I try to browse the repository it doesn't show any thing. Please see the attached screenshot of browsing the repository and the the screenshot of Jobs-Queue-By Upload Hi Lin, Your job status screenshot shows that that the license job finished. The Browse screenshot shows you are browsing the top level folder Software Repository. Did you upload your files into this directory, or the one below it (Gallia)? If the latter, then you need to click on that folder to see it's contents. Bob Gobeille___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Error when executing cp2foss command using 1.1.1 RC5
Argh. I should have checked this before I told you to remove pkgmetagetta. The problem is that the scheduler does a self test against a hardcoded set of agents to make sure they are ALL in Scheduler.conf. By using - t you ran the self test. However, every time the scheduler starts, it runs that test itself. Sorry about this. I was looking for a quick fix and didn't verify it would actually work. So the options are: 1) You can change the source code (I can send you new versions of the file that needs to change). If you want to do this, you will have to do a make; sudo make install of the scheduler on your system. Are you comfortable with that? 2) We go back to figuring out why you are having pkgmetagetta problem in the first place. Which would you prefer? Bob Gobeille On Jul 20, 2009, at 11:56 AM, cloud...@yahoo.com wrote: 2009-07-20 13:53:32 scheduler[28452] : FATAL: Agent type 'pkgmetagetta' not in Scheduler.conf. 2009-07-20 13:53:32 scheduler[28452] : FATAL: Self Test failed. Inconsistent agent(s) detected. Exiting. --- On Mon, 7/20/09, Bob Gobeille bob.gobei...@hp.com wrote: From: Bob Gobeille bob.gobei...@hp.com Subject: Re: [FOSSology] Error when executing cp2foss command using 1.1.1 RC5 To: cloud...@yahoo.com cloud...@yahoo.com Cc: fossology@fossology.org, Mary Laser mary.la...@hp.com Date: Monday, July 20, 2009, 12:14 PM If the scheduler dies, the fo_watchdog should restart it, but it can take upto 10 minutes. Since this problem is more than 10 minutes old, I'm guessing that fo_watchdog keeps starting the scheduler, and the scheduler keeps dying. You can test the scheduler with the command: fossology-scheduler -TL stdout ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Missing Files without license information
On Jul 17, 2009, at 9:08 AM, Stefan Walenda wrote: Hello *, There is the possibility in licence Tree View to show the results in CVS file. But I miss the files which havo no license informations. Is there a proper function in Fossology or does anybody know the SQL statement for the database that could solve this problem, so I could look up those files. RTFM? http://fossology.org/task_list We are planning this for version 1.2 but that probably won't be out for a few months. We are just finishing release 1.1 but I can try to give you some sql to do it after I get some other tasks done. Or if you know SQL, you can always do it yourself. Or maybe someone on the list can do it for you. http://fossology.org/schemaspy/ Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Error when executing fossinit using 1.1.1 RC5
That should have worked in 1.1 as well and I'll fix that. I wasn't thinking about everything that needs to happen when you update a license. I don't know if I can fix today. I'm in meeting almost the whole day. Bob Gobeille Sent from my iPhone On Jul 16, 2009, at 5:43 AM, cloud...@yahoo.com cloud...@yahoo.com wrote: The instructions I used before were the following which worked in 1.0.0. Place your file under /usr/local/share/fossology/agents/licenses/. Run 'sudo fossinit'. The ”…” sequence indicates that the licenses are being installed. --- On Wed, 7/15/09, Bob Gobeille bob.gobei...@hp.com wrote: From: Bob Gobeille bob.gobei...@hp.com Subject: Re: [FOSSology] Error when executing fossinit using 1.1.1 RC5 To: cloud...@yahoo.com cloud...@yahoo.com Cc: fossology@fossology.org fossology@fossology.org Date: Wednesday, July 15, 2009, 6:52 PM On Jul 15, 2009, at 1:28 PM, cloud...@yahoo.com wrote: r...@rh licenses]# /usr/bin/fossinit `pwd` You were following http://fossology.org/how_to_add_a_license_template_or_license_phrase_and_re-analyze_licenses right? Using fossinit to to rebuild License.bsam seems like a terrible idea. We used to have a mkcache script to recreate License.bsam but it looks like it has been replaced by something and I don't recall what. I need to look into this, update the wiki page, and get back to you. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Error when executing fossinit using 1.1.1 RC5
On Jul 15, 2009, at 1:28 PM, cloud...@yahoo.com wrote: r...@rh licenses]# /usr/bin/fossinit `pwd` You were following http://fossology.org/how_to_add_a_license_template_or_license_phrase_and_re-analyze_licenses right? Using fossinit to to rebuild License.bsam seems like a terrible idea. We used to have a mkcache script to recreate License.bsam but it looks like it has been replaced by something and I don't recall what. I need to look into this, update the wiki page, and get back to you. Bob Gobeille___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Delete files
On the top menu click on Organize Uploads Delete to delete single uploads. Use Organize Folders Delete folder to delete a folder and all the uploads in that folder. There isn't an option to select a list of folders to delete or to delete ALL uploads. Bob Gobeille Sent from my iPhone On Jul 14, 2009, at 6:47 AM, heinz.h.hi...@daimler.com heinz.h.hi...@daimler.com wrote: Hello, We have a lot of single files uploaded into fossology. We have currently about 200 single files on the system and there are in different folders. And now we don't need them any longer and would like to delete them. Do you have any idea how we can delete all of them immediatley? Regards, Heinz If you are not the intended addressee, please inform us immediately that you have received this e-mail in error, and delete it. We thank you for your cooperation. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Stuck on Licensing
On Jul 14, 2009, at 10:06 AM, Landon Jurgens wrote: Here's what I got from the command date; ps -ef | grep sched Tue Jul 14 04:02:29 EDT 2009 landon 11652 11626 0 04:02 pts/000:00:00 grep sched fossy13841 1 5 Jun30 ? 18:56:34 /usr/lib/fossology/fossology-scheduler -d I think 18:56:34 NOT EQUAL 04:04:29 so the thing is hung. When I restarted it I got all this garbled junk WARNING: nonstandard use of escape in a string literal LINE 2: ', record_update=now(), agent_param='akey=\244063\ a= \480... ^ HINT: Use the escape string syntax for escapes, e.g., E'\r\n'. WARNING: nonstandard use of escape in a string literal LINE 2: ', record_update=now(), agent_param='akey=\244069\ a= \087... ^ HINT: Use the escape string syntax for escapes, e.g., E'\r\n'. WARNING: nonstandard use of escape in a string literal LINE 2: ', record_update=now(), agent_param='akey=\244071\ a= \2A3... ^ HINT: Use the escape string syntax for escapes, e.g., E'\r\n'. These warnings are ok. We have most of them cleaned up in 1.1. They will slow things down because of all the logging, but won't hurt the data. You could change you postgres logging level, in postgresql.conf, to ignore them, if you wish. I didn't know if this made any sense. I'm looking to install 1.1.0 but there isn't an Ubuntu Package for it yet but I think I might try to make one. I think the debian package will probably work. See http://fossology.org/download There are three fossology developers who have been making packages (Bruno, Vincent, and Matt). I'm pretty sure Bruno and Vincent will be making Ubuntu packages, but they may not get to it for another week or so. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Delete files
Mary mentioned delagent. You can try delagent -F 1 if you want to delete ALL uploads. Bob On Jul 14, 2009, at 7:18 AM, Gobeille, Robert wrote: On the top menu click on Organize Uploads Delete to delete single uploads. Use Organize Folders Delete folder to delete a folder and all the uploads in that folder. There isn't an option to select a list of folders to delete or to delete ALL uploads. Bob Gobeille Sent from my iPhone On Jul 14, 2009, at 6:47 AM, heinz.h.hi...@daimler.com heinz.h.hi...@daimler.com wrote: Hello, We have a lot of single files uploaded into fossology. We have currently about 200 single files on the system and there are in different folders. And now we don't need them any longer and would like to delete them. Do you have any idea how we can delete all of them immediatley? Regards, Heinz If you are not the intended addressee, please inform us immediately that you have received this e-mail in error, and delete it. We thank you for your cooperation. ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Scheduler Status page
On Jul 14, 2009, at 12:46 PM, Laser, Mary wrote: Yes, there are a handful of open bugs that will not get fixed for 1.1. (At least 3 I know of, are license identification issues that should be addressed in 1.2.) As Bob suggests, we should add these to the Known Issues section of the release notes http://fossology.org/release_notes#known_issues . Is someone volunteering to do this? Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] Fwd: Scheduler hangs
FYI, this is a conversation Neal and I were having that should have gone to the list instead of only me. Forwarded here in case anyone wants to join in. Bob Gobeille Begin forwarded message: From: Bob Gobeille bob.gobei...@hp.com Date: July 14, 2009 1:12:42 PM MDT To: Dr. Neal Krawetz ne...@fossology.org Subject: Re: Scheduler hangs On Jul 14, 2009, at 1:13 PM, Dr. Neal Krawetz wrote: Hi Bob, I just had a thought about detecting and debugging scheduler hangs, as well as a possible workaround. The last I knew... the bug was a signal handling conflict between the DB library and the scheduler. Right now, the scheduler intercepts signals. When sigaction() is called in scheduler.c, there is an optional parameter for storing the old signal handler. I currently ignore the old handler value since I am hijacking the signal handler and have no intention of putting it back. In some new code I added (when testing agents), I save the old sig, set SIG_IGN, and then restore it after giving the agents enough time to die and send their SIG_CHLD. Without this we were reporting unexpected child deaths, when in fact, we wanted to ignore those deaths. However... Try this: - Save the old signal handler. In the scheduler's signal handler, call the original handler before exiting. (if old is not null then call old with same parameters that my signal handler received) - I also set a few signals to SIG_IGN. Instead of ignoring them, create a new signal handler that receives them, does nothing, and calls the old handler. - Occasionally (in the main loop, after the sleep) check to see if the handler is still set the way I set it. If it isn't, then it means that Postgres hijacked my handler. I suspect that Postgres is hijacking interrupts. Between my hijacking and their hijacking, things are getting messed up. If this turns out to be the case, then there is a solution: - Before hijacking any handlers: call the DB and give it a few simple exercises. This will force the SQL library to hijack any interrupts it wants. - Then configure the scheduler's interrupts, holding onto the old handlers. - Have the scheduler pass though all signals to the old handler. Good ideas. I wrote the scheduler watchdog as a hack because the hangs/dies are rare and I thought that diagnosing the problem would be a bitch. Actually, diagnosing the problem would be a good thing to do. ;-) Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Fossology 1.1.0 RC5 installation problem: Failed to load default extractor libraries
On Jul 14, 2009, at 3:01 PM, 西门烧雪 Simon wrote: I am installing Fossology 1.1.0 RC5 on Linux RH 5.1 (see results of 'uname -a' below) Linux rhle51x86 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:02 EDT 2007 i686 i686 i386 GNU/Linux I used yum to install Fossology. Everything went smoothly. Except that the following error: FATAL: Failed to load default extractor libraries. FATAL: '/usr/lib/fossology/agents/pkgmetagetta -i' failed to initialize I do have libextractor installed from a RPM. [r...@rhle51x86 lib]# ls -l /usr/lib/libextractor* lrwxrwxrwx 1 root root21 Jul 14 16:55 /usr/lib/libextractor - libextractor.so.1.1.1 lrwxrwxrwx 1 root root28 Jul 14 16:40 /usr/lib/libextractor_common.so.0 - libextractor_common.so.0.0.0 -rwxr-xr-x 1 root root 46272 May 29 20:47 /usr/lib/libextractor_common.so.0.0.0 lrwxrwxrwx 1 root root21 Jul 14 16:40 /usr/lib/libextractor.so.1 - libextractor.so.1.1.1 -rwxr-xr-x 1 root root 45276 May 29 20:47 /usr/lib/ libextractor.so.1.1.1 you also need libextractor-devel libextractor is only used by the pkgmeta agent. So if you are only doing license analysis, you could simply skip (probably should skip) running the pkgmettagetta agent. You could also skip the specfile agent. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] New Release Fossology 1.1.0
On Jul 7, 2009, at 10:45 AM, Abraham Bennis wrote: If I could download now then I could also test the installation. Super. Mark, as soon as you tag rc4 could you send the install info to Abraham and the list. Thanks, Bob Gobeille___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Upload and Analyze of Zip/tar.bz2 files
Hello Heinz, The problem with uploading zip files is fixed in version 1.1: http://www.mail-archive.com/fossology@fossology.org/msg00422.html Bob Gobeille On Jul 6, 2009, at 9:04 AM, heinz.h.hi...@daimler.com wrote: Hello, I have managed to install fossology on my computer ;-) Now I am keen on using this tool. It seems to me that I can just upload and analyse singel files, e.g. single dot-c files or single dot-cpp files. On the webpage http://fossology.org/overview_of_the_user_interface I have seen screenshots including dot-zip files and dot-tar.bz2 files. When I started to upload a zip file I could only see the binary content. And then of course I couldn't analyse the content of the zip file. How can I upload and analyse them? Best Regards, Heinz If you are not the intended addressee, please inform us immediately that you have received this e-mail in error, and delete it. We thank you for your cooperation. ATT1.txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Error in PHP Test Installation
Hi Severine,
This looks like a mistake in test/Install.php. I'd just change lines
51- 58 in Install.php.
From:
if (is_readable('/etc/fossology/Proxy.conf')) {
$cmd = . /etc/fossology/Proxy.conf; .
wget -q 'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
}
else if (is_readable('/usr/local/etc/fossology/Proxy.conf')) {
$cmd = . /etc/fossology/Proxy.conf; .
wget -q 'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
}
To:
$cmd = wget -q 'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
I'll file a bug so it gets fixed asap.
Bob Gobeille
On Jun 18, 2009, at 7:44 AM, Séverine Scheidt wrote:
Hi,
I'm just starting to make use of the fossology PHP UI tests, but I
had trouble doing the installation. The first execution of the
fossology/tests/Install.php failed providing the following message:
installing fo-runTests into /usr/local/bin
Check to see if simpletest is installed in /usr/local
Attempting to download and install simpletest into /usr/local
.: 1: Can't open /etc/fossology/Proxy.conf
ERROR! problem with downloading simpletest, need a proxy?
After looking at the Install.php, I found out that the Path in line
56 is not right, as you can see from my svn diff:
Index: tests/Install.php
===
--- tests/Install.php (revision 2252)
+++ tests/Install.php (working copy)
@@ -1,5 +1,5 @@
#!/usr/bin/php
-?php
+?php
/***
Copyright (C) 2008 Hewlett-Packard Development Company, L.P.
@@ -53,7 +53,7 @@
wget -q 'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
}
else if (is_readable('/usr/local/etc/fossology/Proxy.conf')) {
-$cmd = . /etc/fossology/Proxy.conf; .
+$cmd = . /usr/local/etc/fossology/Proxy.conf; .
wget -q 'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
}
if(chdir('/usr/local/')) {
Best regards,
Severine Scheidt
--
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Error in PHP Test Installation
Severine,
My mistake (the fix below). Sorry I didn't look at the code closely.
The best thing to do is probably to create a dummy Proxy.conf file.
I've filed the bug.
Bob
On Jun 18, 2009, at 8:03 AM, Gobeille, Robert wrote:
Hi Severine,
This looks like a mistake in test/Install.php. I'd just change lines
51- 58 in Install.php.
From:
if (is_readable('/etc/fossology/Proxy.conf')) {
$cmd = . /etc/fossology/Proxy.conf; .
wget -q
'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
}
else if (is_readable('/usr/local/etc/fossology/Proxy.conf')) {
$cmd = . /etc/fossology/Proxy.conf; .
wget -q
'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
}
To:
$cmd = wget -q
'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
I'll file a bug so it gets fixed asap.
Bob Gobeille
On Jun 18, 2009, at 7:44 AM, Séverine Scheidt wrote:
Hi,
I'm just starting to make use of the fossology PHP UI tests, but I
had trouble doing the installation. The first execution of the
fossology/tests/Install.php failed providing the following message:
installing fo-runTests into /usr/local/bin
Check to see if simpletest is installed in /usr/local
Attempting to download and install simpletest into /usr/local
.: 1: Can't open /etc/fossology/Proxy.conf
ERROR! problem with downloading simpletest, need a proxy?
After looking at the Install.php, I found out that the Path in line
56 is not right, as you can see from my svn diff:
Index: tests/Install.php
===
--- tests/Install.php (revision 2252)
+++ tests/Install.php (working copy)
@@ -1,5 +1,5 @@
#!/usr/bin/php
-?php
+?php
/***
Copyright (C) 2008 Hewlett-Packard Development Company, L.P.
@@ -53,7 +53,7 @@
wget -q
'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
}
else if (is_readable('/usr/local/etc/fossology/Proxy.conf')) {
-$cmd = . /etc/fossology/Proxy.conf; .
+$cmd = . /usr/local/etc/fossology/Proxy.conf; .
wget -q
'http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz'
;
}
if(chdir('/usr/local/')) {
Best regards,
Severine Scheidt
--
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01
___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Installation and download of Fossology
On Jun 15, 2009, at 6:55 AM, heinz.h.hi...@daimler.com wrote: I am looking for the binary files of fossology. If I use the apt packages configuration for Debian, I get a mistake. I have added the following lines into /etc/apt/sources.list deb http://fossology.org/debian/ ./ Then I used “apt-get update and apt-get install fossology” on the command line, but it did not work. Does anybody know what I have to do to download and install the binaries? Hello Heinz, What error do you get when you do the apt-get install? You might also be interested to know: The default install locations for fossology files are in: http://fossology.org/file_locations There will be a new release (v 1.1) in probably less than two weeks. The 1.1 release notes are in: http://fossology.org/release_notes Bob Gobeille___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] [fossology] Is there some benchmark information?
On Jun 8, 2009, at 7:07 PM, Matt Taggart wrote: Hi Alice, It's been talked about a couple of times on the mailing list: http://fossology.org/pipermail/fossology/2009-June/000980.html http://fossology.org/pipermail/fossology/2009-May/000957.html The problem is that it is very memory and processors dependent and data dependent. We are hoping for a 10X speedup in 1.2. You mean you are hoping the new license agent, which uses a different method, will do the scan in 1/10th the time. But the bSAM agent, which uses a very thorough method that may or may not find more than the new agent, will still take the same amount of time. Right? Yes, exactly. Thanks for the clarification, Matt. Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] [fossology] Is there some benchmark information?
Hi Alice, It's been talked about a couple of times on the mailing list: http://fossology.org/pipermail/fossology/2009-June/000980.html http://fossology.org/pipermail/fossology/2009-May/000957.html The problem is that it is very memory and processors dependent and data dependent. We are hoping for a 10X speedup in 1.2. Bob Gobeille On Jun 8, 2009, at 3:44 AM, Alice Liu wrote: Hi all, Is there some benchmark information about fossology? Such as, upload different package, the runtime and memory information? or by different computer, the performance information? Thanks:) Best Regards, Alice Liu ATT1.txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Fossology Performance
100 - 300 GB? I assume you mean 100 - 300 MB, or 1-3 GB??? License analysis times vary a lot because it depends on the size of the unpacked files (large files are slower/byte than small files), and files already in the repository from a previous upload don't need to be reanalyzed. One of our installations uses an HP blade server with 6 blades dedicated for agents. That's a total of 32 cores, and each blade has from 12-16 GB RAM. The Fedora 9 DVD iso (5.2 M files, 3.68 GB) took (I think) 2-3 days to do the license analysis. Yes, the performance is pretty bad. This is why we are doing a new license analyzer for the 1.2 release (actually two license analyzers). Our target it do an iso like this in under 24 hrs on a single processor. So far the timings look promising but we are not quite ready to time a very large upload (we are making the analyzers into agents now). Bob Gobeille On Jun 2, 2009, at 11:12 AM, Abraham Bennis wrote: License analysis is taking days to complete the process. We usually upload files of size between 100GB to 300GB. We are running this application on a Virtual Machine with 2 CPU's and 4 GB of RAM. If I am uploading a file of size 300GB, it should complete the license analysis in a day. Please let me know the ideal hardware requirements for this application. __ Thanks, Bennis Abraham ATT1.txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Fw: Question
Daniel, Do you have this resolved yet? If so - great. If not, it looks like your fossology install did not go well. You could either reinstall, or if you want to debug this further, I have a couple of questions: 1) What happens when you access the user interface? Do you get any errors? 2) What does the file pathinclude.php look like? In a default install this will be in /usr/share/fossology or /usr/local/share/ fossology. In your case I'm not sure where it would be. You can find it by typing: cd /srv/fossology find . -name pathinclude.php Thanks, Bob Gobeille On May 27, 2009, at 9:40 AM, daniel.goe...@clearstream.com wrote: Hi Mark, I checked the points that you mentioned below, please find the answers: 1. yes, cp2foss is installed in /srv/fossology/bin 2. here is what I find under /srv/fossology/ [r...@reikland fossology]# pwd /srv/fossology [r...@reikland fossology]# ls -l total 80 drwxr-sr-x 2 root fossy 4096 Feb 26 18:24 bin -rw-r--r-- 1 root fossy13402 Apr 23 12:17 cp2foss.php drwxr-sr-x 5 root fossy 4096 Feb 26 18:43 etc drwxr-sr-x 4 root fossy 4096 Feb 26 18:24 include drwxr-sr-x 5 root fossy 4096 Feb 26 18:24 lib -rw-r--r-- 1 fossyfossy22593 Feb 26 18:40 list drwxr-sr-x 3 root fossy 4096 Feb 26 18:06 man drwx-- 11 postgres postgres 4096 Mar 17 15:27 pg_data drwxrws--- 4 fossyfossy 4096 Mar 18 09:06 repository drwxr-sr-x 7 root fossy 4096 Feb 26 18:24 share drwxr-sr-x 3 root fossy 4096 Feb 26 18:22 src drwxr-sr-x 3 root fossy 4096 Feb 26 18:43 srv [r...@reikland fossology]# 3. I do not remember where I was when I ran the command but I tried from the two locations with the same error message. The two versions are located in: -bash-3.2$ pwd /srv/fossology/bin -bash-3.2$ ls -l cp2foss -rwxr-xr-x 1 root fossy 13051 Feb 26 18:24 cp2foss -bash-3.2$ -bash-3.2$ pwd /srv/fossology/srv/fossology/bin -bash-3.2$ ls -l cp2foss -rwxr-xr-x 1 root fossy 13051 Feb 26 18:43 cp2foss I am not an expert in Linux and I am not the person who installed Fossology but when I check the installation time they are very close. 4. There are a few differences oident is replaced by fossy and www- data by apache (see below) drwxrws--- 3 fossy fossy 4096 2009-05-04 21:25 fossology/ drwxrws--- 4 fossy fossy 4096 2009-05-11 14:22 repository drwxrws--- 5 fossy fossy 4096 2009-05-11 09:21 localhost drwxrws--- 3 fossy fossy 4096 2009-05-05 18:13 ununpack drwxrws--- 219 apache fossy 4096 2009-05-05 18:41 files drwxrws--- 16 apache fossy 4096 2009-05-05 18:43 gold drwxrws--- 205 fossy fossy 4096 2009-05-05 18:53 license Thank you for your support and best regards, Daniel. - Forwarded by Daniel Goebel/CBF/GDB on 27/05/2009 17:14 - Mark Donohoe mark.dono...@hp.com 11/05/2009 23:37 Please respond to mark.dono...@hp.com To mark.dono...@hp.com cc daniel.goe...@clearstream.com, fossology@fossology.org fossology@fossology.org Subject Re: [FOSSology] Question Mark Donohoe wrote: daniel.goe...@clearstream.com wrote: Dear user group, we have installed the patch but I am still not able to load a complete directory. I have also tried to use the command line cp2foss but I am also having some issues, please see below the syntax of the command line and the error message. Any help is more than welcome. /srv/fossology/bin/cp2foss -f Software Repository/Folder_dg -d Test spantax -n Test spantax /opt/fossy/SPANTAX/SPANTAX/appl/ script PHP Warning: Call-time pass-by-reference has been deprecated - argument passed by value; If you would like to pass it by reference, modify the declaration of menu_insert_r(). If you would like to enable call-time pass-by-reference, you can set allow_call_time_pass_reference to true in your INI file. However, future versions may not support this any longer. in /srv/fossology/share/fossology/www/common/common-menu.php on line 244 PHP Notice: Undefined index: updcache in /srv/fossology/share/fossology/www/plugins/ui-license.php on line 114 PHP Notice: Undefined offset: 1 in /srv/fossology/share/fossology/www/common/common-parm.php on line 98 PHP Notice: Undefined variable: PageHex in /srv/fossology/share/fossology/www/plugins/ui-view.php on line 92 PHP Notice: Undefined variable: PageText in /srv/fossology/share/fossology/www/plugins/ui-view.php on line 93 PHP Notice: Undefined variable: PageText in /srv/fossology/share/fossology/www/plugins/ui-view.php on line 94 Loading /opt/fossy/SPANTAX/SPANTAX/appl/script tar: /cp2foss-4a081eff990ed.tar: Cannot open: Permission denied tar: Error is not recoverable: exiting now FATAL: '/cp2foss-4a081eff990ed.tar' does not exist. Best regards, Daniel Goebel. Daniel, Sorry the patch as not fixed your problem. I'm looking into it further now. I'll get back with you as soon as I can. Daniel, I have some questions as I have not been
Re: [FOSSology] RPM Spec Recommendations
On May 27, 2009, at 10:55 PM, Jeff Sheltren wrote: On May 15, 2009, at 1:27 PM, Jeff Sheltren wrote: The only build requirement that I can see which isn't in EL5 base or in EPEL is libextractor-devel. I rebuilt a fedora libextractor package in order to get the fossology build working -- if you feel Would it be a good idea reduce the overall build complexity by splitting each agent into its own install package? For example, libextractor is only used by the pkgmetagetta module, which most people probably don't care about anyway. We are probably going to have optional modules (agents + UI) in the future anyway. We have talked about this in the past but perhaps we need to do something about it. IF this is a good idea, the first thing I would do is make pkgmetagetta optional. Optional modules is also seems like a good way to introduce most new functionality since the new code will probably be less stable than the base code. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] RPM Spec Recommendations
Hi George, This is why the Linux Standard Base (LSB) was created: http://www.linuxfoundation.org/collaborate/workgroups/lsb The biggest problem we have is with different ways to do packaging (apt, rpm, yum) and the fast and distributed release cycle across the distros. Bob On May 28, 2009, at 9:22 AM, george.p...@prudential.com wrote: Bob (and everyone else on the list) I wanted to take this opportunity and thank the team for making such great strides getting FOSSology to the point of an RPM install. I am looking for to the official release After some thought, I think this would be a GREAT case study to demonstrate why Linux needs to have certain standards in place that are COMMON across all distributions.The efforts that were spent on getting FOSSology running on another distribution of Linux (from Debian to Redhat / Centos) should help open some eyes that this really is a problem From: Bob Gobeille bob.gobei...@hp.com To: Jeff Sheltren j...@osuosl.org Cc: fossology@fossology.org Date: 05/28/2009 11:01 AM Subject:Re: [FOSSology] RPM Spec Recommendations Sent by:fossology-boun...@fossology.org On May 27, 2009, at 10:55 PM, Jeff Sheltren wrote: On May 15, 2009, at 1:27 PM, Jeff Sheltren wrote: The only build requirement that I can see which isn't in EL5 base or in EPEL is libextractor-devel. I rebuilt a fedora libextractor package in order to get the fossology build working -- if you feel Would it be a good idea reduce the overall build complexity by splitting each agent into its own install package? For example, libextractor is only used by the pkgmetagetta module, which most people probably don't care about anyway. We are probably going to have optional modules (agents + UI) in the future anyway. We have talked about this in the past but perhaps we need to do something about it. IF this is a good idea, the first thing I would do is make pkgmetagetta optional. Optional modules is also seems like a good way to introduce most new functionality since the new code will probably be less stable than the base code. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] debugging tips
Here are some tips for debugging fossology jobs: 1) Check fossology.log for errors 1.1) You might also want to run the scheduler interactively and in verbose mode: /usr/local/lib/fossology/fossology-scheduler -k to kill the scheduler then /usr/local/lib/fossology/fossology-scheduler -v to start it interactively in verbose mode (note -h will show the scheduler usage) 2) On the Show Jobs detail report, click on the Job id. It looks on the left of the report under Job/Dependency. For example, if the Job/Dependency shows 1400/1399 you would click on the 1400. This will show you the job queue record and one of the fields is jq_endtext. After a successful agent run, you will see Completed: All data processed in that field. But if there is an error, the error should be reported. Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] FW: Error while uploading
Hi Bennis, Off the top of my head, a few things come to mind: 1) your scheduler is in some loop Do a top and see if fossology_scheduler is using lots of cpu time. and/or 2) you need to do a vacuum analyze on your database and/or 3) you are memory starved. and/or 4) your postgresql statement timeout is set very low In postgresql.conf the statement_timeout variable and/or 5) your disk performance is poor (perhaps due to other processes) Bob Gobeille On May 22, 2009, at 9:37 AM, Laser, Mary wrote: From: Abraham Bennis [mailto:abrahamben...@johndeere.com] Sent: Thursday, May 21, 2009 12:32 PM To: Laser, Mary Subject: RE: Error while uploading Thanks, I changed the permission as recommended. Now I could upload the files. I am able to run a license analysis for small file but I am getting another error now. Please see details below and let me know how to proceed. # ERROR: DBaccess(7): ERROR: canceling statement due to statement timeout ERROR: DBaccess timeout: 'SELECT DISTINCT(pfile_pk) as Akey, pfile_sha1 || '.' || pfile_md5 || '.' || pfile_size AS A FROM uploadtree,pfile WHERE uploadtree.pfile_fk = pfile.pfile_pk AND pfile_mimetypefk is NULL AND upload_fk = '38' LIMIT 5000;' WARNING: scheduler is running slow. It took 813 seconds to check agent status: Thu May 21 09:13:39 2009 ERROR: DBaccess(7): ERROR: canceling statement due to statement timeout ERROR: DBaccess timeout: 'SELECT DISTINCT(pfile_pk) as Akey, pfile_sha1 || '.' || pfile_md5 || '.' || pfile_size AS A FROM uploadtree,pfile WHERE uploadtree.pfile_fk = pfile.pfile_pk AND pfile_mimetypefk is NULL AND upload_fk = '38' LIMIT 5000;' WARNING: scheduler is running slow. It took 599 seconds to check agent status: Thu May 21 09:23:38 2009 ERROR: DBaccess(7): ERROR: canceling statement due to statement timeout ERROR: DBaccess timeout: 'SELECT DISTINCT(pfile_pk) as Akey, pfile_sha1 || '.' || pfile_md5 || '.' || pfile_size AS A FROM uploadtree,pfile WHERE uploadtree.pfile_fk = pfile.pfile_pk AND pfile_mimetypefk is NULL AND upload_fk = '38' LIMIT 5000;' ERROR: DBaccess(7): ERROR: syntax error at or near . at character 323 Thanks, Bennis From: Laser, Mary [mailto:mary.la...@hp.com] Sent: Monday, May 18, 2009 11:04 PM To: Abraham Bennis; fossology@fossology.org Subject: RE: Error while uploading Hi Bennis, In an earlier email (attached) you mentioned you were using a multi- system setup. Is this still the case? This is a VERY important distinction. Please open the attached email, follow the instructions therein and let me know your result. Thanks, Mary From: Abraham Bennis [mailto:abrahamben...@johndeere.com] Sent: Monday, May 18, 2009 10:38 AM To: Laser, Mary; fossology@fossology.org Subject: Error while uploading I am not able to upload files for license analyzing. Please see the error below. At the same time I am not seeing the folder called gold files. Please advice. -- ERROR upload 12 Failed to import file into the repository (RepImport=1). LOG upload 12 Failed to import /tmp/phpwnzvRE- uploaded from /tmp/phpwnzvRE-uploaded into gold 8B9DEC2B11954542F0477B0753508DA1AE1D25CE. 2BD616FFA4D43BCE5CB9C8BCD294B6AB.247850297 The file has been uploaded. It is upload #12. # ls /tmp ossology-configs.A14814 fossology-configs.sG2988 random.vintelaflush fossology-configs.A14980 fossology-configs.T20759 updatecron.bak fossology-configs.Ah7621 fossology-configs.tT7567 vmware-config0 fossology-configs.b17067 fossology-configs.v21272 vmware-config1 fossology-configs.c11086 fossology-configs.w22920 vmware-config2 fossology-configs.II2844 fossology-configs.y25373 fossology-configs.M17024 fossology-configs.Z16985 Thanks, Bennis ATT1.txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] some usage question about fossology
Hi Alice, On May 21, 2009, at 5:46 AM, Alice Liu wrote: Hi Bob, I can uploaded some package successfully, but when my package has one .pdf file, it can't upload... my case is: I uploaded one package successfully and analysis successfully, but when I added one pdf file in this package, reupload, it did not upload successully. In upload page and fossolog.log, it did not show any message, I checked 'Admin-Database-check' in my page, it did not show any message. in 'jobs-queue-summary', it did not have any... I can't tell what's going on here. Could you run ununpack standalone with just a couple of files (including the offending .pdf): ununpack -vRC -d myunpackdir mypkg and send me the output? The ununpack arguments above are: -v verbose -R recursive unpack -C continue on failures -d writes the unpacked files into directory myunpackdir mypkg is the package you are trying to unpack. Reason for using upper case: From your former description, we can use mime-type, metadata, and spec file analysis to analysis some file information that will not related with license analysis, but I want to compare this three analysis and report to my colleague...In my test,when I uploaded one common package, I find if I only choose license analysis, it also has [meta] link. If I choose all analysis(total 4 optional analysis), it show the same contents with the formerI want to use some test case to check... The [meta] link always shows up, even if you have not requested a meta analysis. Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] [Fossology]I can't login fossology, please help!
Hi Alice,
I'm glad this is working. Is your apache web server group (www-
data ?) a member of the fossy group?
The Db.conf file should be readable by the fossy group.
Bob
On May 11, 2009, at 1:01 AM, Alice Liu wrote:
Hi Bob,
Thanks for your helping:)
I can log in fossology now!
The reason is fossology does not connect postgresql.
I modified /usr/local/share/fossology/www/plugins/core-db.php,
remove: $this-_pg_conn = pg_pconnect(str_replace(;, ,
file_get_contents($path))); and add: $this-_pg_conn =
pg_pconnect(dbname=fossology user=fossy);
function db_init($Options=)
{
global $DATADIR, $PROJECT, $SYSCONFDIR;
if (isset($this-_pg_conn)) { return(1); }
$path=$SYSCONFDIR/$PROJECT/Db.conf;
if (empty($Options))
{
$this-_pg_conn = pg_pconnect(dbname=fossology user=fossy);
//$this-_pg_conn = pg_pconnect(str_replace(;, ,
file_get_contents($path)));
}
else
{
$this-_pg_conn = pg_pconnect(str_replace(;, , $Options));
}
if (!isset($this-_pg_conn)) return(0);
$this-Error = 0;
return(1);
}
___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] [FOSSology-commits] SF.net SVN: fossology:[2049] trunk/fossology/utils/installagents
On Apr 30, 2009, at 9:39 PM, Matt Taggart wrote:
Added: trunk/fossology/utils/installagents
===
--- trunk/fossology/utils/installagents
(rev 0)
+++ trunk/fossology/utils/installagents 2009-04-29 21:24:55 UTC
(rev 20
49)
@@ -0,0 +1,21 @@
+#!/bin/bash
+# After you build a fossology tarball with fo-mktar, you can use
+# this script to copy the fossology tarball to the multiple agent
machines
+# specified in the for loop.
+# It then untars, builds and installs fossology on those machines.
+# This does not clean up the unpacked tarball on the agent machines.
+
+# Usage: updateagents fossology-1.1.0~20090205.tar.gz
+
+TARFILE=$1
+TARDIR=${TARFILE%.tar.gz}
+
+for i in buckbeak fawkes
This is HP infrastructure specific, maybe you could make it more
generic or
at least explain that it's an HP specific script.
It's not an HP specific script. The server names are examples that
happen to be meaningful to me. That's why the comments say you can
use this script to copy the fossology tarball to the multiple agent
machines specified in the for loop. I'll clarify that and switch to
fo-postinstall when I get back to work on Monday (I'm on FTO).
Thanks!
Bob
+do
+scp $TARFILE $i:~/$TARFILE
+ssh $i tar -xzvf $TARFILE;
+cd $TARDIR;
+make ; make install;
+sudo ./install.sh -u';
install.sh went away with the old build system. You want
/usr/local/bin/fo-postinstall instead, and depending on which system
you
are running on you can use it's flags to only run the needed stuff.
Use
--help for more info.
--
Matt Taggart
tagg...@fossology.org
___
fossology-commits mailing list
fossology-comm...@fossology.org
http://fossology.org/mailman/listinfo/fossology-commits
___
fossology mailing list
fossology@fossology.org
http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Question reg export of license report as file
On Apr 21, 2009, at 4:55 AM, Fendt, Oliver wrote: Thank you very much for the valuable information. Ok then will will start soon with the implementation. One other question is there a feature planned / available which lists also the files of a package which contain no license information, and a feature which lists the copyright holders (I did not find such features)? Thanks again. We are currently working on two new license agents. These agents will use new common license database tables which will make it easy to see which files are without license information. Files with no licenses will also be listed on the standard license reports. However, it will still be a few months before this is released. If you are in a hurry, we can look at modifying the current license browser to give you that info. I think that we will have a report of the copyright holders about the same time. If you are willing to work of the top of our source tree instead of waiting for a release, you could get these sooner. You would have the honor of being an alpha tester. ;-) Or if you want to work on these features yourself, I'd be glad to help. Bob ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] [FOSSology-commits] SF.net SVN: fossology:[1986] trunk/project/adam/Training
Hi Adam, I see all these license training files. What about the non-license training files? b On Apr 19, 2009, at 8:53 PM, adamrba...@users.sourceforge.net wrote: Revision: 1986 http://fossology.svn.sourceforge.net/fossology/?rev=1986view=rev Author: adamrbates Date: 2009-04-20 02:53:02 + (Mon, 20 Apr 2009) Log Message: --- Added Training data for non-license type files. The license sections are wrapped in LICENSE_SECTION tags. Added Paths: --- trunk/project/adam/Training/License_Test/ trunk/project/adam/Training/License_Test/1 ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Question reg export of license report as file
Hi Oliver, No one is currently working on this so it would be great if you can contribute it. Since you aren't currently a fossology developer, you should just post on this list what you want to do and how you plan to go about it. That way you can get feedback to make sure that when you are done, it will fit in. You can have your own copyright or assign it to HP. You can chose your own license as long as it is compatible with the rest of our code, or just use what we use (GPL v2). Since this would be a UI plugin, writing it in php would be best. It would be great if you sent us updates to test as you go. Assuming the updates look good and you want commit access, we will give it to you. If you don't want commit access, just send the code to the list, I'll check it out and check it in for you. You won't be on the hook for maintenance unless you want to be. Is that painless enough? ;-) BTW, if you look at most (all?) of the UI plugins, you will see that there was initially the idea that the output could take multiple forms (html, text, xml). But everything except html is usually ignored. Bob Gobeille Hewlett Packard Open Source Program Office (and http://fossology.org) On Apr 20, 2009, at 8:36 AM, Fendt, Oliver wrote: Hi all, I browsed the task list an found a feature to be developed that will make it possible to export the license report as pdf my question is whether there is currently some one working on it or not? Further I want to know that if nobody is working on this feature and we would start to work on it, what the conditions would be to be to get our implementation accepted as contribution and who will take care about further maintenance (we need an editable textfile as export)? I'm asking because I did not find any information regarding that, if I simply missed it please point me to the right location. thanks Best regards Oliver Fendt Siemens AG Corporate Technology CT SE 2, Embedded Systems Open Source Platforms Otto-Hahn-Ring 6 81739 München, Deutschland Tel.: +49 (89) 636-46033 Fax: +49 (89) 636-45450 mailto:oliver.fe...@siemens.com Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Gerhard Cromme; Vorstand: Peter Löscher, Vorsitzender; Wolfgang Dehen, Heinrich Hiesinger, Joe Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. Solmssen; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322 ATT1.txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
Re: [FOSSology] Bug in the file upload web interface
Hi Mike, That sucks. email notification is a new feature that will be in v1.1. Thanks for letting us know. I'll file a bug. Bob Gobeille On Apr 16, 2009, at 1:42 AM, Mike Kinghan wrote: There is a non-critical but confusing bug affecting file uploads in the current trunk (1983) of fossology. If I create a user and do not specify an email address and do not select email-notifications for completed analyses, then when that user uploads a file they receive a upload failed: FATAL... error dialog as per attached screen shot. In fact the upload has not failed, and since the user does not want email notifications, nothing is wrong. The error message goes on to explain that the job has been scheduled, but that no email notfication will be sent when it is done. So it doesn't even mean that the upload has failed. There should at most be an informational message in these circumstances. Cheers, Mike upload_failed.pngATT1.txt ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
[FOSSology] RPM packaging
Welcome Vincent! Good news everyone. Vincent just joined the fossology family and he is going to be working on the RPM packaging. So please help him get up to speed as he asks questions (and reads the list archive). Bob Gobeille ___ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
