Re: CPE lists was Re: Equivalent packages between Linux distributions

[Petter Reinholdtsen]

[Silvio Cesare]
> I created an automatically generated CPE list for Fedora13
> packages. It only has 300 or so packages in it, but this will
> improve as say Debian increase the list of packages they track (they
> only track 1100 or so currently).
>
> https://github.com/silviocesare/Equivalent-Packages/blob/master/CPE/Fedora13.CPE.generated

Very interesting.  I created the CPE entries for Debian manually, by
comparing the set of affected packages reported in the CVE database
for Debian and NVD.  Perhaps something similar could be done for
Fedora, assuming the project track CVEs in a structured way?

Note that there are several duplicate CPE entries used by NVD.  A list
of the ones I have identified so far is in data/CPE/aliases.

Note that there is a bug in your list.  xen is claimed to be
grub-legacy.  Perhaps check your code?

Happy hacking,
-- 
Petter Reinholdtsen


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/2fllj202i1e@login2.uio.no

CPE lists was Re: Equivalent packages between Linux distributions

[Silvio Cesare]

I created an automatically generated CPE list for Fedora13 packages. It only
has 300 or so packages in it, but this will improve as say Debian increase
the list of packages they track (they only track 1100 or so currently).

https://github.com/silviocesare/Equivalent-Packages/blob/master/CPE/Fedora13.CPE.generated

To generate the list I build a list of equivalent packages between Debian
and Fedora
https://github.com/silviocesare/Equivalent-Packages/blob/master/NearestNeighbour/Debian5_Fedora13_Matches.
I then use Debian's CPE list
http://svn.debian.org/wsvn/secure-testing/data/CPE/listto
document the equivalent packages in Fedora.

This should work fine for other Distributions also.

--
Silvio Cesare

Re: Equivalent packages between Linux distributions

[Petter Reinholdtsen]

[Raphael Geissert]
> It would be great if anyone could make any progress on that.

Yeah.

> Some time ago it was mentioned as a possible way to automate the
> processing of new CVE ids (i.e. when MITRE publishes the description
> and other info) and to detect incorrect Not-For-Us entries in the
> security tracker.

Yes.  I did a quick implementation here at the university for tracking
our localy maintained software, and today mapped around 150
package/version pairs to CPEs allowing me to see which of our packages
had known security holes.

> One way to get started is by using the tracker's list of affected
> packages per CVE and match them with the CPEs provided by MITRE. It
> would be even better if in the future that information is provided
> by source packages themselves.

I suspect doing it manually is just as easy for now.  The 2240 entries
in my /var/lib/debsecan/history file only represent 293 binary
packages, which should be quick to look up in the CPE dictionary.

If it is to be stored in the source package, I suspect putting it
directly in the control file alongside the homepage URL make most
sense.  It would allow anyone to figure out relevant CVEs and make it
trivial to compare Debian and Ubuntu derivatives for the packages
originating from Debian.  Perhaps something like:

  Xs-CPE: cpe:/a:bash:bash

in debian/control would do it?  To get a versioned CPE, ":$version"
could be appended.

Happy hacking,
-- 
Petter Reinholdtsen


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/2fl39ogais0@login1.uio.no

Re: Equivalent packages between Linux distributions

[Raphael Geissert]

Petter Reinholdtsen wrote:
[...]
> It would be great if you or someone else could provide a mapping from
> distribution packages to CPE entries. :)

It would be great if anyone could make any progress on that.

Some time ago it was mentioned as a possible way to automate the processing 
of new CVE ids (i.e. when MITRE publishes the description and other info) 
and to detect incorrect Not-For-Us entries in the security tracker.

One way to get started is by using the tracker's list of affected packages 
per CVE and match them with the CPEs provided by MITRE. It would be even 
better if in the future that information is provided by source packages 
themselves.

Cheers,
-- 
Raphael Geissert - Debian Developer
www.debian.org - get.debian.net


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/ihn599$8pd$1...@dough.gmane.org

Re: Equivalent packages between Linux distributions

[Petter Reinholdtsen]

[Silvio Cesare]
> Do you think such a list could be useful to Debian? A possible use
> would be that a user could identify an equivalent package knowing
> only Fedora's package name.

I've been looking into a similar task the last few days, to try to
track security issues in multiple distributions and locally maintained
software.

The Common Platform Enumeration dictionary,
http://nvd.nist.gov/cpe.cfm >, provide a common vocabulary for
packages, and it would be very useful if Debian would provide the CPE
entry for each of the packages in the archive.

The CPE dictionary contain IDs for packages (applications), operating
systems and hardware, and allow these IDs to be used to look up CVEs.
If such IDs were provided the packages in for linux distributions, it
would be trivial to find equivalent packages.

The package/application IDs look like this, for a few of the packages
in the Debian archive.

  cpe:/a:bash:bash:4.1
  cpe:/a:gnu:gzip:1.3.12
  cpe:/a:apache:subversion:1.6.12
  cpe:/a:apache:http_server:2.2.16

The IDs can also be used without version numbers.

It would be great if you or someone else could provide a mapping from
distribution packages to CPE entries. :)

Happy hacking,
-- 
Petter Reinholdtsen


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/2flaaip46qu@login2.uio.no

Re: Equivalent packages between Linux distributions

[Enrico Zini]

On Fri, Jan 21, 2011 at 04:53:07PM +0100, Matthias Klumpp wrote:

> I'm the developer of Listaller, a cross-distro software installer/manager
> and therefore also interested in this topic, as it is one of the major
> technical problems which are currently not solved very well :)
> Unfortunately I couldn't attend the AppInstaller meeting. (I wanted to,
> but couldn't take the time off)

After some unit testing and cleanup the tool finally started to become
useful to the point I could allow myself to spend time blogging instead
of coding: you can find details at
http://www.enricozini.org/2011/debian/distromatch/

On Fri, Jan 21, 2011 at 08:03:14PM +1100, Silvio Cesare wrote:

>The method I am using is based on similarity between filename lists of
>source packages. I use the Jaccard index
>([1]http://en.wikipedia.org/wiki/Jaccard_index) between�sets of filenames
>to calculate similarity. This was done as an offshoot�from the�PhD
>research I'm currently undertaking at Deakin University.

Neat! Are you publishing the sample dataset only at this stage, or do
you also intend to public the code? (I didn't see the code in your git
repo, maybe I just missed it)

I could be interested in playing with it a bit more, potentially
replicating the method on binary packages or seeing if an approximation
on it can be run on the Xapian indices that distromatch is using.

If the datasets prove useful it could also be interesting to schedule
periodic runs on a Debian machine.


Ciao,

Enrico

-- 
GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini 


signature.asc
Description: Digital signature

Re: Equivalent packages between Linux distributions

[Matthias Klumpp]

Hi there!
I'm the developer of Listaller, a cross-distro software installer/manager
and therefore also interested in this topic, as it is one of the major
technical problems which are currently not solved very well :)
Unfortunately I couldn't attend the AppInstaller meeting. (I wanted to,
but couldn't take the time off)

On Thu, 20 Jan 2011 23:02:07 +0100, Enrico Zini 
wrote:
> On Wed, Jan 19, 2011 at 10:54:44AM +1100, Silvio Cesare wrote:
> 
>>I have generated a list of roughly equivalent packages between Linux
>>distributions (currently Debian 5 and Fedora 13). The list is
>>automatically generated.
> [...]
> [...]
> The main use case we have in mind is to be able to fall back on other
> distros when a package doesn't have some piece of information. For
> example:
> 
>  - does package $foo have a screenshot in Debian?
>  - if no, how about in Fedora?
>  - if no, how about in OpenSUSE?
>  - if no, how about in Mandriva?
> 
> The example uses screenshots, but it could be other kinds of metadata,
> like categories (it's a way for example to port at least some of Debtags
> to other distros), ratings or user comments.
Yeah, especially the Debtags would be extremely valuable!

> The heuristics I've been implementing so far are:
> 
>  - trivial package name matching
>  - 'stemming' specific kinds of package names (debian:lifoo-dev->foo;
>fedora:foo-devel->foo)
>  - matching packages that contain the same .desktop files or the same
>pkg-config files
>  - similarity matching of file lists
I to _exactly_ the same at time! I compare data from Sophie (
http://sophie.zarb.org/ ) and UDD (mostly similar files and names) and then
cache the data in a SQLite DB.

> I still don't have results because the implementation is not complete,
> but I should have something in a day or two. You have something *today*,
> which is, wow. Tomorrow (Friday) I'll download your dataset and try to
> add another euristic that just uses it. It'll also be interesting to use
> all these methods to cross-validate each other.
Although I don't have much time at time, I'm interested in helping... (But
I have two other projects which take priority...)
How can I get some more information about the project, and is there a
possibility to add some more, extended data, which is not directly required
to fetch meta-data? (Then I could drop my own implementation of
"PackageCompare")

Kind regards
   Matthias Klumpp



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4e46f573fa89af609bd04e1a2abab...@mb8-2.1blu.de

Re: Equivalent packages between Linux distributions

[Silvio Cesare]

Hi, Thanks for the response and interest.

I should note the package names I have listed in the dataset are source
package names, not binary package names.

 The method I am using is based on similarity between filename lists of
source packages. I use the Jaccard index (
http://en.wikipedia.org/wiki/Jaccard_index) between sets of filenames to
calculate similarity. This was done as an offshoot from the PhD research I'm
currently undertaking at Deakin University.

--
Silvio

On Fri, Jan 21, 2011 at 9:02 AM, Enrico Zini  wrote:

> On Wed, Jan 19, 2011 at 10:54:44AM +1100, Silvio Cesare wrote:
>
> >I have generated a list of roughly equivalent packages between Linux
> >distributions (currently Debian 5 and Fedora 13). The list is
> >automatically generated.
> [...]
>
> Hi Silvio,
>
> thank you for your work, it is extremely valuable work.  I'm currently
> at a cross-distro meeting on app installers[1] and it's precisely
> something we've been working on today. I'd be greatly interested to
> exchange algorithms with you.
>
> The main use case we have in mind is to be able to fall back on other
> distros when a package doesn't have some piece of information. For
> example:
>
>  - does package $foo have a screenshot in Debian?
>  - if no, how about in Fedora?
>  - if no, how about in OpenSUSE?
>  - if no, how about in Mandriva?
>
> The example uses screenshots, but it could be other kinds of metadata,
> like categories (it's a way for example to port at least some of Debtags
> to other distros), ratings or user comments.
>
> The euristics I've been implementing so far are:
>
>  - trivial package name matching
>  - 'stemming' specific kinds of package names (debian:lifoo-dev->foo;
>   fedora:foo-devel->foo)
>  - matching packages that contain the same .desktop files or the same
>   pkg-config files
>  - similarity matching of file lists
>
> I still don't have results because the implementation is not complete,
> but I should have something in a day or two. You have something *today*,
> which is, wow. Tomorrow (Friday) I'll download your dataset and try to
> add another euristic that just uses it. It'll also be interesting to use
> all these methods to cross-validate each other.
>
> [1] http://distributions.freedesktop.org/wiki/Meetings/AppInstaller2011
>
>
> Ciao,
>
> Enrico
>
> --
> GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini 
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQEcBAEBCAAGBQJNOLDeAAoJEON4Oc9CHQta7ckH/1IsATAFZss4NprTfzO0LMWi
> hXn8ds1GvPIxzokgKnX6v3JAq0rX56kFe4yDMFL2JA0GHTHR7bpXtClYBFtP9ErX
> XWv6caymfqmJVQLDDwUuDMPUBrVLeT+U4syv7B47JI/paGMfDPYfcRn74qEVrSlL
> T3P9cMYKzAwvgrNpL+EGAP3Kw34nfiMra3hmD7SeeYluo3trNUV3/BP6oRxIiLu0
> RBSvRzf6+W2P+jE2TsR/KSPYQQ9Ji6CjFPElzNYgW6N3ZKte985vA5AadX91pE2G
> QuKeW9PouddjCok1G9qgUCbDLz/WEQqbwkvC6/Wi5TVvpyRwqWmoj6Pmcx9klKM=
> =KtYB
> -END PGP SIGNATURE-
>
>

Re: Equivalent packages between Linux distributions

[Enrico Zini]

On Wed, Jan 19, 2011 at 10:54:44AM +1100, Silvio Cesare wrote:

>I have generated a list of roughly equivalent packages between Linux
>distributions (currently Debian 5 and Fedora 13). The list is
>automatically generated.
[...]

Hi Silvio,

thank you for your work, it is extremely valuable work.  I'm currently
at a cross-distro meeting on app installers[1] and it's precisely
something we've been working on today. I'd be greatly interested to
exchange algorithms with you.

The main use case we have in mind is to be able to fall back on other
distros when a package doesn't have some piece of information. For
example:

 - does package $foo have a screenshot in Debian?
 - if no, how about in Fedora?
 - if no, how about in OpenSUSE?
 - if no, how about in Mandriva?

The example uses screenshots, but it could be other kinds of metadata,
like categories (it's a way for example to port at least some of Debtags
to other distros), ratings or user comments.

The euristics I've been implementing so far are:

 - trivial package name matching
 - 'stemming' specific kinds of package names (debian:lifoo-dev->foo;
   fedora:foo-devel->foo)
 - matching packages that contain the same .desktop files or the same
   pkg-config files
 - similarity matching of file lists

I still don't have results because the implementation is not complete,
but I should have something in a day or two. You have something *today*,
which is, wow. Tomorrow (Friday) I'll download your dataset and try to
add another euristic that just uses it. It'll also be interesting to use
all these methods to cross-validate each other.

[1] http://distributions.freedesktop.org/wiki/Meetings/AppInstaller2011


Ciao,

Enrico

-- 
GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini 


signature.asc
Description: Digital signature

Re: Equivalent packages between Linux distributions

[Paul Wise]

You might want to look at the PackageMap project started by a
Debian/Gentoo contributor:

http://blog.hartwork.org/?p=373
http://lists.freedesktop.org/mailman/listinfo/packagemap

-- 
bye,
pabs

http://wiki.debian.org/PaulWise


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/AANLkTi=gsel56fde7eaz5nedeo5h_y1uh56j-3vly...@mail.gmail.com

Re: Equivalent packages between Linux distributions

[Adrian von Bidder]

On Wednesday 19 January 2011 00.54:44 Silvio Cesare wrote:
> I have generated a list of roughly equivalent packages between Linux
> distributions (currently Debian 5 and Fedora 13). The list is
> automatically generated.

Cool!

Maybe I have missed a pointer or whatever: how did you compute this 
similarity? Number of identical files?  Or filenames?

I was just wondering: GSoC project: 
 * run this after every release of a major distribution
 * add this info to aptitude/... : if I install a package that doesn't 
exist, add this to the search database (not sure what aptitude looks at 
right now, exactly, but it already has the infrastructure for proposing a 
package if the given name doesn't exist.)

But please make this optional, on small systems apt already is only barely 
usable.

cheers
-- vbi

-- 
to debug such lockups in the future you can do:
...
NOTE: dont use the keyboard in this mode for too long, it can lock up.
-- Ingo Molnar, lkml


signature.asc
Description: This is a digitally signed message part.

Re: Equivalent packages between Linux distributions

[Tollef Fog Heen]

]] Silvio Cesare 

| Do you think such a list could be useful to Debian? A possible use would be
| that a user could identify an equivalent package knowing only Fedora's
| package name.

I think it'd be useful.  Also, take a look at the whohas package, it
seems to do something similar, but doesn't look at the contents, afaik.

Regards,
-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87y66hwiy0@qurzaw.varnish-software.com

Re: Equivalent packages between Linux distributions

[Ben Finney]

Silvio Cesare  writes:

> I have generated a list of roughly equivalent packages between Linux
> distributions (currently Debian 5 and Fedora 13). The list is
> automatically generated.
[…]

> Do you think such a list could be useful to Debian? A possible use
> would be that a user could identify an equivalent package knowing only
> Fedora's package name.

The Freedesktop forum for inter-distribution discussion
http://lists.freedesktop.org/mailman/listinfo/distributions> would
be a good place for that information too; please let them know of your
work.

> Please CC me on any responses.

Done.

-- 
 \“Don't fight forces, use them.” —Richard Buckminster Fuller, |
  `\   _Shelter_, 1932 |
_o__)  |
Ben Finney 


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87pqrt4twy@benfinney.id.au