On Wed, Oct 26, 2005 at 07:43:38PM -0400, Benjamin Mesing wrote: > However I think the Tokenizer I wrote could be reused - to create the > <package>.token files. It is most likely smarter than simply taking the > output of "apt-cache dumpavail". Even though I hate all those "gut
Yes, it's a good idea. So far I'm using apt-cache dumpavail as mornfall
told me that it looked like dbacl was ignoring noisy things (like
Installed-Size) quite well. However, there's probably always space for
improvements (for example, the tokenizer could convert the size into
some more meaningful tokens like 'small', 'medium', 'large', 'huge').
> Hanna, whom I hoped might have some time to do this (and some more
> knowledge), did not deign to respond any of the mails I sent to the list
> or directly to her.
I know she's been busy, and I hope she'll eventually get a bit involved.
> However some of the lessons I've learned with the AI tagger might come
> in handy for dbacl too. One is that the a smart tokenizer increases the
> precision. Another is that a training ratio of 2:1 (bad:good) might
> increase the precision too. But that may also depend on the underlying
> bayesian implementation.
Good hint. I took that 2:1 into account.
> Perhaps the framework of the bayesian-tagger perlscript can be also used
> to launch the dbacl -- but probably Enrico is faster hacking some bash
> scripts...
I've actually rewritten some things in perl, to be able to cache
intermediate results and run things quicker.
I've implemented a script that spits a tag patch. It's quite cool, and
on my machine it takes like 4 minutes to train and 2.4 seconds per
package to run.
It works like this:
svn co svn+ssh://svn.debian.org/svn/debtags/autodebtag/trunk/dbacl dbacl
cd dbacl
mkdir data
./debtags-ai train
./debtags-ai patch > patch
I attach the partial patch I have so far, so that you can evaluate.
At the beginning of the script there's a line setting $verbose to 1.
Set it to 2 to see timings for every package.
I blacklist some tags, both for speed and because the results are quite
useless. For example, culture::* was assigned quite randomly (like, if
there was a romanian translation of a python manual, many python
packages were likely to be tagged romanian). I guess it's because
there's very few culture::foo tags assigned to be used for training.
There's also a script called perfacet, which outputs which tags it
thinks would be the best for every facet (reformatted and re-sorted for
better viewing):
$ ./debtags-ai train
$ ./perfacet debtags
100% x11::application
99% use::searching
93% suite::debian
93% made-of::lang:c++
92% works-with::software:package
85% devel::library
84% sound::player
76% interface::commandline
73% special::auto-inst-parts
72% role::sw:utility
67% culture::japanese
66% field::mathematics
61% langdevel::perl
55% network::service
55% filetransfer::ftp
46% uitoolkit::gtk
46% game::toys
43% web::server
37% junior::arcade
27% protocol::http
25% security::authentication
20% accessibility::input
10% hardware::storage:cd
6% mail::filters
4% admin::configuring
If I remove all the ones below 90%, I get:
100% x11::application
99% use::searching
93% suite::debian
93% made-of::lang:c++
92% works-with::software:package
Besides the x11::application glitch (I guess what's happening is that,
among the various x11 alternatives, 'application' is the one that
definitely sounds better: remember it's selecting among the tags in the
same facet).
> /me was impressed by what Enrico did with bash together with pipes and
> all those neat little tools. Perhaps I should learn bash and some more
> Linux commands after 5 years of excessivly using Linux.
:-)
Bash tricks are really cute, sometimes essential for doing quick
prototypes. Then one can have clearer ideas and implement things in
something efficient, or cleaner.
Btw, /me points at the 'perfacet' script for a creative use of xargs :)
Ciao,
Enrico
--
GPG key: 1024D/797EBFAB 2000-12-05 Enrico Zini <[EMAIL PROTECTED]>
patch.gz
Description: Binary data
signature.asc
Description: Digital signature
_______________________________________________ Debtags-devel mailing list [email protected] http://lists.alioth.debian.org/mailman/listinfo/debtags-devel
