Re: [gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in dev-util/netbeans: ChangeLog netbeans-6.5-r1.ebuild netbeans-6.5.ebuild

2008-11-23 Thread Miroslav Šulc (fordfrog)
Thank you both for the suggestions, I fixed these issues in updated 
netbeans-6.5-r1.


Miroslav Šulc (fordfrog)
Gentoo Java Team

Jeremy Olexa napsal(a):

Peter Volkov wrote:

В Сбт, 22/11/2008 в 18:11 +, Miroslav Sulc (fordfrog) пишет:

fordfrog08/11/22 18:11:25
  Added:netbeans-6.5-r1.ebuild
  Log: netbeans compiles fine even with JDK 1.6 so I dropped the 
restriction on JDK, also commons-fileupload linking fixed



Index: netbeans-6.5-r1.ebuild
===
pkg_setup() {
if use netbeans_modules_apisupport  ! ( use 
netbeans_modules_harness  use netbeans_modules_ide  use 
netbeans_modules_java ) ; then
eerror 'apisupport' USE flag requires 'harness', 'ide' and 
'java' USE flags


Additionally, 'apisupport', 'harness', 'ide' and 'java' are not USE 
flags. You have to set NETBEANS_MODULES, not USE. You can easily test 
this by trying NETBEANS_MODULES=apisupport USE=java emerge -pv 
netbeans and see that netbeans_modules_java does not get set.

-Jeremy


exit 1
fi


Why do you use exit 1 instead of die?


local tmpfileplatform=${T}/platform.txt
cat ${tmpfile} | grep -v libs.jna/external/jna-3.0.2.jar  
${tmpfileplatform}

mv ${tmpfileplatform} ${tmpfile}


grep can read files on it's own so no need for cat file | grep... Also 
possibly


sed -e /libs\.jna\/external\/jna-3\.0\.2\.jar/d -i ${tmpfile}

will work better here and in some other places...








[gentoo-dev] Some support for Sunrise Overlay :-)

2008-11-23 Thread Thomas Sachau
I would like to ask all people, who get to users, who want to contribute with 
ebuilds or similar,
tell them about our sunrise overlay[1][2]. People can learn how to write 
ebuilds, they can learn how
to write ebuilds with some nice QA, they learn how to manage a SCM (in this 
case svn) and some may
even enhance to gentoo developers. :-)
As a side effect, we get a nice collection of ebuild, which are easily 
maintainable even by users
and which are easily accessable by everyone.

Esp. i would ask those who wrangle bug: If a user contributes an ebuild for an 
app not already in
portage, tell them about sunrise, so they help out a little bit more, if they 
are interested. With
that done, something like sunrise suggested in Whiteboard would be nice.





Anyone knows an easy way to point people to our overlay from e.g. our main 
website? Something like
www.gentoo.org - I want to contribute/Contributions - Sunrise/others would 
be nice. It could
help to get more contributions and activity to sunrise on the one side and will 
make it easier for
people who want to contribute to find some place where they can do it.

Other ideas/suggestions for Sunrise are also welcome, tell me about them 
on-list/off-list/by
mail/#gentoo-sunrise or other preferred way.

[1]: http://www.gentoo.org/proj/en/sunrise
[2]: http://overlays.gentoo.org/proj/sunrise/wiki


-- 
Thomas Sachau

Gentoo Linux Developer



signature.asc
Description: OpenPGP digital signature


[gentoo-dev] Automated Package Removal and Addition Tracker, for the week ending 2008-11-23 23h59 UTC

2008-11-23 Thread Robin H. Johnson
The attached list notes all of the packages that were added or removed
from the tree, for the week ending 2008-11-23 23h59 UTC.

Removals:
dev-scheme/drscheme 2008-11-17 00:13:38 pchrist
dev-perl/Archive-Tar2008-11-19 06:09:49 tove
dev-perl/Class-ISA  2008-11-19 06:10:13 tove
dev-perl/Compress-Raw-Zlib  2008-11-19 06:10:23 tove
dev-perl/Compress-Zlib  2008-11-19 06:10:32 tove
dev-perl/ExtUtils-CBuilder  2008-11-19 06:10:44 tove
dev-perl/IO-Compress-Base   2008-11-19 06:10:55 tove
dev-perl/IO-Compress-Zlib   2008-11-19 06:11:04 tove
dev-perl/IO-Zlib2008-11-19 06:11:13 tove
dev-perl/Locale-Maketext-Simple 2008-11-19 06:11:22 tove
dev-perl/Math-BigInt-FastCalc   2008-11-19 06:11:32 tove
dev-perl/module-build   2008-11-19 06:11:41 tove
dev-perl/Module-Pluggable   2008-11-19 06:11:50 tove
dev-perl/Pod-Escapes2008-11-19 06:11:59 tove
dev-perl/Pod-Simple 2008-11-19 06:12:08 tove
dev-perl/Time-Piece 2008-11-19 06:12:19 tove
dev-perl/version2008-11-19 06:12:47 tove
dev-perl/Term-ANSIColor 2008-11-19 06:13:32 tove
dev-perl/extutils-parsexs   2008-11-19 06:13:55 tove
dev-perl/Digest-SHA 2008-11-19 06:14:16 tove
dev-perl/VCP-autrijus   2008-11-19 06:26:50 tove
dev-perl/PodToHTML  2008-11-19 06:26:59 tove
dev-perl/FileHandle-Deluxe  2008-11-19 06:27:11 tove
dev-perl/FileHandle-Rollback2008-11-19 06:27:28 tove
dev-perl/Audio-Tools2008-11-19 06:27:37 tove
app-emulation/sheepshaver   2008-11-19 15:17:19 darkside
x11-base/xdirectfb  2008-11-19 15:19:22 darkside
app-admin/powertweak2008-11-19 15:22:32 darkside
net-libs/libpcap-ringbuffer 2008-11-20 04:32:52 darkside
net-misc/ssh2008-11-20 04:48:53 darkside
net-firewall/ipchains-firewall  2008-11-23 00:21:30 antarus
net-irc/charybdis   2008-11-23 00:38:45 antarus

Additions:
dev-scheme/plt-scheme   2008-11-17 00:07:29 pchrist
media-plugins/vdr-systeminfo2008-11-17 17:56:13 hd_brummy
sci-calculators/speedcrunch 2008-11-17 23:58:15 bicatali
app-emacs/quilt-el  2008-11-18 16:13:58 ulm
media-plugins/vdr-devstatus 2008-11-18 20:25:34 zzam
media-gfx/galleryadd2008-11-20 02:08:36 robbat2
app-office/multitalk2008-11-20 16:30:27 matsuu
app-admin/checkrestart  2008-11-20 21:30:02 jer
sys-block/spindown  2008-11-21 18:20:42 wschlich
sys-apps/unscd  2008-11-22 22:09:11 vapier
app-benchmarks/forkbomb 2008-11-23 10:53:12 patrick
dev-dotnet/webkit-sharp 2008-11-23 16:24:37 loki_val
app-emacs/nxml-svg-schemas  2008-11-23 19:33:41 flameeyes

--
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail : [EMAIL PROTECTED]
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85
Removed Packages:
dev-scheme/drscheme,removed,pchrist,2008-11-17 00:13:38
dev-perl/Archive-Tar,removed,tove,2008-11-19 06:09:49
dev-perl/Class-ISA,removed,tove,2008-11-19 06:10:13
dev-perl/Compress-Raw-Zlib,removed,tove,2008-11-19 06:10:23
dev-perl/Compress-Zlib,removed,tove,2008-11-19 06:10:32
dev-perl/ExtUtils-CBuilder,removed,tove,2008-11-19 06:10:44
dev-perl/IO-Compress-Base,removed,tove,2008-11-19 06:10:55
dev-perl/IO-Compress-Zlib,removed,tove,2008-11-19 06:11:04
dev-perl/IO-Zlib,removed,tove,2008-11-19 06:11:13
dev-perl/Locale-Maketext-Simple,removed,tove,2008-11-19 06:11:22
dev-perl/Math-BigInt-FastCalc,removed,tove,2008-11-19 06:11:32
dev-perl/module-build,removed,tove,2008-11-19 06:11:41
dev-perl/Module-Pluggable,removed,tove,2008-11-19 06:11:50
dev-perl/Pod-Escapes,removed,tove,2008-11-19 06:11:59
dev-perl/Pod-Simple,removed,tove,2008-11-19 06:12:08
dev-perl/Time-Piece,removed,tove,2008-11-19 06:12:19
dev-perl/version,removed,tove,2008-11-19 06:12:47
dev-perl/Term-ANSIColor,removed,tove,2008-11-19 06:13:32
dev-perl/extutils-parsexs,removed,tove,2008-11-19 06:13:55
dev-perl/Digest-SHA,removed,tove,2008-11-19 06:14:16
dev-perl/VCP-autrijus,removed,tove,2008-11-19 06:26:50
dev-perl/PodToHTML,removed,tove,2008-11-19 06:26:59
dev-perl/FileHandle-Deluxe,removed,tove,2008-11-19 06:27:11
dev-perl/FileHandle-Rollback,removed,tove,2008-11-19 06:27:28
dev-perl/Audio-Tools,removed,tove,2008-11-19 06:27:37
app-emulation/sheepshaver,removed,darkside,2008-11-19 15:17:19
x11-base/xdirectfb,removed,darkside,2008-11-19 15:19:22
app-admin/powertweak,removed,darkside,2008-11-19 15:22:32
net-libs/libpcap-ringbuffer,removed,darkside,2008-11-20 04:32:52
net-misc/ssh,removed,darkside,2008-11-20 04:48:53
net-firewall/ipchains-firewall,removed,antarus,2008-11-23 00:21:30
net-irc/charybdis,removed,antarus,2008-11-23 00:38:45
Added Packages:

[gentoo-dev] Re: [gentoo-commits] gentoo-x86 commit in profiles: use.local.desc ChangeLog

2008-11-23 Thread Robin H. Johnson
On Mon, Nov 24, 2008 at 01:59:52AM +, Marcelo Goes (vanquirius) wrote:
 vanquirius08/11/24 01:59:52
   Modified: use.local.desc ChangeLog
   Log:
   Update net-analyzer/snort local use flags for bug 245752. Thanks to Jason 
 Wallace
   jason.r.wallace at gmail dot com.
You must NOT commit to use.local.desc.

Edit metadata.xml for your package instead.

-- 
Robin Hugh Johnson
Gentoo Linux Developer  Infra Guy
E-Mail : [EMAIL PROTECTED]
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85


pgpXN682CN4C7.pgp
Description: PGP signature


[gentoo-dev] Re: Some support for Sunrise Overlay :-)

2008-11-23 Thread Nikos Chantziaras

Thomas Sachau wrote:

I would like to ask all people, who get to users, who want to contribute with 
ebuilds or similar,
tell them about our sunrise overlay[1][2]. People can learn how to write 
ebuilds, they can learn how
to write ebuilds with some nice QA, they learn how to manage a SCM (in this 
case svn) and some may
even enhance to gentoo developers. :-)
As a side effect, we get a nice collection of ebuild, which are easily 
maintainable even by users
and which are easily accessable by everyone.

Esp. i would ask those who wrangle bug: If a user contributes an ebuild for an 
app not already in
portage, tell them about sunrise, so they help out a little bit more, if they 
are interested. With
that done, something like sunrise suggested in Whiteboard would be nice.


I was told to try sunrise with some of my ebuilds.  There's a reason I 
didn't, and I suspect it's the same other people also don't go to 
sunrise.  There's no good way of contributing an ebuild.  There's no 
bugzilla or website where I can upload the ebuild.  All I'm told is to 
go to IRC.  I would as well might have been required to go to McDonald's.





[gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Emma Strubell
Hi everyone. My name is Emma, and I am completely new to this list. I've
been using Gentoo since 2004, including Portage of course, and before I say
anything else I'd like to say thanks to everyone for such a kickass package
management system!!

Anyway, for my final project in my Data Structures  Algorithms class this
semester, I would like to modify the search functionality in emerge.
Something I've always noticed about 'emerge -s' or '-S' is that, in general,
it takes a very long time to perform the searches. (Although, lately it does
seem to be running faster, specifically on my laptop as opposed to my
desktop. Strangely, though, it seems that when I do a simple 'emerge -av
whatever' on my laptop it takes a very long time for emerge to find the
package and/or determine the dependecies -  whatever it's doing behind that
spinner. I can definitely go into more detail about this if anyone's
interested. It's really been puzzling me!) So, as my final project I've
proposed to improve the time it takes to perform a search using emerge. My
professor suggested that I look into implementing indexing.

However, I've started looking at the code, and I must admit I'm pretty
overwhelmed! I don't know where to start. I was wondering if anyone on here
could give me a quick overview of how the search function currently works,
an idea as to what could be modified or implemented in order to improve the
running time of this code, or any tip really as to where I should start or
what I should start looking at. I'd really appreciate any help or advice!!

Thanks a lot, and keep on making my Debian-using professor jealous :]
Emma


Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Pacho Ramos
El dom, 23-11-2008 a las 16:01 +0200, tvali escribió:
 Try esearch.
 
 emerge esearch
 esearch ...
 
 2008/11/23 Emma Strubell [EMAIL PROTECTED]
 Hi everyone. My name is Emma, and I am completely new to this
 list. I've been using Gentoo since 2004, including Portage of
 course, and before I say anything else I'd like to say thanks
 to everyone for such a kickass package management system!!
 
 Anyway, for my final project in my Data Structures 
 Algorithms class this semester, I would like to modify the
 search functionality in emerge. Something I've always noticed
 about 'emerge -s' or '-S' is that, in general, it takes a very
 long time to perform the searches. (Although, lately it does
 seem to be running faster, specifically on my laptop as
 opposed to my desktop. Strangely, though, it seems that when I
 do a simple 'emerge -av whatever' on my laptop it takes a very
 long time for emerge to find the package and/or determine the
 dependecies -  whatever it's doing behind that spinner. I can
 definitely go into more detail about this if anyone's
 interested. It's really been puzzling me!) So, as my final
 project I've proposed to improve the time it takes to perform
 a search using emerge. My professor suggested that I look into
 implementing indexing.
 
 However, I've started looking at the code, and I must admit
 I'm pretty overwhelmed! I don't know where to start. I was
 wondering if anyone on here could give me a quick overview of
 how the search function currently works, an idea as to what
 could be modified or implemented in order to improve the
 running time of this code, or any tip really as to where I
 should start or what I should start looking at. I'd really
 appreciate any help or advice!!
 
 Thanks a lot, and keep on making my Debian-using professor
 jealous :]
 Emma
 
 
 
 -- 
 tvali
 
 Kuskilt foorumist: http://www.cooltests.com - kui inglise keelt oskad.
 Muide, üle 120 oled väga tark, üle 140 oled geenius, mingi 170 oled ju
 mingi täica pea nagu prügikast...

I use eix:
emerge eix

;-)




Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Emma Strubell
Thanks for the replies! I know there are a couple programs out there that
basically already do what I'm looking to do... Unfortunately I wasn't aware
of these pre-existing utilities until after I submitted my project proposal
to my professor. So, I'm looking to implement a better search myself.
Preferably by editing the existing portage code, not writing a separate
program. So if anyone can offer any help regarding the actual implementation
of search in portage, I would greatly appreciate it!

Or, if anyone has an idea for a more productive/useful project I could work
on relating to portage (about the same difficulty, preferably at least a
little bit data structure related), please, let me know! Thanks again guys,

Emma

On Sun, Nov 23, 2008 at 9:33 AM, Pacho Ramos 
[EMAIL PROTECTED] wrote:

 El dom, 23-11-2008 a las 16:01 +0200, tvali escribió:
  Try esearch.
 
  emerge esearch
  esearch ...
 
  2008/11/23 Emma Strubell [EMAIL PROTECTED]
  Hi everyone. My name is Emma, and I am completely new to this
  list. I've been using Gentoo since 2004, including Portage of
  course, and before I say anything else I'd like to say thanks
  to everyone for such a kickass package management system!!
 
  Anyway, for my final project in my Data Structures 
  Algorithms class this semester, I would like to modify the
  search functionality in emerge. Something I've always noticed
  about 'emerge -s' or '-S' is that, in general, it takes a very
  long time to perform the searches. (Although, lately it does
  seem to be running faster, specifically on my laptop as
  opposed to my desktop. Strangely, though, it seems that when I
  do a simple 'emerge -av whatever' on my laptop it takes a very
  long time for emerge to find the package and/or determine the
  dependecies -  whatever it's doing behind that spinner. I can
  definitely go into more detail about this if anyone's
  interested. It's really been puzzling me!) So, as my final
  project I've proposed to improve the time it takes to perform
  a search using emerge. My professor suggested that I look into
  implementing indexing.
 
  However, I've started looking at the code, and I must admit
  I'm pretty overwhelmed! I don't know where to start. I was
  wondering if anyone on here could give me a quick overview of
  how the search function currently works, an idea as to what
  could be modified or implemented in order to improve the
  running time of this code, or any tip really as to where I
  should start or what I should start looking at. I'd really
  appreciate any help or advice!!
 
  Thanks a lot, and keep on making my Debian-using professor
  jealous :]
  Emma
 
 
 
  --
  tvali
 
  Kuskilt foorumist: http://www.cooltests.com - kui inglise keelt oskad.
  Muide, üle 120 oled väga tark, üle 140 oled geenius, mingi 170 oled ju
  mingi täica pea nagu prügikast...

 I use eix:
 emerge eix

 ;-)





Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Douglas Anderson
Emma,

It would be great it you could speed search up a bit!

As these other guys have pointed out, we do have some indexing tools in
Gentoo already. Most users don't understand why that kind of functionality
isn't built directly into Portage, but IIRC it has something to do with the
fact that these fast search indexes aren't able to be written to by more
than one process at the same time, so for example if you had two emerges
finishing at the same time, Portage's current flat hash file can handle
that, but the faster db-based indexes can't.

Anyways, that's the way I, as a curious user, understood the problem.

You might be interested in reading this, very old forum thread about a
previous attempt:
http://forums.gentoo.org/viewtopic-t-261580-postdays-0-postorder-asc-start-0.html

On Sun, Nov 23, 2008 at 11:33 PM, Pacho Ramos 
[EMAIL PROTECTED] wrote:

 El dom, 23-11-2008 a las 16:01 +0200, tvali escribió:
  Try esearch.
 
  emerge esearch
  esearch ...
 
  2008/11/23 Emma Strubell [EMAIL PROTECTED]
  Hi everyone. My name is Emma, and I am completely new to this
  list. I've been using Gentoo since 2004, including Portage of
  course, and before I say anything else I'd like to say thanks
  to everyone for such a kickass package management system!!
 
  Anyway, for my final project in my Data Structures 
  Algorithms class this semester, I would like to modify the
  search functionality in emerge. Something I've always noticed
  about 'emerge -s' or '-S' is that, in general, it takes a very
  long time to perform the searches. (Although, lately it does
  seem to be running faster, specifically on my laptop as
  opposed to my desktop. Strangely, though, it seems that when I
  do a simple 'emerge -av whatever' on my laptop it takes a very
  long time for emerge to find the package and/or determine the
  dependecies -  whatever it's doing behind that spinner. I can
  definitely go into more detail about this if anyone's
  interested. It's really been puzzling me!) So, as my final
  project I've proposed to improve the time it takes to perform
  a search using emerge. My professor suggested that I look into
  implementing indexing.
 
  However, I've started looking at the code, and I must admit
  I'm pretty overwhelmed! I don't know where to start. I was
  wondering if anyone on here could give me a quick overview of
  how the search function currently works, an idea as to what
  could be modified or implemented in order to improve the
  running time of this code, or any tip really as to where I
  should start or what I should start looking at. I'd really
  appreciate any help or advice!!
 
  Thanks a lot, and keep on making my Debian-using professor
  jealous :]
  Emma
 
 
 
  --
  tvali
 
  Kuskilt foorumist: http://www.cooltests.com - kui inglise keelt oskad.
  Muide, üle 120 oled väga tark, üle 140 oled geenius, mingi 170 oled ju
  mingi täica pea nagu prügikast...

 I use eix:
 emerge eix

 ;-)





Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Emma Strubell
Wow, that's extremely helpful!! I happen to particularly enjoy tries, so the
suffix trie sounds like a great idea. The trie class example is really
helpful too, because this will be my first time programming in Python, and
it's a bit easier to figure out what's going on syntax-wise in that simple
trie class than in the middle of the portage source code!

Seriously, thanks again :]

On Sun, Nov 23, 2008 at 11:56 AM, Lucian Poston [EMAIL PROTECTED]wrote:

  Thanks for the replies! I know there are a couple programs out there that
  basically already do what I'm looking to do... Unfortunately I wasn't
 aware
  of these pre-existing utilities until after I submitted my project
 proposal
  to my professor. So, I'm looking to implement a better search myself.
  Preferably by editing the existing portage code, not writing a separate
  program. So if anyone can offer any help regarding the actual
 implementation
  of search in portage, I would greatly appreciate it!

 Most of the search implementation is in
 /usr/lib/portage/pym/_emerge/__init__.py in class search.  The class's
 execute() method simply iterates over all packages (and descriptions
 and package sets) and matches against the searchkey.  You might need
 to look into pym/portage/dbapi/porttree.py for portdbapi as well.

 If you intend to index and support fast regex lookup, then you need to
 do some fancy indexing, which I'm not terribly familiar with.  You
 could follow in the steps of eix[1] or other indexed search utilities
 and design some sort of index layout, which is easier than the
 following thought.  You might consider implementing a suffix trie or
 similar that has sublinear regexp lookup and marshalling the structure
 for the index.  I couldn't find a python implementation for something
 like this, but here is a general trie class[2] that you might start
 with if you go that route.  There is a perl module[3],
 Tie::Hash::Regex, that does that, but properly implementing that in
 python would be a chore. :)

 That project sounds interesting and fun. Good luck!

 Lucian Poston

 [1] https://projects.gentooexperimental.org/eix/wiki/IndexFileLayout
 [2]
 http://www.koders.com/python/fid7B6BC1651A9E8BBA547552FE3F039479A4DECC45.aspx
 [3]
 http://search.cpan.org/~davecross/Tie-Hash-Regex-1.02/lib/Tie/Hash/Regex.pmhttp://search.cpan.org/%7Edavecross/Tie-Hash-Regex-1.02/lib/Tie/Hash/Regex.pm




Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread tvali
Yes if it would be a low-level implementation to portage, speeding up it's
native code for searching and using indexes, then it would make everything
faster, including emerge (because emerge does search first for package
relations). I have actually wanted to do it myself several years ago, so
reacting here to have my ideas discussed, too.

Douglas Anderson 16:46 reply is about locks and I think that it would need
to rethink portages locking methods - what, when and why it locks. This is
probably quite hard task by itself. Anyway, as portage actually lets user
make two emerges at the same time, locks might be OK, too.

I think that the best thing would be bottom-up refactoring - first to make a
list of lowest-level functions, which have to do with reading data from
portage tree or writing into it; then making indexer class, which will be
used by all of those low-level functions.

To have it OOP, it should be implemented in such way:

   - Low-level portage tree handler does everything with portage tree, no
   function in portage uses it directly.
   - Tree handler has several needed and several optional methods - so that
   implementing new handler would be easy, but creating things like native
   regex search would be possible.
   - One could implement a new tree handler with SQLite or other interface
   instead of filesystem and do other tricks through this interface; for
   example, boost it.

So, nice way to go would be:

   1. Implementing portage tree handler and it's proxy, which uses current
   portage tree in non-indexed way and simply gives methods for the same kind
   of access, as currently implemented one.
   2. Refactoring portage to rely only on portage tree handler and use
   direct portage tree nowhere. To test if it is so, portage tree should be
   moved to another directory, about which only this handler knows, and check
   if portage works well. Indexing all places, where portage uses it's tree
   handler (by std. comment, for example) and making clear, which methods would
   contain all boostable code of it.
   3. Implementing those methods in proxy, which could simulate fast regex
   search and other stuff using simplest possible interface of portage tree
   handler (smth. like four methods add, remove, get, list). Proxy should be
   able to use handler's methods if they are implemented.
   4. Refactoring portage to use advanced methods in proxy.
   5. Now, having taken all code together into one place and looking this
   nice and readable code, real optimizations could be discussed here, for
   example. Ideally, i think, portage would have such tree handlers:
  - Filesystem handler - fast searches over current portage tree
  structure.
  - SQL handler - rewrite of tree functions into SQL queries.
  - Network-based handler - maybe sometimes it would nice to have
  portage tree only in one machine of cluster, for example if I
want to have
  100 really small computers with fast connection with mother-computer and
  portage tree is too big to be wholly copied to all of them.
  - Memory-buffered handler with daemon, which is actually proxy to some
  other handler - daemon, which reads whole tree (from filesystem
or SQL) into
  memory on boot or first use, creates really fast index (because
now it does
  matter to have better indexing) and optionally deletes some [less needed]
  parts of it's index from memory when it's becoming full and behaves as
  really simple proxy if it stays full. This should be implemented after
  critical parts of filesystem or SQL handler.

2008/11/23 Emma Strubell [EMAIL PROTECTED]

 Wow, that's extremely helpful!! I happen to particularly enjoy tries, so
 the suffix trie sounds like a great idea. The trie class example is really
 helpful too, because this will be my first time programming in Python, and
 it's a bit easier to figure out what's going on syntax-wise in that simple
 trie class than in the middle of the portage source code!

 Seriously, thanks again :]


 On Sun, Nov 23, 2008 at 11:56 AM, Lucian Poston [EMAIL PROTECTED]wrote:

  Thanks for the replies! I know there are a couple programs out there
 that
  basically already do what I'm looking to do... Unfortunately I wasn't
 aware
  of these pre-existing utilities until after I submitted my project
 proposal
  to my professor. So, I'm looking to implement a better search myself.
  Preferably by editing the existing portage code, not writing a separate
  program. So if anyone can offer any help regarding the actual
 implementation
  of search in portage, I would greatly appreciate it!

 Most of the search implementation is in
 /usr/lib/portage/pym/_emerge/__init__.py in class search.  The class's
 execute() method simply iterates over all packages (and descriptions
 and package sets) and matches against the searchkey.  You might need
 to look into pym/portage/dbapi/porttree.py for portdbapi as well.

 If you intend to index and support fast regex 

Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Mike Auty

Hiya Emma,
	Good luck on your project.  A couple of things to be weary of are disk 
I/O, metadata cache backends and overlays.
	Disk I/O can be a significant bottleneck.  Loading up a lot of files 
from disk (be it the metadata cache or whatever) can take a long time 
initially, but then be cached in RAM and so be much faster to access in 
the future.
	Portage allows for its internal metadata cache to be stored in a 
variety of formats, as long as there's a backend to support it.  This 
means simple speedups can be achieved using cdb or sqlite (if you google 
these and portage you'll get gentoo-wiki tips, which unfortunately 
you'll have to read from google's cache at the moment).  It also means 
that if you want to make use of this metadata from within portage, 
you'll have to rely on the API to tell the backend to get you all the 
data (and it may be difficult to speed up without writing your own backend).
	Finally there are overlays, and since these can change outside of an 
emerge --sync (as indeed can the main tree), you'll have to reindex 
these before each search request, or give the user stale data until they 
manually reindex.
	If you're interesting in implementing this in python, you may be 
interested in another package manager that can handle the main tree, 
also implemented in python, called pkgcore.  From what I understand, 
it's a similar code-base to portage, but its internal architecture may 
have changed a lot.
	I hope some of that helps, and isn't off putting.  I look forward to 
seeing the results!  5:)

Mike  5:)



Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread René 'Necoro' Neumann
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Mike Auty schrieb:
 Finally there are overlays, and since these can change outside of an
 emerge --sync (as indeed can the main tree), you'll have to reindex
 these before each search request, or give the user stale data until they
 manually reindex.

Determining whether there has been a change to the ebuild system is a
major point in the whole thing. What does a great index serves you, if
it does not notice the changes the user made in his own local overlay?
:) Manually re-indexing is not a good choice I think...

If somebody comes up here with a good (and fast) solution, this would be
a nice thing ;) (need it myself).

Regards,
René
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkkp0kAACgkQ4UOg/zhYFuAhTACfYDxNeQQG6dysgU5TrNEZGOiH
3CoAn2wV6g8/8uj+T99cxJGdQBxTtZjI
=2I2j
-END PGP SIGNATURE-



Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread tvali
There is daemon, which notices about filesystem changes -
http://pyinotify.sourceforge.net/ would be a good choice.

In case many different applications use portage tree directly without using
any portage API (which is a bad choice, I think, and should be deprecated),
then there is a kind of hack - using
http://www.freenet.org.nz/python/lufs-python/ to create a new filesystem
(damn now I would like to have some time to join this game). I hope it's
possible to build it everywhere where gentoo should work, but it'n no
problem if it's not - you can implement it in such way that it's not needed.
I totally agree, that filesystem is a bottleneck, but this suffix trie would
check for directories first, I guess. Now, having this custom filesystem,
which actually serves portage tree like some odd API, you can have backwards
compability and still create your own thing.

Having such classes (numbers show implementation order; this is not
specified here if proxies are abstract classes, base classes or smth. other,
just it shows some relations between some imaginary objects):

   - *1. PortageTreeApi* - Proxy for different portage trees on FS or SQL or
   other.
   - *2. PortageTreeCachedApi *- same, as previous, but contains boosted
   memory cache. It should be able to save it's state, which is simply writing
   it's inner variables into file.
   - *3. PortageTreeDaemon *- has interface compatible with PortageTreeAPI,
   this daemon serves portage tree to PortageTreeFS and portage tree itself. In
   reality, it should be base class of *PortageTreeApi* and *
   PortageTreeCachedApi* so that they could be directly used as daemons.
   When cached API is used as daemon, it should be able to check filesystem
   changes - thus, implementations should contain change trigger callbacks.
   - *4. PortageTreeFS *- filesystem, which can be used to map any of those
   to filesystem. Connectable with PortageTreeApi or PortageTreeDaemon. This
   creates filesystems, which can be used for backwards-compability. This
   cannot be used on architectures, which dont implement lufs-python or analog.
   - *6. PortageTreeServer *- server, which serves data from
   PortageTreeDaemon, PortageTreeCachedApi or PortageTreeApi to some other
   computer.
   - Implementations can be proxied through *PortageTreeApi*, *
   PortageTreeCachedApi* or *PortageTreeDaemon*.
  - *5. PortageTreeImplementationAsSqlDb*
  - *1. PortageTreeImplementationAsFilesystem*
  - *3. PortageTreeImplementationAsDaemon* - client, actually.
  - *6. PortageTreeImplementationAsServer* - client, too.

So, *1* - creating PortageTreeApi and PortageTreeImplementationAsFilesystem
is pure refactoring task, at first. Then, adding more advanced functions to
PortageTreeApi is basically refactoring, too. PortageTreeApi should not
become too complex or contain any advanced tasks, which are not purely
db-specific, so some common baseclass could implement more high-level
things.
Then, *2* - this is finishing your schoolwork, but not yet in most powerful
way as we are having only index then, and first search is still slow. At
beginning this cache is unable to provide data about changes in portage tree
(which could be implemented by some versioning after this new api is only
place to update it), so it should have index update command and be only used
in search.
Then, *3* - having portage tree daemon means that things can really be
cached now and this cache can be kept in memory; also it means updates on
filesystem changes.
Then, *4* - having PortageTreeFS means that now you can easily implement
portage tree on faster medium without losing backwards-compability.
Now, *5* - implementation as SQL DB is logical as SQL is standardized and
common language for creating fast databases.
Eventually, *6* - this has really nothing to do with boosting search, but in
fast network it could still boost emerge by removing need for emerge --sync
for local networks.

I think that then it would be considered to have synchronization also in
those classes - CachedApi almost needs it to be faster with server-client
connections. After that, ImplementationAsSync and ImplementationAsWebRsSync
could be added and sync server built onto this daemon. As I really doubt
that emerge --sync is currently also ultraslow - I see no meaning in waiting
a long time to get few new items as currently seems to happen -, it would
boost another life-critical part of portage.

So, hope that helps a bit - have luck!

2008/11/23 René 'Necoro' Neumann [EMAIL PROTECTED]

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Mike Auty schrieb:
  Finally there are overlays, and since these can change outside of an
  emerge --sync (as indeed can the main tree), you'll have to reindex
  these before each search request, or give the user stale data until they
  manually reindex.

 Determining whether there has been a change to the ebuild system is a
 major point in the whole thing. What does a great index serves you, if
 it 

Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Marius Mauch
On Sun, 23 Nov 2008 07:17:40 -0500
Emma Strubell [EMAIL PROTECTED] wrote:

 However, I've started looking at the code, and I must admit I'm pretty
 overwhelmed! I don't know where to start. I was wondering if anyone
 on here could give me a quick overview of how the search function
 currently works, an idea as to what could be modified or implemented
 in order to improve the running time of this code, or any tip really
 as to where I should start or what I should start looking at. I'd
 really appreciate any help or advice!!

Well, it depends how much effort you want to put into this. The current
interface doesn't actually provide a search interface, but merely
functions to
1) list all package names - dbapi.cp_all()
2) list all package names and versions - dbapi.cpv_all()
3) list all versions for a given package name - dbapi.cp_list()
4) read metadata (like DESCRIPTION) for a given package name and
version - dbapi.aux_get()

One of the main performance problems of --search is that there is no
persistent cache for functions 1, 2 and 3, so if you're just
interested in performance aspects you might want to look into that.
The issue with implementing a persistent cache is that you have to
consider both cold and hot filesystem cache cases: Loading an index
file with package names and versions might improve the cold-cache case,
but slow things down when the filesystem cache is populated.
As has been mentioned, keeping the index updated is the other major
issue, especially as it has to be portable and should require little or
no configuration/setup for the user (so no extra daemons or special
filesystems running permanently in the background). The obvious
solution would be to generate the cache after `emerge --sync` (and other
sync implementations) and hope that people don't modify their tree and
search for the changes in between (that's what all the external tools
do). I don't know if there is actually a way to do online updates while
still improving performance and not relying on custom system daemons
running in the background.

As for --searchdesc, one problem is that dbapi.aux_get() can only
operate on a single package-version on each call (though it can read
multiple metadata variables). So for description searches the control
flow is like this (obviously simplified):

result = []
# iterate over all packages
for package in dbapi.cp_all():
# determine the current version of each package, this is 
# another performance issue.
version = get_current_version(package)
# read package description from metadata cache
description = dbapi.aux_get(version, [DESCRIPTION])[0]
# check if the description matches
if matches(description, searchkey):
result.append(package)

There you see the three bottlenecks: the lack of a pregenerated package
list, the version lookup for *each* package and the actual metadata
read. I've already talked about the first, so lets look at the other
two. The core problem there is that DESCRIPTION (like all standard
metadata variables) is version specific, so to access it you need to
determine a version to use, even though in almost all cases the
description is the same (or very similar) for all versions. So the
proper solution would be to make the description a property of the
package name instead of the package version, but that's a _huge_ task
you're probably not interested in. What _might_ work here is to add
support for an optional package-name-description cache that can be
generated offline and includes those packages where all versions have
the same description, and fall back to the current method if the
package is not included in the cache. (Don't think about caching the
version lookup, that's system dependent and therefore not suitable for
caching).

Hope it has become clear that while the actual search algorithm might
be simple and not very efficient, the real problem lies in getting the
data to operate on. 

That and the somewhat limited dbapi interface.

Disclaimer: The stuff below involves extending and redesigning some
core portage APIs. This isn't something you can do on a weekend, only
work on this if you want to commit yourself to portage development
for a long time.

The functions listed above are the bare minimum to
perform queries on the package repositories, but they're very
low-level. That means that whenever you want to select packages by
name, description, license, dependencies or other variables you need
quite a bit of custom code, more if you want to combine multiple
searches, and much more if you want to do it efficient and flexible.
See http://dev.gentoo.org/~genone/scripts/metalib.py and
http://dev.gentoo.org/~genone/scripts/metascan for a somewhat flexible,
but very inefficient search tool (might not work anymore due to old
age).

Ideally repository searches could be done without writing any
application code using some kind of query language, similar to how SQL
works for generic database searches (obviously not that complex).
But 

Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread devsk
 not relying on custom system daemonsrunning in the background.

Why is a portage daemon such a bad thing? Or hard to do? I would very much like 
a daemon running on my system which I can configure to sync the portage tree 
once a week (or month if I am lazy), give me a summary of hot fixes, security 
fixes in a nice email, push important announcements and of course, sync caches 
on detecting changes (which should be trivial with notify daemons all over the 
place) etc. Why is it such a bad thing?

Its crazy to think that security updates need to be pulled in Linux.

-devsk



- Original Message 
From: Marius Mauch [EMAIL PROTECTED]
To: gentoo-portage-dev@lists.gentoo.org
Sent: Sunday, November 23, 2008 7:12:57 PM
Subject: Re: [gentoo-portage-dev] search functionality in emerge

On Sun, 23 Nov 2008 07:17:40 -0500
Emma Strubell [EMAIL PROTECTED] wrote:

 However, I've started looking at the code, and I must admit I'm pretty
 overwhelmed! I don't know where to start. I was wondering if anyone
 on here could give me a quick overview of how the search function
 currently works, an idea as to what could be modified or implemented
 in order to improve the running time of this code, or any tip really
 as to where I should start or what I should start looking at. I'd
 really appreciate any help or advice!!

Well, it depends how much effort you want to put into this. The current
interface doesn't actually provide a search interface, but merely
functions to
1) list all package names - dbapi.cp_all()
2) list all package names and versions - dbapi.cpv_all()
3) list all versions for a given package name - dbapi.cp_list()
4) read metadata (like DESCRIPTION) for a given package name and
version - dbapi.aux_get()

One of the main performance problems of --search is that there is no
persistent cache for functions 1, 2 and 3, so if you're just
interested in performance aspects you might want to look into that.
The issue with implementing a persistent cache is that you have to
consider both cold and hot filesystem cache cases: Loading an index
file with package names and versions might improve the cold-cache case,
but slow things down when the filesystem cache is populated.
As has been mentioned, keeping the index updated is the other major
issue, especially as it has to be portable and should require little or
no configuration/setup for the user (so no extra daemons or special
filesystems running permanently in the background). The obvious
solution would be to generate the cache after `emerge --sync` (and other
sync implementations) and hope that people don't modify their tree and
search for the changes in between (that's what all the external tools
do). I don't know if there is actually a way to do online updates while
still improving performance and not relying on custom system daemons
running in the background.

As for --searchdesc, one problem is that dbapi.aux_get() can only
operate on a single package-version on each call (though it can read
multiple metadata variables). So for description searches the control
flow is like this (obviously simplified):

result = []
# iterate over all packages
for package in dbapi.cp_all():
# determine the current version of each package, this is 
# another performance issue.
version = get_current_version(package)
# read package description from metadata cache
description = dbapi.aux_get(version, [DESCRIPTION])[0]
# check if the description matches
if matches(description, searchkey):
result.append(package)

There you see the three bottlenecks: the lack of a pregenerated package
list, the version lookup for *each* package and the actual metadata
read. I've already talked about the first, so lets look at the other
two. The core problem there is that DESCRIPTION (like all standard
metadata variables) is version specific, so to access it you need to
determine a version to use, even though in almost all cases the
description is the same (or very similar) for all versions. So the
proper solution would be to make the description a property of the
package name instead of the package version, but that's a _huge_ task
you're probably not interested in. What _might_ work here is to add
support for an optional package-name-description cache that can be
generated offline and includes those packages where all versions have
the same description, and fall back to the current method if the
package is not included in the cache. (Don't think about caching the
version lookup, that's system dependent and therefore not suitable for
caching).

Hope it has become clear that while the actual search algorithm might
be simple and not very efficient, the real problem lies in getting the
data to operate on. 

That and the somewhat limited dbapi interface.

Disclaimer: The stuff below involves extending and redesigning some
core portage APIs. This isn't something you can do on a weekend, only
work on this if you want to commit yourself to portage development
for a long 

Re: [gentoo-portage-dev] search functionality in emerge

2008-11-23 Thread Marius Mauch
On Sun, 23 Nov 2008 21:01:40 -0800 (PST)
devsk [EMAIL PROTECTED] wrote:

  not relying on custom system daemonsrunning in the background.
 
 Why is a portage daemon such a bad thing? Or hard to do? I would very
 much like a daemon running on my system which I can configure to sync
 the portage tree once a week (or month if I am lazy), give me a
 summary of hot fixes, security fixes in a nice email, push important
 announcements and of course, sync caches on detecting changes (which
 should be trivial with notify daemons all over the place) etc. Why is
 it such a bad thing?

Well, as an opt-in solution it might work (though most of what you
described is IMO just stuff for cron, no need to reinvent the wheel).

What I was saying is that _relying_ on custom system
daemons/filesystems for a _core subsystem_ of portage is the wrong
way, simply because it adds a substantial amount of complexity to the
whole package management architecture. It's one more thing that can
(and will) break, one more layer to take into account for any design
decisions, one more component that has to be secured, one more obstacle
to overcome when you want to analyze/debug things.
And special care must be taken if it requires special kernel support
and/or external packages. Do you want to make inotify support mandatory
to use portage efficiently? (btw, looks like inotify doesn't really
work with NFS mounts, which would already make such a daemon completely
useless for people using a NFS-shared repository)

And finally, if you look at the use cases, a daemon is simply overkill
for most cases, as the vast majority of people only use emerge
--sync (or wrappers) and maybe layman to change the tree, usually once
per day or less often. Do you really want to push another system daemon
on users that isn't of use to them?

 Its crazy to think that security updates need to be pulled in Linux.

That's IMO better be handled via an applet (bug #190397 has some code),
or just check for updates after a sync (as syncing is the only
way for updates to become available at this time). Maybe a message
could be added after sync if there are pending GLSAs, now that the glsa
support code is in portage.

Marius



[gentoo-portage-dev] Re: search functionality in emerge

2008-11-23 Thread Duncan
devsk [EMAIL PROTECTED] posted
[EMAIL PROTECTED], excerpted below, on  Sun, 23
Nov 2008 21:01:40 -0800:

 Why is a portage daemon such a bad thing? Or hard to do? I would very
 much like a daemon running on my system which I can configure to sync
 the portage tree once a week (or month if I am lazy), give me a summary
 of hot fixes, security fixes in a nice email, push important
 announcements and of course, sync caches on detecting changes (which
 should be trivial with notify daemons all over the place) etc. Why is it
 such a bad thing?
 
 Its crazy to think that security updates need to be pulled in Linux.

Well, this is more a user list discussion than a portage development 
discussion, but...

For one thing, it's terribly inefficient to keep a dozen daemons running 
checking only a single thing each, each week, when we have a cron 
scheduling daemon, and it's both efficient and The Unix Way (R) to setup 
a script to do whatever you need it to do, and then have the cron daemon 
run each of a dozen different scripts once each week, instead of having 
those dozen different daemons running constantly when they're only active 
once a week.

IOW, it only requires a manual pull if you've not already setup cron to 
invoke an appropriate script once a week, and that involves only a single 
constantly running daemon, the cron daemon of your choice.

Now, perhaps it can be argued that there should be a package that 
installs such a pre-made script.  For all I know, maybe there is one 
already.  And perhaps it can be argued that said script, if optional, 
should at least be mentioned in the handbook.  I couldn't argue with the 
logic of either of those.  But there's no reason to run yet another 
daemon constantly, when (1) it's not needed constantly, and (2), there's 
already a perfectly functional way of scheduling something to run when 
it /is/ needed, complete with optional results mailing, etc, if it's 
scripted to do that.

-- 
Duncan - List replies preferred.   No HTML msgs.
Every nonfree program has a lord, a master --
and if you use the program, he is your master.  Richard Stallman




[gentoo-portage-dev] unsubscribe

2008-11-23 Thread Lanza Fabrizio
 

AVVISO DI RISERVATEZZA 
Informazioni riservate possono essere contenute nel messaggio 
o nei suoi allegati. 
Se non siete i destinatari indicati nel messaggio, o responsabili 
per la sua consegna alla persona, o se avete ricevuto il messaggio 
per errore, siete pregati di non trascriverlo, copiarlo o inviarlo a nessuno. 
In tal caso vi invitiamo a cancellare il messaggio ed i suoi allegati. 
Grazie.
 
CONFIDENTIALITY NOTICE 
Confidential information may be contained in this message 
or in its attachments. 
If you are not the addressee indicated in this message, or responsible 
for message delivering to that person, or if you have received this message 
in error, you may not transcribe, copy or deliver this message to anyone. 
In that case, you should delete this message and its attachments. 
Thank you.