Re: [ZODB-Dev] Searching/wo/Zope
Tamas Hegedus wrote at 2006-1-6 15:54 -0500: > ... >Answer for Dieter's email: >Several days ago somebody already suggested to use IndexedCatalog. But >it have not seem to be very active. Maybe, because it is good enough? -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Searching/wo/Zope
Hi, Tell me if you are tired from me. I do not use to write so long emails to email lists... Answer for Dieter's email: Several days ago somebody already suggested to use IndexedCatalog. But it have not seem to be very active. The stand alone ZCatalog definitely not active. My decision to use ZCatalog was: it is an extraction of code from Zope; Zope will be always maintained (isn't it?), just I have to figure out how to 'hack out' the stand alone ZCatalog based on Kevin Dangoor's 'hack'. Kevin already mentioned that Zope 3 would be better for this purpose. At this moment I think the best way for me in long term: handle the querying through a wrapper class of mine. Put a layer between my application(s) and the indexing application (something similar to RDBMS DB-API; changing the db backend (in my case changing the indexing/searching backend) without (big) changes in the application code). Does some similar standards exist for ODBMSs? Similar to DB-API... - Please do not forget that I am not a real programmer but a consumer: That's OK -- in return, please don't forget that you're posting to a ZODB developer's list ;-). Not a real programmer: I have not learn it in the school; I read and try to use developer lists (and brains) ;-) I just want to state: it is very difficult for me to formulate my questions in a way you understand them easily. Although I think I put more then enough time and energy to form relevant questions with relevant terminology avoid wastin the time of other people. -- That's the only solid reason Zope Corp has to _pay_ for ZODB development. Zope pays the bills here, and ZODB is supporting infrastructure for Zope. Then OK. But if there is bigger potential inside it... Then what, specifically? Nobody works on something unless they want to, and/or are paid to. It's not a matter of cheerleading, it's a matter of someone doing the work. -- Yes, I understand, that you have to feed your family and educate your child. So if it is not payed... I am surprised that you ask 'then what, specificaly'. My opinion (as an outsider): One of the major and important trend in application development: object persistance. At this moment most of the developers use (I may not be true) some solution with RDBMS backend (SQLObject/Python; Hybernate/Java). But I think these solutions are not so transparent as ODBMSs, like ZODB, db4o (I tried these). I think other programmers mostly use a solution with an RDBMS backend as they do not want to handle/code the searching/indexing (I know that ODBMS performance told to be not so good as RDBMS, but I think if you calculate with the OR mapping overhead than some ODBMS are not so bed. E.g. some heavily loaded biological application Versant's ODBMS was reported better then one of the leader RDBMS ***). "Specificaly" I think if you would implement a stand alone search/index possibility for ZODB (if you would use some similar Zope-hacking approach it would NOT be a huge, completely stand-alone project needing a lot of efforts) ZODB could be an ODBMS leader, competitor of other ODBMS solutions. So you may have more paying consumers, too... (footnotes) (***My biggest problem is: if I want other biologists to try my solutions/applications I do not have any chance for this with an RDBMS backend. They just will not learn how to setup and will not setup an RDBMS to try out something. For this reason an ODBMS system would be OK for me.) (I think it would be also worth to implement some standards for ODBMSs. Some ODB-API. Just to be able to change ODBMS backends and/or indexing/searching backends. (I also know that there is no other ODBMS backend to choose from and you do not want to switch users from ZODB to other backend in the future :-) , but I think you would have the potential to do something like this. Just for fun. Just to initiate some ODBMS standardization) A notable exception is IndexedCatalog: http://www.async.com.br/projects/IndexedCatalog/ which is independent of Zope. You said before that you thought that wasn't active, and it indeed doesn't look like it's had a release recently. That could be because it's already perfect ;-) -- or it could be that there's not I am not a 'real' developer, but it seems others also would not go with it, even though if it is perfect. If you (ZODB developers) change something in the next release of ZODB than the perfect IndexedCatalog may not be able to communicate with the new version of the ZODB backend... (By the way: I think it is great and big, and I would like to use it.) To formulate this on a more realistic way: it seems for me that there is no potential to take care about this extra proje
Re: [ZODB-Dev] Searching/wo/Zope
Tamas Hegedus wrote at 2006-1-4 14:38 -0500: >You have released the ZODB/ZEO as a stand alone package. But there is no >stand alone searching possibility. Why? This just does not make any sense. > >Yous suggested and I tried Kevin Dangoor's stand alone ZCatalog. It uses >the Zope source just remove the zope dependences, etc (some type of >hack). But it is not maintained. I think releasing and maintaining >something similar would not be a big work, but would make ZODB/ZEO at >much higher rank among the ODBMSs. People would start to build projects >on them... There is also a stand-alone querying solution. I think, its name is "IndexedCatalog". Search the list archives, to locate it. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Searching/wo/Zope
[Tamas Hegedus] > Please do not forget that I am not a real programmer but a consumer: That's OK -- in return, please don't forget that you're posting to a ZODB developer's list ;-). > 0. I accept if the policy is making ZODB just for Zope. That's the only solid reason Zope Corp has to _pay_ for ZODB development. Zope pays the bills here, and ZODB is supporting infrastructure for Zope. ZODB is an open project, though, and others are free to contribute-- or even to pay others for --ZODB work that doesn't directly support a glaring Zope need. > Then OK. But if there is bigger potential inside it... Then what, specifically? Nobody works on something unless they want to, and/or are paid to. It's not a matter of cheerleading, it's a matter of someone doing the work. For example, I'm sure that someone so motivated could produce an add-on package for ZODB supporting some vision of automatic index/catalog/search. Most people who are likely to contribute time and/or money to do such a thing are likely to do it in the context of Zope, though, simply because Zope pays their bills too. For example, Dieter Maurer has done some hard work on speeding searches (see IncrementalSearch and IncrementalSearch2 at: http://www.dieter.handshake.de/pyprojects/zope ). A notable exception is IndexedCatalog: http://www.async.com.br/projects/IndexedCatalog/ which is independent of Zope. You said before that you thought that wasn't active, and it indeed doesn't look like it's had a release recently. That could be because it's already perfect ;-) -- or it could be that there's not a large enough community who wants it to support ongoing development. I don't know. > (By the way: I think it is great and big, and I would like to use it.) > To formulate this on a more realistic way: it seems for me that there > is no potential to take care about this extra project outside of Zope > AND/OR it would not be good for Zope developers to have it as an > easy-to-use stand alone module (maybe some business policy?). Not sure I followed that. > 1. """That's usually viewed as an application-level problem, and it's up > to applications to solve it in ways best suited for their particular > needs.""" If I translate this for myself, if I understand well: I am very > happy that RDBMSs does not say this, and I can search them not only by > primary keys; I am happy that I do not have to implement something > similar to SQL as it is not considered as "application level problem". A relational database forces you to slam all your data into uniform tables, regardless of whether that's a natural fit. When all you have is uniform tables, then it's relatively easy to define uniform operators for crawling over those tables -- that's what SQL is all about. An object database is more of a general graph structure, and an application's idea of "search" can be correspondingly semantically richer from the start -- or even irrelevant, if the object graph is constructed from the start to make traversals of potential interest follow the natural graph pointers. What's the analogue to SQL in this quite different view of the world? Well, there isn't a standard accepted vision for that. That's what makes it the app's problem. These are tradeoffs. Zope's assorted indices and catalogs _probably_ capture some notion of "search" close to what you're after. > 2. BTrees: I could not find any 'built-in' possibility in the docs, just > the 'primary keys'. If I check the OOBTree, etc, it just give > 'difference', 'intersection', 'union'. I do not see to do full text > search or field search on BTrees. Do I miss something??? BTrees map keys to values. The keys are always maintained in sorted order, and it's both dead easy and efficient to do range searches over a BTree's key space. That's what's built in. > (I do not think as if I would than you would not call the problem > "application" level problem). It depends on the app. I gave SpamBayes as a concrete example of a real app where the builtin abilities of BTrees do all the searching the app requires (and it's not an accident that SpamBayes was designed that way -- just as it wouldn't have been an accident if an RDMS fan had designed SpamBayes to work directly with simple RD tables). > 3. I can not build up another database from the ZODB as I am not a > developer. Do you use Python? I'm at a bit of a loss to figure out how you wound up posting here if you're _not_ a Python programmer. It could be that ZODB is much more general than I thought ;-), but I didn't think non-programmers would have any use for it. > But I think you formulated this not the best way: I think you do not > build the SB database OUT of ZODB's BTrees, I think you just build > up indexes from the BTrees and you implement searches on your indexes > that points back to the BTrees. I suppose you could think of it that way, but I designed SpamBayes and that's not how I thought of it. I thought of it in terms of abstract
Re: [ZODB-Dev] Searching/wo/Zope
Hi, Please do not forget that I am not a real programmer but a consumer: 0. I accept if the policy is making ZODB just for Zope. Then OK. But if there is bigger potential inside it... (By the way: I think it is great and big, and I would like to use it.) To formulate this on a more realistic way: it seems for me that there is no potential to take care about this extra project outside of Zope AND/OR it would not be good for Zope developers to have it as an easy-to-use stand alone module (maybe some business policy?). 1. """That's usually viewed as an application-level problem, and it's up to applications to solve it in ways best suited for their particular needs.""" If I translate this for myself, if I understand well: I am very happy that RDBMSs does not say this, and I can search them not only by primary keys; I am happy that I do not have to implement something similar to SQL as it is not considered as "application level problem". 2. BTrees: I could not find any 'built-in' possibility in the docs, just the 'primary keys'. If I check the OOBTree, etc, it just give 'difference', 'intersection', 'union'. I do not see to do full text search or field search on BTrees. Do I miss something??? (I do not think as if I would than you would not call the problem "application" level problem). 3. I can not build up another database from the ZODB as I am not a developer. But I think you formulated this not the best way: I think you do not build the SB database OUT of ZODB's BTrees, I think you just build up indexes from the BTrees and you implement searches on your indexes that points back to the BTrees. => If you build up a new database why do you use ZODB? => If you just build indexes from the BTrees, the following protocol works for me and you can suggest? 1. walk trough on your BTree taking each object 2. with an external indexing application build the index (on one or more fields, or full text) 3. search in your index that returns with the 'primary key' of objects in the ZODB 4. get the objects from the ZODB via the 'primary keys' from the prev step. ??? Thanks for your comments and suggestion in advance, Tamas Tim Peters wrote: > [Tamas Hegedus] > >> You have released the ZODB/ZEO as a stand alone package. But there is no >> stand alone searching possibility. Why? This just does not make any >> sense. > > > > It makes sense if you know why ZODB is released this way ;-) Primarily, > it's so that people running Zope can install ZEO servers on other machines, > without needing to install Zope there too. If others can make use of > standalone ZODB too, that's great, but it's not the primary intent. > > There are no current plans to add index/catalog/search code to the ZODB > distribution. That's usually viewed as an application-level problem, and > it's up to applications to solve it in ways best suited for their particular > needs. > > As an example, I'm the titular head of the SpamBayes project: > > http://spambayes.sourceforge.net > > and we're in the process there of adopting ZODB as the default backend > database. While SpamBayes needs to search its database frequently and > heavily, there's nothing ZODB could add that we would use! Instead we'll > build the SB database directly out of ZODB's BTrees, whose builtin searching > abilities are a perfect fit to what SB needs. > > Are you quite sure that someone else's idea of a general-purpose searching > facility is something you'd actually use? Don't underestimate the > difficulties in crafting a "one size fits all" approach here that actually > fits anyone ;-) > -- Tamas Hegedus, PhD | phone: (1) 919-966 0329 UNC - Biochem & Biophys | fax: (1) 919-966 5178 5007A Thurston-Bowles Bldg | mailto:[EMAIL PROTECTED] Chapel Hill, NC, 27599-7248 | http://biohegedus.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Searching/wo/Zope
[Tamas Hegedus] > You have released the ZODB/ZEO as a stand alone package. But there is no > stand alone searching possibility. Why? This just does not make any > sense. It makes sense if you know why ZODB is released this way ;-) Primarily, it's so that people running Zope can install ZEO servers on other machines, without needing to install Zope there too. If others can make use of standalone ZODB too, that's great, but it's not the primary intent. There are no current plans to add index/catalog/search code to the ZODB distribution. That's usually viewed as an application-level problem, and it's up to applications to solve it in ways best suited for their particular needs. As an example, I'm the titular head of the SpamBayes project: http://spambayes.sourceforge.net and we're in the process there of adopting ZODB as the default backend database. While SpamBayes needs to search its database frequently and heavily, there's nothing ZODB could add that we would use! Instead we'll build the SB database directly out of ZODB's BTrees, whose builtin searching abilities are a perfect fit to what SB needs. Are you quite sure that someone else's idea of a general-purpose searching facility is something you'd actually use? Don't underestimate the difficulties in crafting a "one size fits all" approach here that actually fits anyone ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Searching/wo/Zope
approach to get this done it to either work on the issue on your own or pay someone...this is just a question of developer resources and priorities. I do not have any finantial resources. So I have to do on my own way... Regards, Tamas -- Tamas Hegedus, PhD | phone: (1) 919-966 0329 UNC - Biochem & Biophys | fax: (1) 919-966 5178 5007A Thurston-Bowles Bldg | mailto:[EMAIL PROTECTED] Chapel Hill, NC, 27599-7248 | http://biohegedus.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Searching/wo/Zope
--On 4. Januar 2006 15:11:37 -0500 Tamas Hegedus <[EMAIL PROTECTED]> wrote: I am not a programmer. This is the source of all my problems. I just did not have the patient (and knowledge) to learn to develop zope applications. But I want to stick with Python and I need a transparent (object) database for my objects. ZODB is very promising. But without querying possibility... Without a maintained querying possibility... Most ppl use the ZODB together with Zope. So likely all related resources are focused on Zope issues. I agree that it might be of interest to support the ZODB and the ZCatalog outside Zope but as Zope core developer I have personally no time for such projects and little interests. So the basic approach to get this done it to either work on the issue on your own or pay someone...this is just a question of developer resources and priorities. From the Zope point of view I consider this issue as an issue of lowest priority. -aj pgpFJSpd7cCzl.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Searching/wo/Zope
Using the catalog stuff from Zope 3(!) would make more sense than dealing with the Zope 2 catalog. This would be great! I think releasing and maintaining something similar would not be a big work, but would make ZODB/ZEO at much higher rank among the ODBMSs. Do you volunteer to take work on this issue? I would be happy. Just somebody should help me a lot at the first time. E.g. I do not know how to create a python package. I was not able to find the 'ZCatalog' in my Zope source. I do not know Zope or ZODB internals, etc. I am not a programmer. This is the source of all my problems. I just did not have the patient (and knowledge) to learn to develop zope applications. But I want to stick with Python and I need a transparent (object) database for my objects. ZODB is very promising. But without querying possibility... Without a maintained querying possibility... Take everything together, I would be very happy to maintain, but I would need help (let say, I would be very proud to be involved in a serious, real world project :-) ). Tamas -- Tamas Hegedus, PhD | phone: (1) 919-966 0329 UNC - Biochem & Biophys | fax: (1) 919-966 5178 5007A Thurston-Bowles Bldg | mailto:[EMAIL PROTECTED] Chapel Hill, NC, 27599-7248 | http://biohegedus.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Searching/wo/Zope
--On 4. Januar 2006 14:38:13 -0500 Tamas Hegedus <[EMAIL PROTECTED]> wrote: Hi! You have released the ZODB/ZEO as a stand alone package. But there is no stand alone searching possibility. Why? This just does not make any sense. Yous suggested and I tried Kevin Dangoor's stand alone ZCatalog. It uses the Zope source just remove the zope dependences, etc (some type of hack). But it is not maintained. I think releasing and maintaining something similar would not be a big work, but would make ZODB/ZEO at much higher rank among the ODBMSs. People would start to build projects on them... Using the catalog stuff from Zope 3(!) would make more sense than dealing with the Zope 2 catalog. I think releasing and maintaining something similar would not be a big work, but would make ZODB/ZEO at much higher rank among the ODBMSs. Do you volunteer to take work on this issue? -aj pgpsKDk7GvwAC.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Searching/wo/Zope
Hi! You have released the ZODB/ZEO as a stand alone package. But there is no stand alone searching possibility. Why? This just does not make any sense. Yous suggested and I tried Kevin Dangoor's stand alone ZCatalog. It uses the Zope source just remove the zope dependences, etc (some type of hack). But it is not maintained. I think releasing and maintaining something similar would not be a big work, but would make ZODB/ZEO at much higher rank among the ODBMSs. People would start to build projects on them... Regards, Tamas -- Tamas Hegedus, PhD | phone: (1) 919-966 0329 UNC - Biochem & Biophys | fax: (1) 919-966 5178 5007A Thurston-Bowles Bldg | mailto:[EMAIL PROTECTED] Chapel Hill, NC, 27599-7248 | http://biohegedus.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
