Re: [ZODB-Dev] SpatialIndex
Am 28.06.2010, 23:42 Uhr, schrieb Laurence Rowe l...@lrowe.co.uk: On 28 June 2010 21:27, Nitro ni...@dr-code.org wrote: ZODB is a general python object database with a much wider audience than just plone. It suits desktop applications just as well as applications you'd normally use twisted and pickle for. Forcing all those zope dependencies like buildout on people does not add to the attractiveness of ZODB for users outside zope. Having indices only in plone does also not make sense. Many applications would benefit from keyword, field, full-text, spatial, younameit indices. Yet extracting individual packages from zope/plone is impossible due to the slew of dependencies. While I can accept a dependency like zope.interface I don't accept a lot of the others. It really prevents ZODB from living up to its full potential in non-plone applications. Remember that Plone is an eight year old application that is built on top of a 12 year old Application server. There has been much progress since then (and plenty of people who build non-Plone ZODB based applications), but the size of the codebase means it is not possible to always be using the current best practice. http://zope2.zope.org/about-zope-2/the-history-of-zope Nobody would recommend that you try to extract stuff from Plone or Zope2. In my opinion there are two main sources of packages for non-Zope2 dependent applications. * The ZTK extracted the core of Zope 3 and is used in application servers such as Grok and BlueBream. It contains zope.catalog and it's related packages. There are several extensions on top of this such as zc.catalog and hurry.query. The 1.0 release has not been there yet, but the underlying packages are stable. Installing zcatalog requires a total of 34 packages (ZODB3 requires 10) http://docs.zope.org/zopetoolkit/releases/packages-trunk.html * The Repoze project has focussed on making zope technologies more easily accessible to applications outside of Zope. Whilst the ZTK project has improved things a lot, it is still a relatively large chunk to swallow whole. repoze.catalog is extracted from zope.catalog and requires only zope.index in addition to ZODB3. http://docs.repoze.org/catalog/ At the very lowest level are the indexes themselves such as zope.index and zc.relation, a spatial index would fit in here too. (Health warning: I'm mostly a Plone developer, so do not yet have experience using these packages) This is all very useful information. Thanks for the pointers Laurence. -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Hi, On 30/06/10 00:39, Nitro wrote: This is all very useful information. Thanks for the pointers Laurence. I'm glad you(we)'ve come up to a useful conclussion: we need clear, ordered, findable, consensued information (documentation) on a very complex technological organism. (Thanks Alan, I'll try to push it in my company) -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Enrique Pérez Arnaud epe...@yaco.es Yaco Sistemas SL| http://www.yaco.es C/ Rioja 5, 41001 Sevilla (España) Tel: (+34) 954 50 00 57 Fax 954 50 09 29 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Nitro wrote: packaging: I don't plan to create a package for this as I don't see much point in adding yet another package to the clutter of packages surrounding zodb. Just a quick remark: Without being available as a package, your code will be far less useful (if not outright useless) to a lot of potential users because installing packages in one way or another is what people do in order to be able to use code, in particular in the Zope and ZODB world. In fact, reading your message through the Gmane interface to the list, I didn't even receive the attachment to your message. (I know I can dig it up from the list archives, but compare this to being able to declare the dependency on your package in my code and have it installed automatically by my build system.) I also don't see how creating another package could do any harm, in particular as it brings original new functionality. -- Thomas ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Am 28.06.2010, 08:21 Uhr, schrieb Thomas Lotze tho...@thomas-lotze.de: Nitro wrote: packaging: I don't plan to create a package for this as I don't see much point in adding yet another package to the clutter of packages surrounding zodb. Just a quick remark: Without being available as a package, your code will be far less useful (if not outright useless) to a lot of potential users because installing packages in one way or another is what people do in order to be able to use code, in particular in the Zope and ZODB world. In fact, reading your message through the Gmane interface to the list, I didn't even receive the attachment to your message. (I know I can dig it up from the list archives, but compare this to being able to declare the dependency on your package in my code and have it installed automatically by my build system.) I also don't see how creating another package could do any harm, in particular as it brings original new functionality. I do not like the current packaging approach as I discussed in length in a recent message to this list [1]. Until this changes I am not going to spend time creating random packages. -Matthias [1] http://aspn.activestate.com/ASPN/Mail/Message/zodb/3867973 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Am 28.06.2010, 14:10 Uhr, schrieb Dylan Jay d...@pretaweb.com: I don't use a lot of other indexes other than what comes with plone but I can see the value of what your suggesting in having an installable tested collection of indexes. I can also see that this is a really big itch for you and you've already identified a bunch of candidates to include. So why not go the next step and create a package that's nothing more than requires specifications and release it with versions corresponding to zodb releases. Then others may find it useful and help you support it. And if it's really popular it may even get taken into consideration with zodb packaging. Who knows? Thanks for your feedback, Dylan. My main problem is not the lack of an index collection. It's one of the problems I faced, but not the main one. Indexing is just a small part of working with a database. The main problem (imo) is that there are already 50 zodb related packages on pypi and none of them gathered a lot of people working on them. I don't see why this should be any different if I publish yet another package. Especially if most people use plone and the built-in indices. Just look at what happened to ZCatalog Standalone. Here's a little metaphor for what I'm trying to say: Once upon a time there was a man who wanted to go to the bakery to buy a bread. He thought I'll be done with this quickly, after all many people want to buy a bread. So he went off to visit the ZBakery. When asking for a bread, the people in the ZBakery told him there's no need to sell whole breads. They said: See, we have all the ingredients here so you can make a bread suiting your own taste. Look, there's ZFlour, ZMilk and ZSalt. And if you rummage the corners of this bakery, you'll also might find ZFlour2, CustomFlour and MyOwnCoolFlour. We don't know if they are any good, because each flour is used by just one or two people.. The man thought about it for a while and went off to try the different flours. When he wanted to try the CustomFlour it did not work. It turned out this was because CustomFlour relied on 3rdPartyMill and 3rdPartyMill had a problem, so CustomFlour was broken. The man shaked his head after he realized a few dozen people already tried to get CustomFlour and nobody pointed out the problem to its producer. Finding out about all of this took the whole morning and so he finally made lunch break. After his lunch break was over he finally found a flour suiting his bread he went off to look at the different milks and salts. He experienced similar problems there. One of the milks had just a label milk on it, the other areas of the packaging were blank. The man had no idea if the milk in question would work for his bread or not. So he had to analyze the contents of the milk to see if it might be useful. It turned out the milk was mislabeled and not a milk. As the sun was already touching the horizon and the air was getting cold the man ignored the milk for the time being and went looking for salt. He did not have to search for long and was delighted to find a single salt which would just work. When the man looked out of a window of the ZBakery he saw it was already dark and went home. When lying in bed he thought to himself: All I wanted to have this morning was a bread. Now I'm about to fall asleep and still don't have one. The bakery even had all the ingredients! But why did they made me try and analyze each ingredient? I even would've taken a bread which tasted a bit worse then the bread which I now have to bake on my own. The other customers of ZBakery surely also want breads, rolls and cake. Aren't they interested in creating a standard package of breads, rolls and cake? If they'd work together on a single bread, they'd all benefit from the improved recipes. New customers would immediately notice that there's a good default bread which many people like. These customers might point their colleagues at ZBakery, because it sells tasty, ready-to-use breads. If there was a special customer he could still bake his own bread using the individual ingredients. Pondering all these things he slided into a deep sleep. When he woke up the next morning he found a handful of committed people who had gathered in the ZBakery to bake and sell their first bread together... -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
On 28 June 2010 15:23, Nitro ni...@dr-code.org wrote: Am 28.06.2010, 14:10 Uhr, schrieb Dylan Jay d...@pretaweb.com: I don't use a lot of other indexes other than what comes with plone but I can see the value of what your suggesting in having an installable tested collection of indexes. I can also see that this is a really big itch for you and you've already identified a bunch of candidates to include. So why not go the next step and create a package that's nothing more than requires specifications and release it with versions corresponding to zodb releases. Then others may find it useful and help you support it. And if it's really popular it may even get taken into consideration with zodb packaging. Who knows? Thanks for your feedback, Dylan. My main problem is not the lack of an index collection. It's one of the problems I faced, but not the main one. Indexing is just a small part of working with a database. The main problem (imo) is that there are already 50 zodb related packages on pypi and none of them gathered a lot of people working on them. I don't see why this should be any different if I publish yet another package. Especially if most people use plone and the built-in indices. Just look at what happened to ZCatalog Standalone. Here's a little metaphor for what I'm trying to say: Once upon a time there was a man who wanted to go to the bakery to buy a bread. He thought I'll be done with this quickly, after all many people want to buy a bread. So he went off to visit the ZBakery. When asking for a bread, the people in the ZBakery told him there's no need to sell whole breads. They said: See, we have all the ingredients here so you can make a bread suiting your own taste. Look, there's ZFlour, ZMilk and ZSalt. And if you rummage the corners of this bakery, you'll also might find ZFlour2, CustomFlour and MyOwnCoolFlour. We don't know if they are any good, because each flour is used by just one or two people.. The man thought about it for a while and went off to try the different flours. When he wanted to try the CustomFlour it did not work. It turned out this was because CustomFlour relied on 3rdPartyMill and 3rdPartyMill had a problem, so CustomFlour was broken. The man shaked his head after he realized a few dozen people already tried to get CustomFlour and nobody pointed out the problem to its producer. Finding out about all of this took the whole morning and so he finally made lunch break. After his lunch break was over he finally found a flour suiting his bread he went off to look at the different milks and salts. He experienced similar problems there. One of the milks had just a label milk on it, the other areas of the packaging were blank. The man had no idea if the milk in question would work for his bread or not. So he had to analyze the contents of the milk to see if it might be useful. It turned out the milk was mislabeled and not a milk. As the sun was already touching the horizon and the air was getting cold the man ignored the milk for the time being and went looking for salt. He did not have to search for long and was delighted to find a single salt which would just work. When the man looked out of a window of the ZBakery he saw it was already dark and went home. When lying in bed he thought to himself: All I wanted to have this morning was a bread. Now I'm about to fall asleep and still don't have one. The bakery even had all the ingredients! But why did they made me try and analyze each ingredient? I even would've taken a bread which tasted a bit worse then the bread which I now have to bake on my own. The other customers of ZBakery surely also want breads, rolls and cake. Aren't they interested in creating a standard package of breads, rolls and cake? If they'd work together on a single bread, they'd all benefit from the improved recipes. New customers would immediately notice that there's a good default bread which many people like. These customers might point their colleagues at ZBakery, because it sells tasty, ready-to-use breads. If there was a special customer he could still bake his own bread using the individual ingredients. Pondering all these things he slided into a deep sleep. When he woke up the next morning he found a handful of committed people who had gathered in the ZBakery to bake and sell their first bread together... There are some valid criticisms in here. One problem with PyPI is that there is no way to clearly mark a package as having been superseded, as zc.relationship was by zc.relation. So why don't we all work on the same packages? The main reason is one of legacy. Plone is built on Zope2 and ZCatalog. It works, but it is not without it's issues - we can't have queries that join from that catalog to a zc.relation catalog. Standalone ZCatalog failed because it came to early - Zope2 was only recently eggified, so to be successful the standalone ZCatalog would need to be used in Zope2.
Re: [ZODB-Dev] SpatialIndex
Although I think this sounds like an optimal solution, I think it isn't necessary. I wish only that there was a complements in ZODB wiki with a simple list of best of breed package for each task and maybe a list of alternatives. To make it even better a good deprecated section would be good so you don't need to look at old packages trying to figure out if they are the new thing that is not on the wiki or the so-old-that-no-one-uses thing. This would be a great start. On Jun 28, 2010, at 12:52 PM, Alan Runyan wrote: I have similar feelings as Matthias. My thoughts: - Have a canonical namespace which denotes quality, proven, and widely used packages. Packages should never be in this namespace unless the package can meet some quality level. The namespace could be, zodb. - We could use something similar to the Apache incubation process. - We should come up with a list of criteria which we want this complete package to achieve. I believe everyone feels that the Standalone ZODB package is simply not enough to build a sophisticated Real World application. So we need to outline what are the min. features of such an aggregation. This is a huge project. So I completely understand the hesitation to invest into something so daunting. Maybe the beginning is to blog and talk about the packages that DO work and get confirmation that the authors are still interested in maintaining these packages. I'm sure the ZODB documentation project (http://zodbdocs.blogspot.com/) would gladly accept a contribution towards clarifying what current best practice is for catalogs in ZODB. Yes. If you are interested in sharing what packages you are using in production. Let me know. I will give you access and you can post blog entries on zodbdocs. No one has taken me up on the offer. I would really like to see others contribute with their ZODB experiences. Preferably talking about direct experiences with packaging instead of talking about the LACK of packaging on the blog would be good ;) cheers alan ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Leonardo Santagada santagada at gmail.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
On Mon, Jun 28, 2010 at 11:52 AM, Alan Runyan runy...@gmail.com wrote: ... - Have a canonical namespace which denotes quality, proven, and widely used packages. Packages should never be in this namespace unless the package can meet some quality level. Mixing meta data into names is a bad idea. You don't want to change the name of something just because some of its meta data changes. The namespace could be, zodb. No it can't. ZODB isn't a namespace package and can't be one without breaking backward compatibility. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Mixing meta data into names is a bad idea. You don't want to change the name of something just because some of its meta data changes. Your right. It's just that sometimes people may have a package name that is not agreeable to the broader community. The namespace could be, zodb. No it can't. ZODB isn't a namespace package and can't be one without breaking backward compatibility. Should it be a namespace package over the long term? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Am 28.06.2010, 16:52 Uhr, schrieb Laurence Rowe l...@lrowe.co.uk: So why don't we all work on the same packages? The main reason is one of legacy. Plone is built on Zope2 and ZCatalog. It works, but it is not without it's issues - we can't have queries that join from that catalog to a zc.relation catalog. Standalone ZCatalog failed because it came to early - Zope2 was only recently eggified, so to be successful the standalone ZCatalog would need to be used in Zope2. Nobody has bothered with this because non-legacy code shouldn't be using ZCatalog anyway - there are newer and better ways of doing it. Oh, nice to know. I was already writing test cases for standalone ZCatalog integration in my project as all other indices seemed tied to plone :) -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
On 06/28/2010 04:44 AM, Nitro wrote: I do not like the current packaging approach as I discussed in length in a recent message to this list [1]. Until this changes I am not going to spend time creating random packages. Around here, Python packages are how we communicate code. Think of a released package as a well-refined email. If you choose not to release packages, you are really choosing not to communicate through the established channels, so many people will not hear you. If you want the largest possible audience to hear you, you should release packages, just like you should use version control and bug trackers. The packaging system works a lot better than you think it does. You owe it to yourself to learn Buildout, which solves the package versioning problem you ran into. In my experience, the ecosystem of packages we have now is actually a good deal better than the Python standard library. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Am 28.06.2010, 17:52 Uhr, schrieb Alan Runyan runy...@gmail.com: I have similar feelings as Matthias. - We could use something similar to the Apache incubation process. Yes. I think in general the library should be rather flexible. I.e. (minor) things can be deprecated and removed rather fast. I'd also like to be able to make non-backwards compatible releases. We don't want to support legacy apis and code forever. People relying on the old behaviour are forced to upgrade their code or they'll have to stick with the old version. This means we can focus less on maintaining compatibility and more on the real issues. What are your takes on this? Do you prefer a more stable approach? - We should come up with a list of criteria which we want this complete package to achieve. I believe everyone feels that the Standalone ZODB package is simply not enough to build a sophisticated Real World application. So we need to outline what are the min. features of such an aggregation. Yes. We should probably should start by naming a few core things which most applications will need. Here are a few that popped in my head: - indexing (full-text, fields, keywords, spatial, ...). - querying (auto-generate indices, ...) - maintenance (packing, error recovery, zeo administration, btrees validity checking, ...) - gui browser to inspect database - profiling tools (finding hotspots, visualizing problematic parts of your application, find objects consuming most space, ...) - standard way to hook invalidations/registrations to get MVC-like functionality That's already a lot. And probably still a lot missing. We might restrict us to less than this. If there are many contributors, we can certainly extend the list. This is a huge project. So I completely understand the hesitation to invest into something so daunting. Maybe the beginning is to blog and talk about the packages that DO work and get confirmation that the authors are still interested in maintaining these packages. Yes, see my other post. I'll start with gathering a list of all packages and useful snippets which I can find on the web. I'll do a rough categorization and then send it here so you guys can give input on it (e.g. by adding your own experience). I'm sure the ZODB documentation project (http://zodbdocs.blogspot.com/) would gladly accept a contribution towards clarifying what current best practice is for catalogs in ZODB. Yes. If you are interested in sharing what packages you are using in production. Let me know. I will give you access and you can post blog entries on zodbdocs. No one has taken me up on the offer. I would really like to see others contribute with their ZODB experiences. I don't have any money to donate to the doc project, but I can provide a small article here and there. E.g. I can write about my experience with transactions and data managers/commit hooks/synchronizers. -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Am 28.06.2010, 18:23 Uhr, schrieb Leonardo Santagada santag...@gmail.com: Although I think this sounds like an optimal solution, I think it isn't necessary. I wish only that there was a complements in ZODB wiki with a simple list of best of breed package for each task and maybe a list of alternatives. To make it even better a good deprecated section would be good so you don't need to look at old packages trying to figure out if they are the new thing that is not on the wiki or the so-old-that-no-one-uses thing. This would be a great start. I will start by gathering a list of the packages and categorizing them roughly (by scope and last release date). We'll choose the good packages from them and contact the maintainers on important/interesting packages to ask for their support. The wiki is not the right place for this, it should probably go straight into a toplevel menu point of the zodb webpage. -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
On Mon, Jun 28, 2010 at 2:30 PM, Alan Runyan runy...@gmail.com wrote: ... No it can't. ZODB isn't a namespace package and can't be one without breaking backward compatibility. Should it be a namespace package over the long term? Not IMO. That's partly because I think ZODB should be small. I also don't think that just because something adds value to an underlying framework, it should be added to a framework namespace package. Jim -- Jim Fulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Am 28.06.2010, 20:31 Uhr, schrieb Shane Hathaway sh...@hathawaymix.org: Around here, Python packages are how we communicate code. Think of a released package as a well-refined email. If you choose not to release packages, you are really choosing not to communicate through the established channels, so many people will not hear you. If you want the largest possible audience to hear you, you should release packages, just like you should use version control and bug trackers. The communication failed for me. I don't want to be communicated code. That's a technical detail I am not interested in. What I want to be communicated is which package is important? which package is reliable? what's the standard solution?. The packaging system does not deal with this level of information. It allows to download and install files in a complicated way, that's all. The best I get is a score on PyPI. So I've heard all the packages and their authors on why their package is great. Just imagine what will happen if the moderate package count of 50 zodb packages raises to 250. Do you want to listen to each one one of these communication packages before actually getting started on your real work? Whereas a communication channel like this message list is much more effective for this thing. E.g. Laurence told me Standalone ZCatalog is basically a left over from older days. I can get a vague impression of this of the last release date on the package page, but I still don't know about the new or the standard solution. Just give me the bread, don't make me figure out which flour there is and which kind of flour I need. A standard library signals This is the preferred way to do it. Look here if you need reliable stuff. This is the standard solution.. I'll trust it and take it. If I want to contribute something I'll contribute it to the standard library as I know my efforts are not in vain. My code is compatible with other people's code using the standard library. E.g. the C++ STL made communication between coders much easier, finally everybody is using std::vector and friends so I can understand code at a glance and don't have to delve into the peculiarities of how everybody rolls his own vector, list or array class. This also increased interoperability between libraries. Btw, I use different vcs, bug trackers and also package my things. I just highly prefer staying away from any package system. Give me a single file including everything, I'll install it and it will just work. Make me resolve all the conflicts and dependency issues of a .rpm file and I might get cranky :-) The packaging system works a lot better than you think it does. You owe it to yourself to learn Buildout, which solves the package versioning problem you ran into. In my experience, the ecosystem of packages we have now is actually a good deal better than the Python standard library. I did not run into a package versioning problem. Different packages were outright broken. The .eggs were missing some meta-files and installed dlls into the wrong places on windows. The ratio of broken packages I encountered is probably around 10%. But these 10% take up 90% of my time. A lot of these problems could have been detected and resolved sooner if the packages had more users. The more users, the less serious problems. You get more users per package by not spreading users over similar packages. The current healthy ecosystem is neither healthy nor a system. It's a pile of random packages with no semantic meaning. Some good in there, some crap. Very hard for me to find my way through it. Not organized by criteria I am interested in. I do not gain a lot from buildout. Almost all recipes are geared towards plone/zope/web development. Neither of which I am involved in. In fact I have my own build system based on SCons which also use to build C++ libraries, docs and more. ZODB is a general python object database with a much wider audience than just plone. It suits desktop applications just as well as applications you'd normally use twisted and pickle for. Forcing all those zope dependencies like buildout on people does not add to the attractiveness of ZODB for users outside zope. Having indices only in plone does also not make sense. Many applications would benefit from keyword, field, full-text, spatial, younameit indices. Yet extracting individual packages from zope/plone is impossible due to the slew of dependencies. While I can accept a dependency like zope.interface I don't accept a lot of the others. It really prevents ZODB from living up to its full potential in non-plone applications. -Matthias ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
On 28 June 2010 19:31, Nitro ni...@dr-code.org wrote: Am 28.06.2010, 16:52 Uhr, schrieb Laurence Rowe l...@lrowe.co.uk: So why don't we all work on the same packages? The main reason is one of legacy. Plone is built on Zope2 and ZCatalog. It works, but it is not without it's issues - we can't have queries that join from that catalog to a zc.relation catalog. Standalone ZCatalog failed because it came to early - Zope2 was only recently eggified, so to be successful the standalone ZCatalog would need to be used in Zope2. Nobody has bothered with this because non-legacy code shouldn't be using ZCatalog anyway - there are newer and better ways of doing it. Oh, nice to know. I was already writing test cases for standalone ZCatalog integration in my project as all other indices seemed tied to plone :) In general, if it's not on PyPI it doesn't exist as far as the Zope world is concerned. (I can't find any references to standalone ZCatalog after 2005.) Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Matthias, You are conflating two conversations, and as a result you are reacting to the non-stated suggestion that the packaging system provides sufficient stability information to consumers. That suggestion would obviously be false, since anyone, including people with hostile intentions, can create packages. Let's separate the conversations for clarity: 1) Yes, there is a lack of ZODB documentation. Please feel free to contribute by writing documentation, especially for the upcoming book. However, keep in mind that there is no need to throw out packaging in order to document ZODB and third party functionality. Other people rely heavily on packaging and have much more success with it than you apparently do. 2) You seem to be interested in contributing a spatial index. Great, that sounds interesting! You don't have to create a package, but that is the best thing to do to gain an audience. I do not gain a lot from buildout. Almost all recipes are geared towards plone/zope/web development. Neither of which I am involved in. In fact I have my own build system based on SCons which also use to build C++ libraries, docs and more. While Buildout is suggested, it's certainly not required. A simple alternative to Buildout is to create your own mini-PyPI, which is amazingly easy. To do it, simply put the packages you want in a directory (or make them downloadable from a web page), then tell easy_install to use your index instead of PyPI. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
On 28 June 2010 21:27, Nitro ni...@dr-code.org wrote: ZODB is a general python object database with a much wider audience than just plone. It suits desktop applications just as well as applications you'd normally use twisted and pickle for. Forcing all those zope dependencies like buildout on people does not add to the attractiveness of ZODB for users outside zope. Having indices only in plone does also not make sense. Many applications would benefit from keyword, field, full-text, spatial, younameit indices. Yet extracting individual packages from zope/plone is impossible due to the slew of dependencies. While I can accept a dependency like zope.interface I don't accept a lot of the others. It really prevents ZODB from living up to its full potential in non-plone applications. Remember that Plone is an eight year old application that is built on top of a 12 year old Application server. There has been much progress since then (and plenty of people who build non-Plone ZODB based applications), but the size of the codebase means it is not possible to always be using the current best practice. http://zope2.zope.org/about-zope-2/the-history-of-zope Nobody would recommend that you try to extract stuff from Plone or Zope2. In my opinion there are two main sources of packages for non-Zope2 dependent applications. * The ZTK extracted the core of Zope 3 and is used in application servers such as Grok and BlueBream. It contains zope.catalog and it's related packages. There are several extensions on top of this such as zc.catalog and hurry.query. The 1.0 release has not been there yet, but the underlying packages are stable. Installing zcatalog requires a total of 34 packages (ZODB3 requires 10) http://docs.zope.org/zopetoolkit/releases/packages-trunk.html * The Repoze project has focussed on making zope technologies more easily accessible to applications outside of Zope. Whilst the ZTK project has improved things a lot, it is still a relatively large chunk to swallow whole. repoze.catalog is extracted from zope.catalog and requires only zope.index in addition to ZODB3. http://docs.repoze.org/catalog/ At the very lowest level are the indexes themselves such as zope.index and zc.relation, a spatial index would fit in here too. (Health warning: I'm mostly a Plone developer, so do not yet have experience using these packages) Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SpatialIndex
Am 28.06.2010, 23:30 Uhr, schrieb Shane Hathaway sh...@hathawaymix.org: Matthias, You are conflating two conversations, and as a result you are reacting to the non-stated suggestion that the packaging system provides sufficient stability information to consumers. I've experienced the packaging system is not useful to me without meta information like stability, widespread use, interoperability etc.. Yet people keep telling me people can use your code if you package it. This is highly contradictory to me. 1) Yes, there is a lack of ZODB documentation. Please feel free to contribute by writing documentation, especially for the upcoming book. However, keep in mind that there is no need to throw out packaging in order to document ZODB and third party functionality. Other people rely heavily on packaging and have much more success with it than you apparently do. I do not want to focus on the technically broken packages. There are enough which work. Packaging as a means to make code downloadable and installable works. The problem about documentation and packaging is this: I want to build an app. I go to zodb.org. I don't find lots of information how to query or index my database. If I was the casual observer, I'd leave the site right now and never come back. I'm not interested in the raw core skeleton of a database. Losing a user = losing a possible future contributor = losing a possible future customer (for those people who sell services around zodb/zope/plone). Ideally it goes like this: I want to build an app. I go to zodb.org. I get to know zodb has built in tools for most of the problems I'm likely to face. Great. Let's hit the download button. After installation I want to see/test a few of those features together. Then I read up on them in detail. Look at the wxPython package for a killer demo application which serves the purpose of pulling all the different widgets together for an end-user. I do not have to google all over the place to find a decent calendar or list widget. So we could now go and collect a list of good packages and publish it. We could even write a tutorial which shows a use case involving three or so of these packages. We can publish it in a book or on the zodb main page. We can write a demo showcase application involving indexing and querying packages. In a few months or a year this won't work anymore, because each of the three packages will have gone down its own way. The package maintainers won't care about fixing the tutorial or application, they are already over-burdened with the work of keeping documentation of the package itself up-to-date. In two years one of the packages will be superseded. Most developers completely forgot there is a tutorial showing how to put three good modules together. That's the way current ZODB documentation and packaging work in my view. The once very useful bigger picture information - these three are good packages, you can use them together like this - is lost again. This information is what made ZODB useful and attractive to me in the first place. The lack of collaboration between the packages and core zodb destroyed useful documentation. Packages pull changes from core zodb, but they don't push anything back. With a standard library structure this cannot happen. I want to emphasize a standard library can pull already existing packages for a start! There can be tests which enforce the interoperability between packages for example. This way the zodb ecosytem remains useable as a whole. It cannot diverge to the point where individual packages get meaningless. The whole thing is more than the sum of its parts. In this spirit I think packaging and documentation are conflated. Packaging does not have to be thrown out at all. But packaging has to be used in a way so the individual components create a whole to which I can assign semantic value. Semantics can then be documented. Right now it's all splittered from the perspective of an end-user. 2) You seem to be interested in contributing a spatial index. Great, that sounds interesting! You don't have to create a package, but that is the best thing to do to gain an audience. I'm not interested in an audience for the package. I'm glad if the index works for me. I can write unit tests to verify this. I'm happy if it's useful to others and want to give something back to the community so new user don't have to face the same issues I've faced. As of right now it I can't contribute in a way which would spare a new user the same issues. And I have a hard time believing I am the only one who's facing them. Other bigger python frameworks like twisted or wxPython never made me feel this way. I'm interested in creating a zodb universe which is useable as a whole right from the start and where I don't have to find and pickup small pieces to glue them together in tedious
Re: [ZODB-Dev] SpatialIndex
On 06/28/2010 07:29 PM, Nitro wrote: P.S.: This will be my last lengthy mail about this topic. I'll rather spend time writing blog entries for the docs project. That sounds fine. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
