Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-29 Thread Erik Enge

On Thu, 28 Jun 2001, Stephan Richter wrote:

 I think we need to setup a separate mailing list for this project. Can
 you do that? I think the discussion is getting too specific now.

We can use the mk-zprod-devel mailinglist at thingamy.com:

URL:http://www.thingamy.com/mailman/listinfo/mk-zprod-devel/


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-29 Thread Erik Enge

On Thu, 28 Jun 2001, Stephan Richter wrote:

 All what the wizard will do is provide the user with a nice
 interface/GUI to enter the information. I will then pass the
 information to the above mentioned management methods and the
 MakeZProduct Product will then generate the code and save it to the
 file system.

Oki.  What I currently have is a Zope product that you instanciate (maybe
in the context of the wizard) and that instance then provides you with
those methods you wanted.  The instance is the only thing persisten, since
all else will be written out to files (it can quite easily be done to
write them as persistent DTML Documents or somesuch if one wanted to
pause the setup or something, but that's outside the scope of round 1).
 
 BTW, we will assume that the user that runs the Zope engine has write
 access to /ZOPE/lib/python/Products.

Yeah.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-29 Thread Stephan Richter

This will be the last post from on this thread.
We moved all discussions to
[EMAIL PROTECTED]
Sign up at:
http://www.thingamy.com/mailman/listinfo/mk-zprod-devel/
Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-29 Thread Giovanni Maruzzelli

The Zope version we use contains the new btree catalog by default.

So, when we recreated the catalog from scratch, it was created as a btree
catalog.

The traces that you saw comes from the new catalog (the btree one).

-giovanni
- Original Message -
From: Chris Withers [EMAIL PROTECTED]
To: Giovanni Maruzzelli [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; Chris McDonough
[EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Thursday, June 28, 2001 6:27 PM
Subject: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a
solution?)


 Giovanni Maruzzelli wrote:
 
  The catalog is a pristine 2.3.3b1 catalog.

 I'm sure that'll need upgrading then...

  We have recreated the catalog from scratch because we tried
  manage_convertBTrees , but it don't work for us, it return with an error
  (and the same happens with 2.3.3 final):
 
  Error Type: TypeError
  Error Value: second argument must be a class

 Weird... from your earlier posting it looked like you _had_ successfully
 upgraded and updated (BTrees.IOBTree in your traceback rather than
 IOBTree.IOBTree)

 cheers,

 Chris


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Tino Wildenhain

Hi,

--On Mittwoch, 27. Juni 2001 15:54 -0500 Stephan Richter [EMAIL PROTECTED] 
wrote:

 At 10:45 PM 6/27/01 +0200, Erik Enge wrote:
 On Wed, 27 Jun 2001, Tino Wildenhain wrote:

  if there are always many objects to create, may be it would be better
  to have a generic mechanism for asking users and represent
  app-/management interfaces rather then copying all the stuff over and
  over?

 That's what mk-zprod does.  Or rather, will do once I've made the
 interface friendlier.  (If I didn't misunderstand you.)

 If I have some time left at all tonight, I will make a wizard that could
 be  the initial front-end to mk-zprod. Could you give me a short list of
 things  you would like to ask the user?

Hm. a simple collection of questions? Certainly not.
I talking of a whole mechanism, where you group input
and output into contexts. From application view it
would be an API, it schould work no matter if the input/
output is generated from and to HTML, WML, XML or even
stand-allone GUIs.
I'm thinking of semantic groups of input, output and types
and ranges and default values including naming conventions.

Regards
Tino 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Erik Enge

On Thu, 28 Jun 2001, Tino Wildenhain wrote:

 Hm. a simple collection of questions? Certainly not. I talking of a
 whole mechanism, where you group input and output into contexts. From
 application view it would be an API, it schould work no matter if the
 input/ output is generated from and to HTML, WML, XML or even
 stand-allone GUIs.

Could you give me an example of how this would work?


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Stephan Richter


If I have some time left at all tonight, I will make a wizard that could
be  the initial front-end to mk-zprod. Could you give me a short list of
things  you would like to ask the user?

Hm. a simple collection of questions? Certainly not.
I talking of a whole mechanism, where you group input
and output into contexts. From application view it
would be an API, it schould work no matter if the input/
output is generated from and to HTML, WML, XML or even
stand-allone GUIs.
I'm thinking of semantic groups of input, output and types
and ranges and default values including naming conventions.

Well, that sounds more like what SmartObjects tries to do. But I am still 
confused. As Erik wrote later, can you give examples?

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Stephan Richter

Here my list of questions:

1. Will we create ZClass or Python Products?

Product-name

2. I think it should ask some more meta-data information here, such as 
License, Author(s), Description 

A list of images (like dtmldoc.gif and such)

3. Okay, here we should have a loop of enter images with upload 
functionality. The only problem is where do we save them in the mean time?

If you want to set a default Catalog name

4. Okay, I do not work with the catalog; what do you need, just a name or 
the path? I think the WizardTreePage will help here.

If you want nice redirection after adding objects

5. Yep, how could that look like; should I provide users with a text area 
or what?

if you want to include standard headers in your dtml files

Okay.

a list of classes

6. Yes, we will start a loop here asking for class information.

  class name
  class type
  class icon name

7. Yeah, this is all the meta-data here, so that could go on one page. But 
you are missing a list of Base Classes. I have no clue where to get these 
from. I guess it is time to look into the ZClass code, but I think they use 
a separate Mechanism. The problem with the current product registry is that 
NO classes are listed anywhere, only their constructors, which is nothing 
worth in this case.

  catalogaware or not (or may just methods to index/reindex th object)

Okay.

   a list of attributes
 attribute name
 attribute type
 attribute default value

8. Should that just use the PropertyManager attribute _properties? I 
usually add the 'default' key to each property dictionary and write a two 
liner in the __init__ method to generate the properties with their default...

   a list of subclasses
 subclass name
 subclass type
 subclass icon name
 a list of attributes
   

9. Again, that would be a loop. But my question here would be: Are you 
using sub-classes that much. I have written one so far in all the products ...

Okay, I think we got a flow together... If you have a couple more minutes 
grin, some sort of storyboards would be nice... I started already 
building a wizard, but I dunno how much time I got today, since I am 
leaving town tomorrow and won't return until the ZopeCon in Berlin is over. 
Are coming to Berlin?

Regards,
Stephan


--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Erik Enge

On Thu, 28 Jun 2001, Stephan Richter wrote:

 Here my list of questions:
 
 1. Will we create ZClass or Python Products?

Python Products.
 
 Product-name
 
 2. I think it should ask some more meta-data information here, such as 
 License, Author(s), Description 

Yepp :)
 
 A list of images (like dtmldoc.gif and such)
 
 3. Okay, here we should have a loop of enter images with upload 
 functionality. The only problem is where do we save them in the mean time?

Upload?  Upload to where?

mk-zprod (as it works today) creates a directory on your filesystem, and
in that directory (which you just copy to your Products directory) it
creates an img/ directory and puts them there.
 
Or are you asking where the wizard should store them?  (Which I thought it
didn't have to.)

 If you want to set a default Catalog name
 
 4. Okay, I do not work with the catalog; what do you need, just a name or 
 the path? I think the WizardTreePage will help here.

Just the name. :)
 
 If you want nice redirection after adding objects
 
 5. Yep, how could that look like; should I provide users with a text area 
 or what?

In the dtml/ directory of mk-zprod you have Add2.dtml, that's the page the
user is redirected to after adding an object.  Maybe a text area with a
suggestion (like Add2.dtml) in it?
 
 7. Yeah, this is all the meta-data here, so that could go on one page.
 But you are missing a list of Base Classes. I have no clue where to
 get these from. I guess it is time to look into the ZClass code, but I
 think they use a separate Mechanism. The problem with the current
 product registry is that NO classes are listed anywhere, only their
 constructors, which is nothing worth in this case.

Hubbliski.  We could provide them with standard classes as defaults
(Persistent, Explicit/Implicit, AccessControl) and suggest some
others.  I'm not too sure about the ZClass thing.

 8. Should that just use the PropertyManager attribute _properties?

Yepp.

 I usually add the 'default' key to each property dictionary and write
 a two liner in the __init__ method to generate the properties with
 their default...

Why not just have:

def __init__(self, attrib1, attrib2='something'):
doc string
self.attrib1 = attrib1
self.attrib2 = attrib2

which could come from such a description:

attribs:
name: attrib1 default value: None
name: attrib2 default value: something
 
What do you think?

 9. Again, that would be a loop. But my question here would be: Are you
 using sub-classes that much. I have written one so far in all the
 products ...

My latest projects sub-classes left, right and center.  If we just have
the logic for a recursive loop there the user could add as many as s/he
wanted without it being much more work for us.
 
 Are coming to Berlin?

Well, I was, but I'm moving to France the 8. of July and I've just come
home (to Norway) from England.  England, Norway, France, Germany, France
all in just three weeks is a bit much :)  (Oh, and I have to get an
apartment in Antibes.)


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Erik Enge

On Thu, 28 Jun 2001, Erik Enge wrote:

   name: attrib1 default value: None
   name: attrib2 default value: something

No, that wouldn't work, because you might want None as the default for
that attribute.  Maybe Nothing, or somesuch then.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Stephan Richter

At 02:23 PM 6/28/01 +0200, Erik Enge wrote:
On Thu, 28 Jun 2001, Erik Enge wrote:

name: attrib1 default value: None
name: attrib2 default value: something

No, that wouldn't work, because you might want None as the default for
that attribute.  Maybe Nothing, or somesuch then.

None == Nothing. Some default we need to have, otherwise we cannot 
initialize the attribute.

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Stephan Richter


Python Products.

Ok.

  3. Okay, here we should have a loop of enter images with upload
  functionality. The only problem is where do we save them in the mean time?

Upload?  Upload to where?

mk-zprod (as it works today) creates a directory on your filesystem, and
in that directory (which you just copy to your Products directory) it
creates an img/ directory and puts them there.

Well, but where are the images coming from. The Wizard is a Web Frontend, 
so we would need to upload the images or copy them from somewhere else from 
the file system.
BTW, the /img directory is non-standard. Better would be www/ since other 
products use it as well.
In fact, we only need to think about adding them with the right permissions 
to the directory. A piece of code in the __init__.py will do the rest:

# load all the icons automatically
from Globals import package_home, ImageFile
misc_ = {}
path = os.path.join(package_home(globals()), 'www/')

for file in os.listdir(path):
 if file[-3:] == 'gif':
 misc_[file[:-4]] = ImageFile('www/%s' %file, globals())


Or are you asking where the wizard should store them?  (Which I thought it
didn't have to.)

Well, you can take two approaches here:
1. As soon as the user uploads the icon, we are adding it to the directory. 
This is a problem, if you want to undo some or all of the wizard actions, 
but is much easier to implement.
2. When the icon is uploaded, save it somewhere temporarily. At the end, 
when all questioning is done, add everything at once. This has the 
advantage that you can freely move back and forth until you finish the wizard.

  5. Yep, how could that look like; should I provide users with a text area
  or what?

In the dtml/ directory of mk-zprod you have Add2.dtml, that's the page the
user is redirected to after adding an object.  Maybe a text area with a
suggestion (like Add2.dtml) in it?

Sounds good.

Hubbliski.  We could provide them with standard classes as defaults
(Persistent, Explicit/Implicit, AccessControl) and suggest some
others.  I'm not too sure about the ZClass thing.

Okay, for the first version that will probably suffice.

  I usually add the 'default' key to each property dictionary and write
  a two liner in the __init__ method to generate the properties with
  their default...

Why not just have:

 def __init__(self, attrib1, attrib2='something'):
 doc string
 self.attrib1 = attrib1
 self.attrib2 = attrib2

Because this is horribly ugly and unflexible when you have 20 attributes. I 
always have to change several places, if I modify, add or edit an 
attribute/property.

which could come from such a description:

 attribs:
 name: attrib1 default value: None
 name: attrib2 default value: something

Well, we can describe that right away in the standard property dictionary 
form; then we do not need to parse or do anything else.

  9. Again, that would be a loop. But my question here would be: Are you
  using sub-classes that much. I have written one so far in all the
  products ...

My latest projects sub-classes left, right and center.  If we just have
the logic for a recursive loop there the user could add as many as s/he
wanted without it being much more work for us.

Okay, I just wanted to make sure it really has the additional usage, for 
the extra effort.

Well, I was, but I'm moving to France the 8. of July and I've just come
home (to Norway) from England.  England, Norway, France, Germany, France
all in just three weeks is a bit much :)  (Oh, and I have to get an
apartment in Antibes.)

Okay. It would have been a good place to discuss this further.

Now, I looked at the mk-zprod code again. We need to make it a real Zope 
Product with a nice management API, so that I can use the functions from 
the Wizard.

For example we would need:

manage_make_initializeProduct(...)
manage_make_addClass(...)
manage_make_addProperty(...)
manage_make_addIcon(...)
manage_make_addSubClass(mainClass, ...)

If we define this API clearly, then we can use it later to automatically 
generate ZClasses as well. Once the work, twice the reward! ;-)

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Erik Enge

On Thu, 28 Jun 2001, Stephan Richter wrote:

 Well, but where are the images coming from. The Wizard is a Web
 Frontend, so we would need to upload the images or copy them from
 somewhere else from the file system.

From the filesystem of the user?

 BTW, the /img directory is non-standard. Better would be www/ since other 
 products use it as well.

Sure thing.  (Although, www/ indicates to me .html pages and various other
things too.)

 In fact, we only need to think about adding them with the right permissions 
 to the directory. A piece of code in the __init__.py will do the rest:

Yes.  This is what mk-zprod does today :)
 
 2. When the icon is uploaded, save it somewhere temporarily. At the end, 
 when all questioning is done, add everything at once. This has the 
 advantage that you can freely move back and forth until you finish the wizard.

Sounds good :)
 
  def __init__(self, attrib1, attrib2='something'):
  doc string
  self.attrib1 = attrib1
  self.attrib2 = attrib2
 
 Because this is horribly ugly and unflexible when you have 20 attributes. I 
 always have to change several places, if I modify, add or edit an 
 attribute/property.

Could you give me an example of how you would do it?  (Then I could
modify the mk-zprod code.)
 
 Well, we can describe that right away in the standard property dictionary 
 form; then we do not need to parse or do anything else.

Nice :)
 
 Well, I was, but I'm moving to France the 8. of July and I've just come
 home (to Norway) from England.  England, Norway, France, Germany, France
 all in just three weeks is a bit much :)  (Oh, and I have to get an
 apartment in Antibes.)
 
 Okay. It would have been a good place to discuss this further.

Morten Petersen from Thingamy will be there, I'm sure he has some thoughts
about it.
 
 Now, I looked at the mk-zprod code again. We need to make it a real
 Zope Product with a nice management API, so that I can use the
 functions from the Wizard.

I'll start typing right now.  I'll probably have something premature by
tomorrow.
 
 If we define this API clearly,

Ok.  I'll just write up mk-zprod as a Python Product and the API is
defined as I go.  Then, you could have a look and we can unite on one API
and I make the necessary changes in the code.  Yeah?

Unless, of course, you have some specific requests right away.

 Once the work, twice the reward! ;-)

I like.



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Erik Enge

On Thu, 28 Jun 2001, Stephan Richter wrote:

 For example we would need:
 
 manage_make_initializeProduct(...)
 manage_make_addClass(...)
 manage_make_addProperty(...)
 manage_make_addIcon(...)
 manage_make_addSubClass(mainClass, ...)

I just realised something.  How do you need this to work?  With regards to
the wizard.  Should mk-zprod produce the files and add them to a folder in
itself?  Or should it return everything for the wizard to handle?


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-28 Thread Chris Withers

Giovanni Maruzzelli wrote:
 
 The catalog is a pristine 2.3.3b1 catalog.

I'm sure that'll need upgrading then...

 We have recreated the catalog from scratch because we tried
 manage_convertBTrees , but it don't work for us, it return with an error
 (and the same happens with 2.3.3 final):
 
 Error Type: TypeError
 Error Value: second argument must be a class

Weird... from your earlier posting it looked like you _had_ successfully
upgraded and updated (BTrees.IOBTree in your traceback rather than
IOBTree.IOBTree)

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Stephan Richter

Erik,

I think we need to setup a separate mailing list for this project. Can you 
do that? I think the discussion is getting too specific now.

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Stephan Richter


I just realised something.  How do you need this to work?  With regards to
the wizard.  Should mk-zprod produce the files and add them to a folder in
itself?  Or should it return everything for the wizard to handle?

All what the wizard will do is provide the user with a nice interface/GUI 
to enter the information. I will then pass the information to the above 
mentioned management methods and the MakeZProduct Product will then 
generate the code and save it to the file system.

BTW, we will assume that the user that runs the Zope engine has write 
access to /ZOPE/lib/python/Products.

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-28 Thread Stephan Richter


  BTW, we will assume that the user that runs the Zope engine has write
  access to /ZOPE/lib/python/Products.
 

Don't forget about those of us that use $INSTANCE_HOME/Products

Oops, that what I meant to say.  have access to the Products directory 
 :-)

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-27 Thread Erik Enge

On Tue, 26 Jun 2001, Stephan Richter wrote:

 Okay, okay...I stayed up and typed it down pretty quick (2 hours). I
 attached it to this mail. It is plain text, since I was too lazy to do
 it in HTML. It might be a little unstructured, but I am too tired to
 fix that now.

So what does it do?  :)

What I'm thinking is this: maybe use SmartWizard to meta-program you
Python Product; that creates a definition file(s) of some sort which is
sent to mk-zprod; mk-zprod consults the WarpFramework do make sure we
don't create too much work for ourselves, and also provides us with nice
default HTML/DTML-pages, and finally, Formulator could be used to do
validation on those HTML/DTML-pages.

Or are we talking past eachother here?

It would be very cool to have a tool like that.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-27 Thread Stephan Richter

At 06:05 PM 6/27/01 +0200, Erik Enge wrote:
On Wed, 27 Jun 2001, Stephan Richter wrote:

  Exactly that. But the SmartWizard would provide you with a framework
  to build this Make New Python Product Wizard. If I get far enough, I
  will release the pre alpha today, just you see the proof of concept...

Cool!  I'll be looking forward to it.

I have already a decent version running now. I want to add the Formulator 
and SmartSections WizardPage Templates to the system and make it better 
looking before the release tonight (or more in the morning).

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-27 Thread Tino Wildenhain

Hm. Wizard?

if there are always many objects to create, may be it would be
better to have a generic mechanism for asking users and
represent app-/management interfaces rather then copying all
the stuff over and over?

Regards
Tino

--On Mittwoch, 27. Juni 2001 08:54 -0500 Stephan Richter [EMAIL PROTECTED] 
wrote:


 So what does it do?  :)

 It is a general Wizard Builder with which you could build a Wizard that
 asks for all the necessary information to auto-generate a Python Product
 (for example).

 What I'm thinking is this: maybe use SmartWizard to meta-program you
 Python Product; that creates a definition file(s) of some sort which is
 sent to mk-zprod; mk-zprod consults the WarpFramework do make sure we
 don't create too much work for ourselves, and also provides us with nice
 default HTML/DTML-pages, and finally, Formulator could be used to do
 validation on those HTML/DTML-pages.

 Exactly that. But the SmartWizard would provide you with a framework to
 build this Make New Python Product Wizard. If I get far enough, I will
 release the pre alpha today, just you see the proof of concept...

 Or are we talking past eachother here?

 Nope, we don't. But SmartWizard is a more general tool than you were
 thinking of it.

 It would be very cool to have a tool like that.

 I agree. It is annoying to code all that stuff all the time.

 Regards,
 Stephan

 --
 Stephan Richter
 CBU - Physics and Chemistry Student
 Web2k - Web Design/Development  Technical Project Management


 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-27 Thread Erik Enge

On Wed, 27 Jun 2001, Tino Wildenhain wrote:

 if there are always many objects to create, may be it would be better
 to have a generic mechanism for asking users and represent
 app-/management interfaces rather then copying all the stuff over and
 over?

That's what mk-zprod does.  Or rather, will do once I've made the
interface friendlier.  (If I didn't misunderstand you.)


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-27 Thread Stephan Richter


if there are always many objects to create, may be it would be
better to have a generic mechanism for asking users and
represent app-/management interfaces rather then copying all
the stuff over and over?

Well, the current wizard version (which I hope I will be able to release in 
a couple more hours) already has some Page Templates that do some of the 
common tasks. I think we could easily map this to the management interface.

And I like the idea a lot. So often I am simple copying a DTML File just to 
change a name or so.

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-27 Thread Stephan Richter

At 10:45 PM 6/27/01 +0200, Erik Enge wrote:
On Wed, 27 Jun 2001, Tino Wildenhain wrote:

  if there are always many objects to create, may be it would be better
  to have a generic mechanism for asking users and represent
  app-/management interfaces rather then copying all the stuff over and
  over?

That's what mk-zprod does.  Or rather, will do once I've made the
interface friendlier.  (If I didn't misunderstand you.)

If I have some time left at all tonight, I will make a wizard that could be 
the initial front-end to mk-zprod. Could you give me a short list of things 
you would like to ask the user?

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Giovanni Maruzzelli
]
OID: 387cc len 67 [BTrees.IIBTree.IISet]
OID: 38b29 len 1228 [BTrees.IOBTree.IOBucket]
OID: 38c19 len 904 [BTrees.IOBTree.IOBucket]
OID: 38d37 len 1007 [BTrees.IOBTree.IOBucket]
OID: 3c610 len 33864 [BTrees.IOBTree.IOBucket]



- Original Message -
Sent: Monday, June 25, 2001 6:07 PM
Subject: Re: [Zope-dev] Zcatalog bloat problem (berkeleydb is a solution?)


  A solution might be a kind of lazy catalog awareness: Instead of
  mangling a new object through one or more catalogs when it is created,
  this object could be added to a list of objects to be cataloged later.
  This way, the transaction to insert a new object would become much
  cheaper. I'm working on this, but right now it is quite messy. (I'm
  new to Python and Zope, and hence I'm stumbling over a few, hmmm,
  trip-wires...)

 This purpose aligns well with those of the ArmoredCatalog proposal as
well..
 see http://dev.zope.org/Wikis/DevSite/Proposals/ArmoredCatalog .

  But even using such a lazy catalog awareness, you might get into
  trouble. Using the ZCatalog's find objects function, I hit the limits
  of my Linux box: 640 MB RAM were not enough...

 This should not happen.  :-(

 I'm really disappointed that the bloat and memory consumption issues are
 still plaguing the ZCatalog.  At one point, I really thought we had it
 pretty much licked.  I suppose this was naive.

  A few weeks ago, I've posted this (admittedly not fully cooked) patch to
  this list, but did not get yet any response.

 I apologize for this.  We have a fairly formalized process for handling
 feature-ish collector issues, and this hasn't come round on the guitar.
I'm
 beyond disappointed that people are still having unacceptable bloat,
enough
 that something like this patch needed to be submitted.  It's
disheartening.
 :-(

 - C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Chris McDonough
: 407e0 len 339 [BTrees.IOBTree.IOBucket]
 OID: 40813 len 387 [BTrees.IIBTree.IISet]
 OID: 407e1 len 339 [BTrees.IOBTree.IOBucket]
 OID: 40814 len 362 [BTrees.IIBTree.IISet]
 OID: 407e2 len 507 [BTrees.IOBTree.IOBucket]
 OID: 37eb9 len 981 [BTrees.IOBTree.IOBucket]
 OID: 38197 len 804 [BTrees.OIBTree.OIBucket]
 OID: 38ac7 len 7947 [BTrees.IIBTree.IISet]
 OID: 387f6 len 97 [BTrees.IIBTree.IISet]
 OID: 383f7 len 850 [BTrees.OOBTree.OOBucket]
 OID: 4081a len 47 [BTrees.IIBTree.IISet]
 OID: 38407 len 850 [BTrees.OOBTree.OOBucket]
 OID: 4081b len 47 [BTrees.IIBTree.IISet]
 OID: 388ac len 92 [BTrees.IIBTree.IISet]
 OID: 387d4 len 152 [BTrees.IIBTree.IISet]
 OID: 3868c len 152 [BTrees.IIBTree.IISet]
 OID: 38681 len 142 [BTrees.IIBTree.IISet]
 OID: 388b0 len 72 [BTrees.IIBTree.IISet]
 OID: 384f1 len 52 [BTrees.IIBTree.IISet]
 OID: 37ca6 len 586 [BTrees.IOBTree.IOBucket]
 OID: 4081c len 686 [BTrees.IOBTree.IOBucket]
 OID: 37ab8 len 39336 [BTrees.IOBTree.IOBTree]
 OID: 381d8 len 594 [BTrees.OIBTree.OIBucket]
 OID: 38ac9 len 1252 [BTrees.IIBTree.IISet]
 OID: 38770 len 52 [BTrees.IIBTree.IISet]
 OID: 37d94 len 1234 [BTrees.IOBTree.IOBucket]
 OID: 3821d len 617 [BTrees.OIBTree.OIBucket]
 OID: 38acb len 557 [BTrees.IIBTree.IISet]
 OID: 38710 len 52 [BTrees.IIBTree.IISet]
 OID: 386ac len 52 [BTrees.IIBTree.IISet]
 OID: 38409 len 1019 [BTrees.OOBTree.OOBucket]
 OID: 4081d len 47 [BTrees.IIBTree.IISet]
 OID: 3870b len 52 [BTrees.IIBTree.IISet]
 OID: 38403 len 816 [BTrees.OOBTree.OOBucket]
 OID: 4081e len 47 [BTrees.IIBTree.IISet]
 OID: 387fe len 57 [BTrees.IIBTree.IISet]
 OID: 387cc len 67 [BTrees.IIBTree.IISet]
 OID: 38b29 len 1228 [BTrees.IOBTree.IOBucket]
 OID: 38c19 len 904 [BTrees.IOBTree.IOBucket]
 OID: 38d37 len 1007 [BTrees.IOBTree.IOBucket]
 OID: 3c610 len 33864 [BTrees.IOBTree.IOBucket]
 
 - Original Message -
 Sent: Monday, June 25, 2001 6:07 PM
 Subject: Re: [Zope-dev] Zcatalog bloat problem (berkeleydb is a solution?)
 
   A solution might be a kind of lazy catalog awareness: Instead of
   mangling a new object through one or more catalogs when it is created,
   this object could be added to a list of objects to be cataloged later.
   This way, the transaction to insert a new object would become much
   cheaper. I'm working on this, but right now it is quite messy. (I'm
   new to Python and Zope, and hence I'm stumbling over a few, hmmm,
   trip-wires...)
 
  This purpose aligns well with those of the ArmoredCatalog proposal as
 well..
  see http://dev.zope.org/Wikis/DevSite/Proposals/ArmoredCatalog .
 
   But even using such a lazy catalog awareness, you might get into
   trouble. Using the ZCatalog's find objects function, I hit the limits
   of my Linux box: 640 MB RAM were not enough...
 
  This should not happen.  :-(
 
  I'm really disappointed that the bloat and memory consumption issues are
  still plaguing the ZCatalog.  At one point, I really thought we had it
  pretty much licked.  I suppose this was naive.
 
   A few weeks ago, I've posted this (admittedly not fully cooked) patch to
   this list, but did not get yet any response.
 
  I apologize for this.  We have a fairly formalized process for handling
  feature-ish collector issues, and this hasn't come round on the guitar.
 I'm
  beyond disappointed that people are still having unacceptable bloat,
 enough
  that something like this patch needed to be submitted.  It's
 disheartening.
  :-(
 
  - C
 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Giovanni Maruzzelli
 [BTrees.IOBTree.IOBucket]
  OID: 4080a len 387 [BTrees.IIBTree.IISet]
  OID: 407d5 len 507 [BTrees.IOBTree.IOBucket]
  OID: 4080b len 387 [BTrees.IIBTree.IISet]
  OID: 407d6 len 507 [BTrees.IOBTree.IOBucket]
  OID: 4080c len 387 [BTrees.IIBTree.IISet]
  OID: 407d7 len 339 [BTrees.IOBTree.IOBucket]
  OID: 4080d len 382 [BTrees.IIBTree.IISet]
  OID: 407d8 len 339 [BTrees.IOBTree.IOBucket]
  OID: 4080e len 382 [BTrees.IIBTree.IISet]
  OID: 407d9 len 339 [BTrees.IOBTree.IOBucket]
  OID: 3d064 len 597 [BTrees.IIBTree.IISet]
  OID: 407da len 347 [BTrees.IOBTree.IOBucket]
  OID: 4080f len 387 [BTrees.IIBTree.IISet]
  OID: 407db len 339 [BTrees.IOBTree.IOBucket]
  OID: 3d1ba len 642 [BTrees.IIBTree.IISet]
  OID: 407dc len 339 [BTrees.IOBTree.IOBucket]
  OID: 40810 len 372 [BTrees.IIBTree.IISet]
  OID: 407dd len 339 [BTrees.IOBTree.IOBucket]
  OID: 40811 len 372 [BTrees.IIBTree.IISet]
  OID: 407de len 339 [BTrees.IOBTree.IOBucket]
  OID: 37f11 len 977 [BTrees.IOBTree.IOBucket]
  OID: 380de len 830 [BTrees.OIBTree.OIBucket]
  OID: 37ac4 len 25537 [BTrees.IIBTree.IISet]
  OID: 37ac7 len 9892 [BTrees.IIBTree.IISet]
  OID: 37aca len 13947 [BTrees.IIBTree.IISet]
  OID: 38922 len 387 [BTrees.IIBTree.IISet]
  OID: 38643 len 827 [BTrees.IIBTree.IISet]
  OID: 3894c len 92 [BTrees.IIBTree.IISet]
  OID: 388ff len 24707 [BTrees.IIBTree.IISet]
  OID: 38581 len 277 [BTrees.IIBTree.IISet]
  OID: 3d7f7 len 319 [BTrees.IOBTree.IOBTree]
  OID: 3d7f8 len 356 [BTrees.IOBTree.IOBTree]
  OID: 40812 len 372 [BTrees.IIBTree.IISet]
  OID: 407e0 len 339 [BTrees.IOBTree.IOBucket]
  OID: 40813 len 387 [BTrees.IIBTree.IISet]
  OID: 407e1 len 339 [BTrees.IOBTree.IOBucket]
  OID: 40814 len 362 [BTrees.IIBTree.IISet]
  OID: 407e2 len 507 [BTrees.IOBTree.IOBucket]
  OID: 37eb9 len 981 [BTrees.IOBTree.IOBucket]
  OID: 38197 len 804 [BTrees.OIBTree.OIBucket]
  OID: 38ac7 len 7947 [BTrees.IIBTree.IISet]
  OID: 387f6 len 97 [BTrees.IIBTree.IISet]
  OID: 383f7 len 850 [BTrees.OOBTree.OOBucket]
  OID: 4081a len 47 [BTrees.IIBTree.IISet]
  OID: 38407 len 850 [BTrees.OOBTree.OOBucket]
  OID: 4081b len 47 [BTrees.IIBTree.IISet]
  OID: 388ac len 92 [BTrees.IIBTree.IISet]
  OID: 387d4 len 152 [BTrees.IIBTree.IISet]
  OID: 3868c len 152 [BTrees.IIBTree.IISet]
  OID: 38681 len 142 [BTrees.IIBTree.IISet]
  OID: 388b0 len 72 [BTrees.IIBTree.IISet]
  OID: 384f1 len 52 [BTrees.IIBTree.IISet]
  OID: 37ca6 len 586 [BTrees.IOBTree.IOBucket]
  OID: 4081c len 686 [BTrees.IOBTree.IOBucket]
  OID: 37ab8 len 39336 [BTrees.IOBTree.IOBTree]
  OID: 381d8 len 594 [BTrees.OIBTree.OIBucket]
  OID: 38ac9 len 1252 [BTrees.IIBTree.IISet]
  OID: 38770 len 52 [BTrees.IIBTree.IISet]
  OID: 37d94 len 1234 [BTrees.IOBTree.IOBucket]
  OID: 3821d len 617 [BTrees.OIBTree.OIBucket]
  OID: 38acb len 557 [BTrees.IIBTree.IISet]
  OID: 38710 len 52 [BTrees.IIBTree.IISet]
  OID: 386ac len 52 [BTrees.IIBTree.IISet]
  OID: 38409 len 1019 [BTrees.OOBTree.OOBucket]
  OID: 4081d len 47 [BTrees.IIBTree.IISet]
  OID: 3870b len 52 [BTrees.IIBTree.IISet]
  OID: 38403 len 816 [BTrees.OOBTree.OOBucket]
  OID: 4081e len 47 [BTrees.IIBTree.IISet]
  OID: 387fe len 57 [BTrees.IIBTree.IISet]
  OID: 387cc len 67 [BTrees.IIBTree.IISet]
  OID: 38b29 len 1228 [BTrees.IOBTree.IOBucket]
  OID: 38c19 len 904 [BTrees.IOBTree.IOBucket]
  OID: 38d37 len 1007 [BTrees.IOBTree.IOBucket]
  OID: 3c610 len 33864 [BTrees.IOBTree.IOBucket]
 
  - Original Message -
  Sent: Monday, June 25, 2001 6:07 PM
  Subject: Re: [Zope-dev] Zcatalog bloat problem (berkeleydb is a
solution?)
 
A solution might be a kind of lazy catalog awareness: Instead of
mangling a new object through one or more catalogs when it is
created,
this object could be added to a list of objects to be cataloged
later.
This way, the transaction to insert a new object would become much
cheaper. I'm working on this, but right now it is quite messy.
(I'm
new to Python and Zope, and hence I'm stumbling over a few, hmmm,
trip-wires...)
  
   This purpose aligns well with those of the ArmoredCatalog proposal as
  well..
   see http://dev.zope.org/Wikis/DevSite/Proposals/ArmoredCatalog .
  
But even using such a lazy catalog awareness, you might get into
trouble. Using the ZCatalog's find objects function, I hit the
limits
of my Linux box: 640 MB RAM were not enough...
  
   This should not happen.  :-(
  
   I'm really disappointed that the bloat and memory consumption issues
are
   still plaguing the ZCatalog.  At one point, I really thought we had it
   pretty much licked.  I suppose this was naive.
  
A few weeks ago, I've posted this (admittedly not fully cooked)
patch to
this list, but did not get yet any response.
  
   I apologize for this.  We have a fairly formalized process for
handling
   feature-ish collector issues, and this hasn't come round on the
guitar.
  I'm
   beyond disappointed that people are still having unacceptable bloat,
  enough
   that something like this patch needed to be submitted.  It's

Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Giovanni Maruzzelli

I use 2.3.3 with python 1.5.2 on freebsd 3

I'm not so picky about bloating, but adding a document of 1K adds some 400K,
and keeps growing.

How much eat for you (I know you cataloged some 50K documents)?

-giovanni
- Original Message -
Sent: Tuesday, June 26, 2001 1:48 PM
Subject: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a
solution?)


 Giovanni, which Zope version are you running?

 On Tue, 26 Jun 2001, Chris McDonough wrote:

  How many indexes do you have, what are the index types, and what do
  they index?  Likewise, what about metadata?  In your last message, you
  said there's about 20.  That's a heck of a lot of indexes.  Do you
  need them all?

 In my installation I have about 30 or 40
 Position(Text)Index/KeywordIndex/FieldIndex.  They don't bloat much, so I
 don't think that's the problem.  (The problem might be that we have
 different views on what bloating is, though :)


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Chris McDonough

Well,  I'm not sure, unfortunately.  I just wanted to get an idea of
what kinds of indexes you had.  The tranalyzer output doesn't mean too
much to me, because it shows BTree buckets and such getting updated,
which is completely understandable... there are at least two data
structures in the Catalog itself that use a BTree, and each index uses
at least two BTrees.  So it's not all that surprising to see that
output.  What is suprising is to hear the amount of growth a transaction
causes.  The only thing I can think of is that:

a) you're committing inappropriately (at times where it would be OK to
not commit)

b) the data fields your indexing or getting metadata from are large.

c) something awful happened between 2.3.2 and 2.3.3 that I dont
understand.

d) the problem is unrelated to the Catalog.

I'm afraid I can't be any more precise than that.

-C


Giovanni Maruzzelli wrote:
 
 Hi Chris,
 
 I don't think this is a problem of ObjectManager, also if it contribute to
 the bloating.
 
 We do breaks the content in subfolders, but our subfolders easily grows to
 contains some hundred objects.
 
 Do you think that the number of indexes contribute to the bloating? If this
 is important, we can try to compact them in a littler number (eg: the
 boolean indexes can become a sort of bitmask, eliminate the meta_type,
 etc.).
 
 This is our indexes (cut and paste from the ZMI), and following there is our
 metadata :
 
 INDEXES:
   PrincipiaSearchSource Text Index 2,524
   autore Keyword Index 4,055
   bflow0 Field Index 4,055
   bflow1 Field Index 4,055
   bflow2 Field Index 4,055
   bflow3 Field Index 4,055
   bflow4 Field Index 4,055
   bflow5 Field Index 4,055
   bflow6 Field Index 4,055
   bflow7 Field Index 4,055
   bflow8 Field Index 4,055
   bflow9 Field Index 4,055
   bobobase_modification_time Field Index 4,300
   dflow0 Field Index 4,055
   dflow1 Field Index 4,055
   id Field Index 4,300
   m_sflow0 Keyword Index 3,960
   m_sflow1 Keyword Index 3,960
   m_sflow2 Keyword Index 3,960
   meta_type Field Index 4,300
   pseudoId Text Index 4,054
   revisore Keyword Index 4,055
   title Text Index 3,844
 
 METADATA:
 
   bobobase_modification_time
   id
   meta_type
   pseudoId
   title

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread abel deuring

Hi Giovanni, Chris and all others,

Chris McDonough wrote:
 
 Hi Giovanni,
 
 How many indexes do you have, what are the index types, and what do they
 index?  Likewise, what about metadata?  In your last message, you said
 there's about 20.  That's a heck of a lot of indexes.  Do you need them
 all?
 
 I can see a potential reason for the problem you explain as and I
 remind you that as the folder get populated, the size that is added to
 each transaction grows, a folder with one hundred objects adds some
 100K... It's true that normal folders (most ObjectManager-derived
 containers actually) cause database bloat within undoing storages when
 an object is added or removed from it.  This is because it keeps a list
 of contained subobject names in an _objects attribute, which is a
 tuple.  When an object is added, the tuple is rewritten in entirety.  So
 for instance, if you've got 100 items in your folder, and you add one
 more, you rewrite all the instance data for the folder itself, which
 includes the (large) _objects tuple (and of course, any other raw
 attributes, like properties).  Over time, this can be problematic.
 
 Shane's BTreeFolder Product attempts to ameliorate this problem a bit by
 keeping the data that is normally stored in the _objects tuple in its
 own persistent object (a btree).
 
 Are you breaking the content up into subfolders?  This is recommended.
 
 I'm temped to postulate that perhaps your problem isn't as much ZCatalog
 as it is ObjectManager overhead.


Well, I'm not very familiar with the details about the sub-object
management of ObjectManager and friends. Moreover, I had yet a closer
look only into UnTextIndex, but not into UnIndex or UnKeywordIndex. So
take my comments with a grain of salt. 

A text index (class SearchIndex.UnTextIndex) is definetely is a cause of
bloating, if you use CatalogAware objects. An UnTextIndex maintains for
each word a list of documents, where this word appears. So, if a
document to be indexed contains, say, 100 words, 100 IIBTrees
(containing mappings documentId - word score) will be updated. (see
UnTextIndex.insertForwardIndexEntry) If you have a larger number of
documents, these mappings may be quite large: Assume 10.000 documents,
and assume that you have 10 words which appear in 30% of all documents.
Hence, each of the IIBTrees for these words contains 3000 entries. (Ok,
one can try to keep this number of frequent words low by using a good
stop word list, but at least for German, such a list is quite difficult
to build. And one can argue that many not too really frequent words
should be indexed in order to allow more precise phrase searches)I don't
know the details, how data is stored inside the BTress, so I can give
only a rough estimate of the memory requirements: With 32 bit integers,
we have at least 8 bytes per IIBTree entry (documentId and score), so
each of the 10 BTree for the frequent words has a minimum length of
3000*8 = 24000 bytes. 

If you now add a new document containing 5 of these frequent words, 5
larger BTrees will be updated. [Chris, let me know, if I'm now going to
tell nonsense...] I assume that the entire updated BTrees = 12 bytes
will be appended to the ZODB (ignoring the less frequent words) -- even
if the document contains only 1 kB text. 

This is the reason, why I'm working on some kind of lazy cataloging.
My approach is to use a Python class (or Base class,if ZClasses are
involved), which has a method manage_afterAdd. This method looks for
superValues of a type like lazyCatalog (derived from ZCatalog), and
inserts self.getPhysicalPath() into the update list of each found
lazyCatalog.

Later, a lazyCatalog can index all objects in this list. Then, then
bloating happens either in RAM (without subtransaction), or in a
temporary file, if you use subtransactions.

OK, another approach which fits better to your (Giovanni) needs might be
to use another data base than ZODB, but I'm afarid that even then
instant indexing will be an expensive process, if you have a large
number of documents.

Abel

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Erik Enge

On Tue, 26 Jun 2001, Giovanni Maruzzelli wrote:

 I'm not so picky about bloating, but adding a document of 1K adds some
 400K, and keeps growing.
: 
 How much eat for you (I know you cataloged some 50K documents)?

I can't remember, but surely not that much.  I had some 30.000 documents
that were about 30-60Kb on average (although some were several megabytes),
in addition to around 50.000 other objects (documents, if you like)
indexed.  My Data.fs would've been around 2.5GB if my memory serves me
correctly.

As I said, I had loads of Indexes too.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Eric Roby


Chris McDonough Wrote:
 CatalogAware is arguably broken and should really not be used.

 In the meantime, if you care at all about cataloging, do not use
 CatalogAware.  Instead, manage the recataloging yourself and don't
 uncatalog a changed object before recataloging it during this manual
 operation, because this defeats all of the carefully set up change
 detection code (which may or may not still be working since I last
 worked on it ;-)

Chris,

Thank you for your candor here.  I wish this minor detail had been disclosed
in the Zope book.  Chapter 9 was my holy grail when I started down this
trail (creating these new ZClasses that would auto catalog themselves).  It
looked good in print...  I have banked a good deal of my project on this
very service and ... well it is a bit frustrating to find out that I need to
go back and re-do my work.

Along this same vein,  I would suggest that (possibly) ZClasses don't really
work, either, and should not be used.  There was a comment from another
developer (on zope-dev a month or so ago) that essentially (in his own
words) made this very claim.  At the time, I chalked it up to this Real
Zope Developers Don't Use ZCLasses kinda comment.  There certainly are
enough Zope products out there that (at least) leverage some of the ZClass
plumbing.

Another claim in the Zope book (chapter 8) says that I can leverage my 6+
years of Perl experience to create Zope scripts.  Well, I would suggest that
this doesn't really work, either...

The bottom line to all this venting (and I am not trying to shoot the
messenger here) is that I need to understand where my efforts should be
focused.  If I need to abandon ZClasses in lieu of pure Python, then I need
to know that now so I don't waste any more time on these false starts. The
Perl thing is just a matter of principle (I think Perl's implementation of
OO stinks).  The way it is presented in the book, I would expect it to be a
core Zope thing and not some appendage that requires a particular compiler
and Andy sitting next to you.

I don't intend to abandon Zope, I just need a reality check...

Eric

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Chris McDonough

Eric Roby wrote:
 
 Chris McDonough Wrote:
  CatalogAware is arguably broken and should really not be used.
 
  In the meantime, if you care at all about cataloging, do not use
  CatalogAware.  Instead, manage the recataloging yourself and don't
  uncatalog a changed object before recataloging it during this manual
  operation, because this defeats all of the carefully set up change
  detection code (which may or may not still be working since I last
  worked on it ;-)
 
 Chris,
 
 Thank you for your candor here.  I wish this minor detail had been disclosed
 in the Zope book.  Chapter 9 was my holy grail when I started down this
 trail (creating these new ZClasses that would auto catalog themselves).  It
 looked good in print...  I have banked a good deal of my project on this
 very service and ... well it is a bit frustrating to find out that I need to
 go back and re-do my work.

Well.. actually, it's pretty simple to change CatalogAware to work
better for you.  
With a little thought, CatalogAware could be hacked at your end to be
sane for your application.  You needn't rewrite all your code.  It's
just hard for DC to release a perfect CatalogAware that works better and
is completely backwards-compatible.  It's much harder to change it to
work perfectly for everybody (which is our job here ;-) than to change
it to work perfectly for a particular application.

Basically, change the reindex_object method to:

  self.index_object()

Instead of:

  self.unindex_object()
  self.index_object()

That makes CatalogAware much saner and will produce less bloat. 
Actually, maybe I should just go make that change in the trunk and the
2.4 branch, although I'm a little afraid of what (if anything) it will
break for everybody.  To be honest, I really don't have much time to
spend thinking about this, and my fears are probably just FUD.

 Along this same vein,  I would suggest that (possibly) ZClasses don't really
 work, either, and should not be used.  There was a comment from another
 developer (on zope-dev a month or so ago) that essentially (in his own
 words) made this very claim.  At the time, I chalked it up to this Real
 Zope Developers Don't Use ZCLasses kinda comment.  There certainly are
 enough Zope products out there that (at least) leverage some of the ZClass
 plumbing.

Well, I dont use ZClasses much.  But that's because I like to use Emacs.
 
 Another claim in the Zope book (chapter 8) says that I can leverage my 6+
 years of Perl experience to create Zope scripts.  Well, I would suggest that
 this doesn't really work, either...

Not sure what you mean by doesnt work, but I assume you've had an
unpleasant experience with zope-perl?
 
 The bottom line to all this venting (and I am not trying to shoot the
 messenger here) is that I need to understand where my efforts should be
 focused.  If I need to abandon ZClasses in lieu of pure Python, then I need
 to know that now so I don't waste any more time on these false starts. The

I'll go out on a limb here.  You should learn how to write Python
Products if you're serious about creating reusable Zope applications. 
There.

 Perl thing is just a matter of principle (I think Perl's implementation of
 OO stinks).  The way it is presented in the book, I would expect it to be a
 core Zope thing and not some appendage that requires a particular compiler
 and Andy sitting next to you.

I've sort of enjoyed myself on all the times when Andy has been sitting
near me, but I understand.  ;-) Jim had a bad experience installing
zope-perl lately.  I wish I could help.  Strangely, myself, I had few
problems getting it installed and working fine.  Maybe I'm just lucky. 
I actually think zope-perl is sort of an engineering marvel myself.
 
 I don't intend to abandon Zope, I just need a reality check...

HTH,

- C

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope] Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Chris McDonough

Off the top of my head, I don't think there are any.  But this is why I
haven't fixed it yet, because I'd need to think about it past off the
top of my head.  ;-)

- C


Casey Duncan wrote:
 
 
 What if any disadvantages are there to not calling unindex_object first?
 If there aren't any good ones, I think I'll be rewriting some of my own
 CatalogAware code...
 --
 | Casey Duncan
 | Kaivo, Inc.
 | [EMAIL PROTECTED]
 `--
 
 ___
 Zope maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope
 **   No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope-dev )

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Erik Enge

On Tue, 26 Jun 2001, Eric Roby wrote:

 The bottom line to all this venting (and I am not trying to shoot the
 messenger here) is that I need to understand where my efforts should
 be focused.  If I need to abandon ZClasses in lieu of pure Python,
 then I need to know that now so I don't waste any more time on these
 false starts.

If your application can't be written in five minutes and you expect to use
it more than once, you shouldn't use ZClasses - IMO.  The only argument
for ZClasses (that I had at the time) was that it was very easy and fast
to set up a couple of classes and some instances.  After I wrote mk-zprod,
making Python Products is even faster than ZClasses, and certainly scales
better.

If you ask me, it would be better to streamline the Zope API a bit and
focus the effort on making it easier to start developing Python Products
at first go, instead of stopping by ZClasses.  I can't see the rationale
for ZClasses, but I'm sure there is one.  Right?

I seem to recall some fuzz about Python Products starting be alive in
the Zope instance (ie. behaving much like ZClasses) in a future release.  
I don't know if that's a good thing or not, but if it means ditching
ZClasses I'm all for it.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Toby Dickenson

INDEXES:
  PrincipiaSearchSource Text Index 2,524
  autore Keyword Index 4,055
  bflow0 Field Index 4,055
  bflow1 Field Index 4,055
  bflow2 Field Index 4,055


Aha! a clue.

If that is the output of the 'Indexes' tab then I dont think you are
using the newest ZCatalog. A recent release (im not surwe which,
2.3.2?) has a new BTree implementation that reduces bloat by modifying
fewer buckets (it also doesnt have the column showing index size)

Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Giovanni Maruzzelli

I'm sorry to say that Toby is right in pointing at the version from which I
cutted and pasted the following, but we are using also a newer version and
the problem is the same.

We're working out our way with the dump the first bytes of the raw dump of
the new, magnificent tranalyzer from Toby (it reallly ought to be a standard
tool in the Zope distro), and we have now some hints of what happen when you
catalog something.

So, we are starting to optimize indexes and metadata, but the problem seems
not to fade away.

-giovanni


- Original Message -
From: Toby Dickenson [EMAIL PROTECTED]
To: Giovanni Maruzzelli [EMAIL PROTECTED]
Cc: Chris McDonough [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Tuesday, June 26, 2001 5:49 PM
Subject: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a
solution?)


 INDEXES:
   PrincipiaSearchSource Text Index 2,524
   autore Keyword Index 4,055
   bflow0 Field Index 4,055
   bflow1 Field Index 4,055
   bflow2 Field Index 4,055


 Aha! a clue.

 If that is the output of the 'Indexes' tab then I dont think you are
 using the newest ZCatalog. A recent release (im not surwe which,
 2.3.2?) has a new BTree implementation that reduces bloat by modifying
 fewer buckets (it also doesnt have the column showing index size)

 Toby Dickenson
 [EMAIL PROTECTED]


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Chris Withers

Toby Dickenson wrote:
 
 INDEXES:
   PrincipiaSearchSource Text Index 2,524
   autore Keyword Index 4,055
   bflow0 Field Index 4,055
   bflow1 Field Index 4,055
   bflow2 Field Index 4,055
 
 Aha! a clue.
 
 If that is the output of the 'Indexes' tab then I dont think you are
 using the newest ZCatalog. A recent release (im not surwe which,
 2.3.2?) has a new BTree implementation that reduces bloat by modifying
 fewer buckets (it also doesnt have the column showing index size)

Has the person concerned run the catalog update tool when they upgraded their
Zope version?

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Giovanni Maruzzelli

The catalog is a pristine 2.3.3b1 catalog.

We have recreated the catalog from scratch because we tried
manage_convertBTrees , but it don't work for us, it return with an error
(and the same happens with 2.3.3 final):

Error Type: TypeError
Error Value: second argument must be a class


Traceback (innermost last):
  File /fs1root/zope/Zope-2.3.3b1-src/lib/python/ZPublisher/Publish.py, line
223, in publish_module
  File /fs1root/zope/Zope-2.3.3b1-src/lib/python/ZPublisher/Publish.py, line
187, in publish
  File /fs1root/zope/Zope-2.3.3b1-src/lib/python/Zope/__init__.py, line 221,
in zpublisher_exception_hook
(Object: Traversable)
  File /fs1root/zope/Zope-2.3.3b1-src/lib/python/ZPublisher/Publish.py, line
171, in publish
  File /fs1root/zope/Zope-2.3.3b1-src/lib/python/ZPublisher/mapply.py, line
160, in mapply
(Object: manage_convertBTrees)
  File /fs1root/zope/Zope-2.3.3b1-src/lib/python/ZPublisher/Publish.py, line
112, in call_object
(Object: manage_convertBTrees)
  File
/fs1root/zope/Zope-2.3.3b1-src/lib/python/Products/ZCatalog/ZCatalog.py,
line 736, in manage_convertBTrees
(Object: Traversable)
  File
/fs1root/zope/Zope-2.3.3b1-src/lib/python/Products/ZCatalog/Catalog.py, line
204, in _convertBTrees
  File /fs1root/zope/Zope-2.3.3b1-src/lib/python/SearchIndex/UnTextIndex.py,
line 211, in _convertBTrees
TypeError: (see above)



- Original Message -
From: Chris Withers [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Cc: Giovanni Maruzzelli [EMAIL PROTECTED]; Chris McDonough
[EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Tuesday, June 26, 2001 5:59 PM
Subject: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a
solution?)


 Toby Dickenson wrote:
 
  INDEXES:
PrincipiaSearchSource Text Index 2,524
autore Keyword Index 4,055
bflow0 Field Index 4,055
bflow1 Field Index 4,055
bflow2 Field Index 4,055
 
  Aha! a clue.
 
  If that is the output of the 'Indexes' tab then I dont think you are
  using the newest ZCatalog. A recent release (im not surwe which,
  2.3.2?) has a new BTree implementation that reduces bloat by modifying
  fewer buckets (it also doesnt have the column showing index size)

 Has the person concerned run the catalog update tool when they upgraded
their
 Zope version?

 cheers,

 Chris


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Giovanni Maruzzelli

We think that Abel is absolutely right:

if in the same almost empty folder we add and catalog an object with one
word (and now we have optimized and reduced the number of indexes to 11) it
make a transaction of 73K, while if the object contains 300 words with the
same other indexes or properties, the transaction is 224K, and if all is the
same but the object contains 535 words the transaction is 331K.

And we are using now a catalog with only some 3000 document indexed with a
medium lenght of each document around 1K.

-giovanni

 Well, I'm not very familiar with the details about the sub-object
 management of ObjectManager and friends. Moreover, I had yet a closer
 look only into UnTextIndex, but not into UnIndex or UnKeywordIndex. So
 take my comments with a grain of salt.

 A text index (class SearchIndex.UnTextIndex) is definetely is a cause of
 bloating, if you use CatalogAware objects. An UnTextIndex maintains for
 each word a list of documents, where this word appears. So, if a
 document to be indexed contains, say, 100 words, 100 IIBTrees
 (containing mappings documentId - word score) will be updated. (see
 UnTextIndex.insertForwardIndexEntry) If you have a larger number of
 documents, these mappings may be quite large: Assume 10.000 documents,
 and assume that you have 10 words which appear in 30% of all documents.
 Hence, each of the IIBTrees for these words contains 3000 entries. (Ok,
 one can try to keep this number of frequent words low by using a good
 stop word list, but at least for German, such a list is quite difficult
 to build. And one can argue that many not too really frequent words
 should be indexed in order to allow more precise phrase searches)I don't
 know the details, how data is stored inside the BTress, so I can give
 only a rough estimate of the memory requirements: With 32 bit integers,
 we have at least 8 bytes per IIBTree entry (documentId and score), so
 each of the 10 BTree for the frequent words has a minimum length of
 3000*8 = 24000 bytes.

 If you now add a new document containing 5 of these frequent words, 5
 larger BTrees will be updated. [Chris, let me know, if I'm now going to
 tell nonsense...] I assume that the entire updated BTrees = 12 bytes
 will be appended to the ZODB (ignoring the less frequent words) -- even
 if the document contains only 1 kB text.

 This is the reason, why I'm working on some kind of lazy cataloging.
 My approach is to use a Python class (or Base class,if ZClasses are
 involved), which has a method manage_afterAdd. This method looks for
 superValues of a type like lazyCatalog (derived from ZCatalog), and
 inserts self.getPhysicalPath() into the update list of each found
 lazyCatalog.

 Later, a lazyCatalog can index all objects in this list. Then, then
 bloating happens either in RAM (without subtransaction), or in a
 temporary file, if you use subtransactions.

 OK, another approach which fits better to your (Giovanni) needs might be
 to use another data base than ZODB, but I'm afarid that even then
 instant indexing will be an expensive process, if you have a large
 number of documents.

 Abel


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-26 Thread Andy McKay

One thing Id been musing about for a while was a ZClass  Python Product
script that took your ZClass and set up your basic python product for you.
It would only work for simple for things like permissions, properties, basic
methods... Then ZClasses could be an easier springboard into python products
for those new to them.

Cheers.
--
  Andy McKay.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread abel deuring

Hi all,

Giovanni Maruzzelli wrote:
 
 We think that Abel is absolutely right:
 
 if in the same almost empty folder we add and catalog an object with one
 word (and now we have optimized and reduced the number of indexes to 11) it
 make a transaction of 73K, while if the object contains 300 words with the
 same other indexes or properties, the transaction is 224K, and if all is the
 same but the object contains 535 words the transaction is 331K.
 
 And we are using now a catalog with only some 3000 document indexed with a
 medium lenght of each document around 1K.

Well, Chris certainly knows more about the internals of ZCatalog than I
do, so we should not ignore his comments to my mail :)

Chris McDonough wrote:

  If you now add a new document containing 5 of these frequent words, 5
  larger BTrees will be updated. [Chris, let me know, if I'm now going to
  tell nonsense...] I assume that the entire updated BTrees = 12 bytes
  will be appended to the ZODB (ignoring the less frequent words) -- even
  if the document contains only 1 kB text.
 
 Nah... I don't think so.  At least I hope not!  Each bucket in a BTree
 is a separate persistent object.  So only the sum of the data in the
 updated buckets will be appended to the ZODB.  So if you add an item to
 a BTree, you don't add 24000+ bytes for each update.  You just add the
 amount of space taken up by the bucket... unfortunately I don't know
 exactly how much this is, but I'd imagine it's pretty close to the
 datasize with only a little overhead.

OK, this made me curious, so I made test similar to the one by Giovanni.
I started with a ZCatalog containing 21616 records; the catalog contains
only one text index, no keyword index, no field index. I copied one of
the indexed documents; the text is 2645 bytes long; wc tells me that it
has 313 words. Next, I packed the data base in order to have a clean
start point. After packing, Data.fs has a size of 233661963 byte.

Then I cataloged the new object using my lazy catalog. Since I have
only one new document, this is basically the same as using
CatalogAwareness. After indexing, the data base has grown to 233851090
bytes -- an increase of 189127 bytes. Then I packed the data base again,
resulting in a size of 233666237 bytes.

So the net increase is indeed 233666237-233661963 = 4274 bytes, as you
expected, but obviously a few more data base records need to be updated.

Abel

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-26 Thread Stephan Richter

Hello everyone,

I gotta join this discussion.

iuveno was also thinking about a tool that would replace ZClasses, since 
their performance is far too bad. We had a not so good experience with the 
ZClass-based Kontentor and now that the first part is rewritten in Python 
we can see the speed-ups (the performance increase can be measured in 
multiples - real tests need to be done).
The reason - in my opinion - that ZClasses are so slow are the huge amount 
of Acquistion lookups and the save rendering. You can often code things 
smarter in Python using much less of the safe Zope environment, but still 
providing safety through specific commands. I think Formulator is a great 
example of how safe the Python programming can be.
There are two thoughts here:

1. We are building a wizard that asks you all the necessary questions to 
generate a basic class framwework. This wizard (which can be used in many 
other fields - such as installers - as well) is currently being built. We 
use Formulator a lot and I support the development of it as much as I can 
(it is a cool product with many cool features). If anyone is interested in 
helping developing that tool (which will be released under the GPL as all 
of the iuveno products), then I can make an electronic copy of my personal 
notes and I setup a CVS.
Formulator at Zope: http://www.zope.org/Members/faassen/Formulator
Formulator at Sourceforge: http://sourceforge.net/projects/formulator/


2. Phillip Auersperg from bluedynamics.com uses ObjectDomain quiet heavily, 
since it has a nice JPython API that comes with it. He already built a 
reverse-engineering tool for ZClasses and is now going to write another 
tool to automatically generate DBObjects from a UML diagram in 
ObjectDomain. I am very excited about this tool, since it will make the 
already fast DBObject development even faster. As soon as Formulator goes 
into 1.0, I am going to think about binding the Formulator to DBObjects, so 
you can quickly generate forms for each object. In Berlin in two weeks, we 
are going to discuss this integration in more detail...
Bluedynamics URL: http://www.bluedynamics.com
DBObjects/SmartObjects URL: http://demo.iuveno-net.de


I am very glad to see that we all have the same vision. We all just need to 
work more together (especially me included). This way we can have some 
strikes against the big ones...

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Chris McDonough

Yikes.  I wonder if this overhead comes from Vocabulary updates... thanks
very much for doing this test.

Clearly we need to pin it down.  This is very disappointing.  :-(  Any
further info you dig up is appreciated.

You didn't have any metadata stuff set up, did you?  I imagine even if you
did, that they couldn't possibly account for 200K worth of extra stuff.

- C

- Original Message -
From: abel deuring [EMAIL PROTECTED]
To: Giovanni Maruzzelli [EMAIL PROTECTED]
Cc: Chris McDonough [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Tuesday, June 26, 2001 2:40 PM
Subject: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a
solution?)


 Hi all,

 Giovanni Maruzzelli wrote:
 
  We think that Abel is absolutely right:
 
  if in the same almost empty folder we add and catalog an object with one
  word (and now we have optimized and reduced the number of indexes to 11)
it
  make a transaction of 73K, while if the object contains 300 words with
the
  same other indexes or properties, the transaction is 224K, and if all is
the
  same but the object contains 535 words the transaction is 331K.
 
  And we are using now a catalog with only some 3000 document indexed with
a
  medium lenght of each document around 1K.

 Well, Chris certainly knows more about the internals of ZCatalog than I
 do, so we should not ignore his comments to my mail :)

 Chris McDonough wrote:

   If you now add a new document containing 5 of these frequent words, 5
   larger BTrees will be updated. [Chris, let me know, if I'm now going
to
   tell nonsense...] I assume that the entire updated BTrees = 12
bytes
   will be appended to the ZODB (ignoring the less frequent words) --
even
   if the document contains only 1 kB text.
 
  Nah... I don't think so.  At least I hope not!  Each bucket in a BTree
  is a separate persistent object.  So only the sum of the data in the
  updated buckets will be appended to the ZODB.  So if you add an item to
  a BTree, you don't add 24000+ bytes for each update.  You just add the
  amount of space taken up by the bucket... unfortunately I don't know
  exactly how much this is, but I'd imagine it's pretty close to the
  datasize with only a little overhead.

 OK, this made me curious, so I made test similar to the one by Giovanni.
 I started with a ZCatalog containing 21616 records; the catalog contains
 only one text index, no keyword index, no field index. I copied one of
 the indexed documents; the text is 2645 bytes long; wc tells me that it
 has 313 words. Next, I packed the data base in order to have a clean
 start point. After packing, Data.fs has a size of 233661963 byte.

 Then I cataloged the new object using my lazy catalog. Since I have
 only one new document, this is basically the same as using
 CatalogAwareness. After indexing, the data base has grown to 233851090
 bytes -- an increase of 189127 bytes. Then I packed the data base again,
 resulting in a size of 233666237 bytes.

 So the net increase is indeed 233666237-233661963 = 4274 bytes, as you
 expected, but obviously a few more data base records need to be updated.

 Abel

 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread abel deuring

Chris McDonough wrote:
 
 Yikes.  I wonder if this overhead comes from Vocabulary updates... thanks
 very much for doing this test.

No, this should definetely _not_ be related to vocabulary: I simply
copied an already indexed document and let ZCatalog.catalog_object munge
the copy. So all words appearing in this copy already have an entry in
the Vocabulary. I also checked it during a test without meta data: The
vocabulary doed not increase.

 
 Clearly we need to pin it down.  This is very disappointing.  :-(  Any
 further info you dig up is appreciated.

Well, I don't have any at present. But allow me to make some guess :) If
a new record is added to a BTree, is can be necessary to move a few
other records around in order to keep the tree balanced. And some of the
BTrees affected by my test are definitely somewhat larger, because I did
not use German stop words during the test, so words like und, der,
die are indexed which appear in _every_ document. (well, at least in
_nearly_ every document)

 
 You didn't have any metadata stuff set up, did you?  I imagine even if you
 did, that they couldn't possibly account for 200K worth of extra stuff.

Ouch, I forgot about the meta data. So here is the result of another
test, with all meta data thrown away:

Packed data base size, one document (same during the last test) to be
cataloged: 
229170221 bytes.

data base size after updating the catalog run: 229310316 bytes
size after packing: 229172566 bytes

So, same as before :(

Abel

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Chris McDonough

 Chris McDonough wrote:
 
  Yikes.  I wonder if this overhead comes from Vocabulary updates...
thanks
  very much for doing this test.

 No, this should definetely _not_ be related to vocabulary: I simply
 copied an already indexed document and let ZCatalog.catalog_object munge
 the copy. So all words appearing in this copy already have an entry in
 the Vocabulary. I also checked it during a test without meta data: The
 vocabulary doed not increase.

OK, that's good to know...

 
  Clearly we need to pin it down.  This is very disappointing.  :-(  Any
  further info you dig up is appreciated.

 Well, I don't have any at present. But allow me to make some guess :) If
 a new record is added to a BTree, is can be necessary to move a few
 other records around in order to keep the tree balanced. And some of the
 BTrees affected by my test are definitely somewhat larger, because I did
 not use German stop words during the test, so words like und, der,
 die are indexed which appear in _every_ document. (well, at least in
 _nearly_ every document)

 
  You didn't have any metadata stuff set up, did you?  I imagine even if
you
  did, that they couldn't possibly account for 200K worth of extra stuff.

 Ouch, I forgot about the meta data. So here is the result of another
 test, with all meta data thrown away:

 Packed data base size, one document (same during the last test) to be
 cataloged:
 229170221 bytes.

 data base size after updating the catalog run: 229310316 bytes
 size after packing: 229172566 bytes

 So, same as before :(

Well, I'm sort of stumped without doing it myself, and I can't at the
moment.  I'm going to add this to the Collector so I don't forget, and
hopefully it will be looked into and fixed by the time that 2.4.0 goes out.

Thanks so much,

- C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Eric Roby


Erik Enge wrote on 26 June:
 If your application can't be written in five minutes and you expect to use
 it more than once, you shouldn't use ZClasses - IMO.  The only argument
 for ZClasses (that I had at the time) was that it was very easy and fast
 to set up a couple of classes and some instances.  After I wrote mk-zprod,
 making Python Products is even faster than ZClasses, and certainly scales
 better.

Thank you for your thoughts.  The comments throughout this thread have been
very insightful.  I see where I need to go from here.  I will be checking
out mk-zprod. I have a nagging question that you might be able to help me
with (in light of mk-zprod).  I understand that the 'Class-Id that a ZClass
gets assigned upon creations is vital to Zope's ability to manage class
instances.  Is it possible to make a Python class to replace the ZClasses I
have already created and be able to support the ZClass instances I have
already created?  If so, how do you get a Python class to answer to a
specific Class-Id.

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-26 Thread Stephan Richter


What is ZPI?

Typo: Read API

  Well, the SmartWizard would be like the frontend for it. If you want
  to see an early non-SmartWizard-framework version, look at the
  ProiektorInstaller.  Just imagine you can build installers like that
  in Zope. I hope to be done with the first version on Thursday.

Great, will you let us know?

Of course. ;-)

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-26 Thread Andy McKay

 What is ZPI?
 
 Typo: Read API

Its the Zope version of an API :)

Cheers.
--
  Andy McKay.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-26 Thread Stephan Richter

At 06:40 PM 6/26/01 -0700, Andy McKay wrote:
  What is ZPI?
 
  Typo: Read API

Its the Zope version of an API :)

If you want to see it. Yeah, we just created another three-letter acronym 
for this world!!! So everyone, it is not anymore API, but ZPI...geez, I am 
starting to get silly, that means I need some sleep...

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: Something better than ZClasses (was: Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?))

2001-06-26 Thread Stephan Richter


  I will upload the documents tomorrow though, since it is late here and
  I have to do some work still.

Ok.  I'll begin thinking about all the stuff I dreamt of making mk-zprod
into.

Okay, okay...I stayed up and typed it down pretty quick (2 hours). I 
attached it to this mail. It is plain text, since I was too lazy to do it 
in HTML. It might be a little unstructured, but I am too tired to fix that now.

Regards,
Stephan

--
Stephan Richter
CBU - Physics and Chemistry Student
Web2k - Web Design/Development  Technical Project Management

SmartWizards - A framework to generate the Every-Day-Wizard
===
by Stephan Richter in June 2001

Version 0.1


The following parts of a *default* WizardPage are defined:

+-+--+
| |  |
|Logo |   Header and Long Description|
| |  |
+-+--+
| |  |
|  Wizard |Wizard|
| |  |
|  Overview   | Main |
| |  |
| |Window|
| |  |
| |  |
| |  |
| |  |
| |  |
+-+--+
|  Status Messages   |
++
|  Navigation Bar (buttons to move)  |
++


Therefore there are the following standard methods defined in a wizard:

wizardHeaderwizardStatus
wizardFooterwizardNavigationBar
wizardOverview  wizardMainWindow
wizardDescription


Functionalities of these methods:
=

wizardOverview:
---
  - should display a list of all pages (display short description/title)
  - needs to know about the active page to highlight it

  Default Output: A numbered list of all the pages with the active one 
  highlighted.


wizardDescription:
--
  - shows the 'long' description of the active page
  - there might be also a static part, depending on the application


wizardStatus:
-
  - status messages/information
  o Errors by form validation
  o administrative messages (required fields, ...)
  o wizard information (page x out of y)


wizardNavigation:
-
  - there should be only buttons here!
  - maybe we should define where the buttons are, since grouping some
of them will be necessary


wizardMainWindow:
-
  - here goes the real page information
  - there can be: forms, other actions, information and everything mixed



Classes
===

SmartWizard -- Folder
--

  - we are going to use Sessions (CoreSessionTracking) and Versions to keep track of 
the
user's status.
  - Since it will be too hard to give all information first and then commit everything 
at the end, we decided to use versions, which we can always not commit, if a roll-
back is requested.
  - All the other info is saved in the session, since several people at once could use
the wizard at once.
  - Also, we will keep track of the active page by simply storing the index of the 
page 
inside the Pages folder; since it is an OrderedFolder we can do this safely.

  - methods: wizardHeader 
 wizardFooter
 wizardCSS
 wizardStatus
 wizardNavigationBar
 wizardOverview
 wizardMainWindow
 wizardDescription
 wizardSession - Contains all the session data information
 Object is of type SessionDataManager.   
 Versions  - Folder that contains versions of the people which use the 
wizard.
 Object is of type Folder.
 Pages - The container of all the WizardPages that being displayed 
during 
 the Wizard.
 Object is of type OrderedFolder (see Zope.org).

  - attributes: activePageIndex - specific value is stored in the session 

Re: [Zope] Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Chris McDonough

abel deuring wrote:
 A text index (class SearchIndex.UnTextIndex) is definetely is a cause of
 bloating, if you use CatalogAware objects. An UnTextIndex maintains for

Right.. if you don't use CatalogAware, however, and don't unindex before
reindexing an object, you should see a huge bloat savings, because the
only things which are supposed to be updated then are indexes and
metadata which have data that has changed.

 each word a list of documents, where this word appears. So, if a
 document to be indexed contains, say, 100 words, 100 IIBTrees
 (containing mappings documentId - word score) will be updated. (see
 UnTextIndex.insertForwardIndexEntry) If you have a larger number of
 documents, these mappings may be quite large: Assume 10.000 documents,
 and assume that you have 10 words which appear in 30% of all documents.
 Hence, each of the IIBTrees for these words contains 3000 entries. (Ok,
 one can try to keep this number of frequent words low by using a good
 stop word list, but at least for German, such a list is quite difficult
 to build. And one can argue that many not too really frequent words
 should be indexed in order to allow more precise phrase searches)I don't
 know the details, how data is stored inside the BTress, so I can give
 only a rough estimate of the memory requirements: With 32 bit integers,
 we have at least 8 bytes per IIBTree entry (documentId and score), so
 each of the 10 BTree for the frequent words has a minimum length of
 3000*8 = 24000 bytes.
 
 If you now add a new document containing 5 of these frequent words, 5
 larger BTrees will be updated. [Chris, let me know, if I'm now going to
 tell nonsense...] I assume that the entire updated BTrees = 12 bytes
 will be appended to the ZODB (ignoring the less frequent words) -- even
 if the document contains only 1 kB text.

Nah... I don't think so.  At least I hope not!  Each bucket in a BTree
is a separate persistent object.  So only the sum of the data in the
updated buckets will be appended to the ZODB.  So if you add an item to
a BTree, you don't add 24000+ bytes for each update.  You just add the
amount of space taken up by the bucket... unfortunately I don't know
exactly how much this is, but I'd imagine it's pretty close to the
datasize with only a little overhead.
 
 This is the reason, why I'm working on some kind of lazy cataloging.
 My approach is to use a Python class (or Base class,if ZClasses are
 involved), which has a method manage_afterAdd. This method looks for
 superValues of a type like lazyCatalog (derived from ZCatalog), and
 inserts self.getPhysicalPath() into the update list of each found
 lazyCatalog.
 
 Later, a lazyCatalog can index all objects in this list. Then, then
 bloating happens either in RAM (without subtransaction), or in a
 temporary file, if you use subtransactions.
 
 OK, another approach which fits better to your (Giovanni) needs might be
 to use another data base than ZODB, but I'm afarid that even then
 instant indexing will be an expensive process, if you have a large
 number of documents.

Another option is to use a session manager, and update the catalog at
session-end.

- C

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope] Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Casey Duncan

Chris McDonough wrote:
 
 abel deuring wrote:
  A text index (class SearchIndex.UnTextIndex) is definetely is a cause of
  bloating, if you use CatalogAware objects. An UnTextIndex maintains for
 
 Right.. if you don't use CatalogAware, however, and don't unindex before
 reindexing an object, you should see a huge bloat savings, because the
 only things which are supposed to be updated then are indexes and
 metadata which have data that has changed.
 
[snip]

What if any disadvantages are there to not calling unindex_object first?
If there aren't any good ones, I think I'll be rewriting some of my own
CatalogAware code...
-- 
| Casey Duncan
| Kaivo, Inc.
| [EMAIL PROTECTED]
`--

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-26 Thread Erik Enge

Giovanni, which Zope version are you running?

On Tue, 26 Jun 2001, Chris McDonough wrote:

 How many indexes do you have, what are the index types, and what do
 they index?  Likewise, what about metadata?  In your last message, you
 said there's about 20.  That's a heck of a lot of indexes.  Do you
 need them all?

In my installation I have about 30 or 40
Position(Text)Index/KeywordIndex/FieldIndex.  They don't bloat much, so I
don't think that's the problem.  (The problem might be that we have
different views on what bloating is, though :)


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-25 Thread Erik Enge

(I removed [EMAIL PROTECTED].)

On Mon, 25 Jun 2001, Giovanni Maruzzelli wrote:

 Any hints on how to manage something like? We use both textindexes,
 fieldindexes, and keywordsindexes (textindex on string properties,
 fieldindexes on boolean and datetime, keywordindex on strings). Maybe
 one kind of indexes is to be avoided?
 
 Erik, any toughts?
 
Well, after ChrisM told me about the behaviour of CatalogAwareness, and I
removed that from my classes, my ZCatalog bloatness has evaporated.  I
really can't see any major bloat-problem on either memory-consumption or
disk-space.  (That was with Zope 2.3.2b2.)

Which is very good for me, but doesn't necessarily help you much.  :)


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] Re: Zcatalog bloat problem (berkeleydb is a solution?)

2001-06-25 Thread Chris McDonough

Part of the problem here is that if, in particular, you use the
reindex_object method of CatalogAware, the database will grow
unnecessarily even if the object hasn't changed.  CatalogAware is
arguably broken and should really not be used.  I'd like to have the
time to fix it, but fixing it implies taking time out that I don't have
at the moment to test the changes, and *may* imply breaking it in
slightly other backwards-incompatible ways that will cause people to
scream.  For instance, unfortunately, CatalogAware also uses the *url*
of the object to index it, which is contrary to the way that newer
Catalogs work (they index the physical path of the object).  

In the meantime, if you care at all about cataloging, do not use
CatalogAware.  Instead, manage the recataloging yourself and don't
uncatalog a changed object before recataloging it during this manual
operation, because this defeats all of the carefully set up change
detection code (which may or may not still be working since I last
worked on it ;-)

- C


Erik Enge wrote:
 
 (I removed [EMAIL PROTECTED].)
 
 On Mon, 25 Jun 2001, Giovanni Maruzzelli wrote:
 
  Any hints on how to manage something like? We use both textindexes,
  fieldindexes, and keywordsindexes (textindex on string properties,
  fieldindexes on boolean and datetime, keywordindex on strings). Maybe
  one kind of indexes is to be avoided?
 
  Erik, any toughts?
 
 Well, after ChrisM told me about the behaviour of CatalogAwareness, and I
 removed that from my classes, my ZCatalog bloatness has evaporated.  I
 really can't see any major bloat-problem on either memory-consumption or
 disk-space.  (That was with Zope 2.3.2b2.)
 
 Which is very good for me, but doesn't necessarily help you much.  :)

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )