Re: [ZODB-Dev] make ZODB as small and compact as expected

[Adam GROSZER]

On 07/22/2013 01:27 PM, Jim Fulton wrote:


- The biggest thing ZODB needs right now is documentation.
   Unfortunately, this isn't easy. There is zodb.org,
   but much better documentation is needed.


There is

https://github.com/cguardia/ZODB-Documentation

but seems like it got stalled 5 months ago.


--
Best regards,
 Adam GROSZER
--
Quote of the day:
Each time you are honest and conduct yourself with honesty, a success 
force will drive you toward greater success.  Each time you lie, even 
with a little white lie, there are strong forces pushing you toward 
failure. (Joseph Sugarman)

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] RelStorage 1.5.1 does not save anything

[Laurence Rowe]

It sounds like you're missing some transaction middleware in your wsgi
pipeline. See
https://groups.google.com/forum/#!topic/zope-core-dev/aB5BzvrVJxw for some
clues.


On 22 July 2013 22:58, Suresh V.  wrote:

> Also happens with RelStorage trunk from github.
>
> Tests run fine after bumping up the max_allowed_packets slightly.
>
>
>
> __**_
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list  -  ZODB-Dev@zope.org
> https://mail.zope.org/mailman/**listinfo/zodb-dev
>
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] RelStorage 1.5.1 does not save anything

[Suresh V.]

Also happens with RelStorage trunk from github.

Tests run fine after bumping up the max_allowed_packets slightly.


___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] BTrees package problems

[Christian Tismer]

On 22.07.13 13:08, Jim Fulton wrote:

On Sat, Jul 20, 2013 at 11:27 PM, Christian Tismer  wrote:

The BTrees package is an attempt to isolate certain things from ZODB.

While I appreciate the general intent, I cannot see the advantage at
this point:

- BTrees can be imported alone, yes. But it has the extensions prepared
with special ZODB slots, which makes this very questionable.

- BTrees furthermore claims the BTrees global bame for it, all though it
is not a general BTree package, but for ZODB BTrees, only.

Yeah, I worried about this when we broke it out.

OTOH, there isn't much concern with namespace
pollution in the Python community. :/


- BTrees has a serious bug, see the following example:


from BTrees import OOBTree as BT
t = BT.BTree()
for num in range(100):

...   k = str(num)
...   t[k] = k
...

t._firstbucket._next = None
len(t)

Bus error: 10
(tmp)minimax:doc tismer$

Ouch.


So there is either an omission to make t._next() read-only, or a check
of its validity is missing.

Yup.  OTOH, you're the first person to encounter this
after many years, so while this is bad, and needs to be
fixed, I'm not sure how serious it is as a practical matter.


Actually, I would like to add a callable-check instead, to allow for more
flexible derivatives.

I don't understand this.


Simple: I am writing BTree forests for versioned, read-only databases.

For that, I need a way to create a version of Bucket that allows to
override the _next field by maybe a callable.
Otherwise all the buckets are chained together and I have no way
to let frozen BTrees share buckets.

When I played with the structure, I was happy/astonished to see the 
_next field

being writable and thought it was intended to be so.
It was not, in the end ;-)

cheers - Chris

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] make ZODB as small and compact as expected

[Jürgen Herrmann]

Am 22.07.2013 13:27, schrieb Jim Fulton:
On Sun, Jul 21, 2013 at 12:12 AM, Christian Tismer 
 wrote:

This is my last emission for tonight.

I would be using ZODB as a nice little package if it was one.

There should be nothing else but

ZODB.

Instead, there is

BTrees
persistent
transaction
zc.lockfile
zc.zlibstorage
ZConfig
zdaemon
ZEO
ZODB
ZODB3   (zlibstorage)
zope.interface

and what I might have forgotton.

Exception:
There is also
zodbpickle
which I think is very usefull and general-purpose, and I wan to keep 
it,

also I will try to push it into standard CPython.

So, while all the packages are not really large, there are too many
namespaces
touched, and things like "Zope Enterprize Objects" are not meant to be 
here

as open source pretending modules which the user never asked for.


Despite it's tech-bubblishish acronym expansion, which
few people are aware of, ZEO is the standard client-server
component of ZODB, is widely used, and is certainly open source.



I think these things could be re-packed into a common namespace
and be made simpler.


If ZODB had been born much later, it would certainly have used
a namespace package.  Now, it would be fairly disruptive to change
it.


Even zope.interface could be removed from
this intended-to-be user-friendly simple package.


I don't understand what you're saying.  It's a dependency
if ZODB.


So while the amount of code is astonishingly small, the amount of
abstraction layering tells the reader that this was never really meant 
to

be small.

And this makes average, simple-minded users like me shy away and go
back to simpler modules like Durus.

But the latter has serious other pitfalls, which made me want to 
re-package
ZODB into something small, pretty, tool-ish, versatile thing for the 
pocket.


Actually I'm trying to re-map ZOPE to the simplistic Durus interface,
without its short-comings and lack of support.
I think a successfully down-scaled, isolated package with ZODB's
great implementation, but a more user-oriented interface would
help ZODB a lot to get widely accepted and incorporated into very
many projects.
Right now people are just too much concerned of implicit complication 
which

actually does not exist.

I volunteer to start such a project. Proposing the name "david", as 
opposed

to "goliath".


ZODB is an old project that has accumulated some cruft over the years,
however:

- I've tried to simplify it and, with the exception of ZEO,
  I think it's pretty straightforward.

- ZODB is used by a lot of people with varying
  needs and tastes.  The fact that it is pretty modular has
  allowed a lot of useful customizations.

- I'm pretty happy with the layered storage architecture.

- With modern package installation tools like buildout and pip,
  having lots of dependencies shouldn't be a problem.
  ZODB uses lots of packages that have uses outside of ZODB.
  I consider this a strength, not a weakness.

  Honestly, I have no interest in catering to users who don't use
  buildout, or pip, or easy_install.

- The biggest thing ZODB needs right now is documentation.
  Unfortunately, this isn't easy. There is zodb.org,
  but much better documentation is needed.


Very much agreed. The most important thing to mention here is IMO
the use of virtualenv which keeps your python installation's
site packages clean. I don't mind if a certain package I need
pulls in any number of dependencies, as the are resolved
automagiaclly. @Chris: Maybe you could explain a bit more, why that
bothers you? Which brings me to the second most important thing here:
documentation. As you seem to be starting a new project using ZODB,
maybe you could come up with a "Getting started with ZODB in your
own project" or so which would (how I picture it) include a step
by step walkthrough from zero to writing some simple Persistent
classes and using them. That woul make your life easier in the
long run as steps written down correctly tend to be very easy to
reproduce. On the other hand the community would benefit from some
neice peace of documentation.

Is there a ZODB wiki (didn't find one), where we could gather this
stuff? I've come accross a whole bunch of useful code snippets that
a use seldomly but which are very, very useful then. For example
"how do i find/retrieve a object with oid 0xYYY?", "How do I copy
transactions starting whith tid xxx from one ZODB to another?"

I keep these in my private CMS, where they are obviously only useful
to me. Time for a ZODB wiki?!

Cheers!
Jürgen



Jim


--

XLhost.de ® - Webhosting von supersmall bis eXtra Large <<


XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany

Geschäftsführer: Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218

Fon:  +49 (0)800 XLHOSTDE [0800 95467833]
Fax:  +49 (0)800 95467830
Web:  http://www.XLhost.de
___
For more inform

Re: [ZODB-Dev] make ZODB as small and compact as expected

[Christian Tismer]

On 22.07.13 18:01, Tres Seaver wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/22/2013 09:15 AM, Stephan Richter wrote:

BTrees

I agree, this could be part of ZODB and it would be fine.

Splitting out BTrees was a conscious decision to serve two goals:

- - Allow evolving it (in particular, the work to port it to Py3k / PyPy)
   without stalling on the larger ZODB project.  For ongoing work, it is
   useful to be able to release a fix for a BTrees-only bug without needing
   to release ZODB.

- - Allow projects which use BTrees (as base classes or attributes) to be
   tested without needing to install all of ZODB.

I consider both of those concerns still important, and so am -1 on
re-absorbing BTrees into ZODB.



Yes, I understand this intention and see no problem:
Just the namespace might be ZODB.Btrees which would not change
the split. They would still live alone, separate projects.

This is just plugged in, like zlibstorage (if it were not ZODB3 ;-) )

Minor point, anyway ;-)

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] make ZODB as small and compact as expected

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/22/2013 09:15 AM, Stephan Richter wrote:
>>> BTrees
> I agree, this could be part of ZODB and it would be fine.

Splitting out BTrees was a conscious decision to serve two goals:

- - Allow evolving it (in particular, the work to port it to Py3k / PyPy)
  without stalling on the larger ZODB project.  For ongoing work, it is
  useful to be able to release a fix for a BTrees-only bug without needing
  to release ZODB.

- - Allow projects which use BTrees (as base classes or attributes) to be
  tested without needing to install all of ZODB.

I consider both of those concerns still important, and so am -1 on
re-absorbing BTrees into ZODB.


Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlHtV3QACgkQ+gerLs4ltQ6PeACeNW11cnoU7IxG37tkOhGxuRXU
RBEAmwXv+JD32emv2MFXaiAqmGcTj+Qt
=oWDa
-END PGP SIGNATURE-

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] make ZODB as small and compact as expected

[Christian Tismer]

On 22.07.13 15:15, Stephan Richter wrote:

On Sunday, July 21, 2013 06:12:34 AM Christian Tismer wrote:

  BTrees

I agree, this could be part of ZODB and it would be fine.

...

  ZODB3   (zlibstorage)

Well, this package is deprecated. It is available for backward-compatibility.



Yes, but I meant zlibstorage, which pulls ZODB3 in.
I would like to put that as an optional package, but in ZODB (4)

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] BTrees package problems

[Christian Tismer]

On 22.07.13 16:38, Patrick Strawderman wrote:

On Jul 20, 2013, at 11:27 PM, Christian Tismer  wrote:


- BTrees has a serious bug, see the following example:


from BTrees import OOBTree as BT
t = BT.BTree()
for num in range(100):

...   k = str(num)
...   t[k] = k
...

t._firstbucket._next = None
len(t)

Bus error: 10
(tmp)minimax:doc tismer$

So there is either an omission to make t._next() read-only, or a check
of its validity is missing.

Maybe you could open an issue on Github?


Yes I can do that (and fix it).

I was just telling it here because I'd like to know how it is meant to
be.

- should the attributes be exposed at all? (I guess yes)

- are they meant to be writable? (probably not, although that is handy :)

I would actually like to be able to derive from Bucket and implement
copy-on-write semantics for FrozenBTree (not yet existing) without
re-coding much in C, this was the reason while I played around here.

For that purpose (sharing buckets) I need a way to make the _next
pointers indirect.

cheers - chris

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] BTrees package problems

[Patrick Strawderman]

On Jul 20, 2013, at 11:27 PM, Christian Tismer  wrote:

> - BTrees has a serious bug, see the following example:
> 
>> >>> from BTrees import OOBTree as BT
>> >>> t = BT.BTree()
>> >>> for num in range(100):
>> ...   k = str(num)
>> ...   t[k] = k
>> ...
>> >>> t._firstbucket._next = None
>> >>> len(t)
>> Bus error: 10
>> (tmp)minimax:doc tismer$
> 
> So there is either an omission to make t._next() read-only, or a check
> of its validity is missing.

Maybe you could open an issue on Github?
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] BTrees and ZODB simplicity

[Jim Fulton]

On Mon, Jul 22, 2013 at 8:11 AM, Christian Tismer  wrote:
> On 22.07.13 13:13, Jim Fulton wrote:
>>
>> On Sat, Jul 20, 2013 at 11:43 PM, Christian Tismer 
>> wrote:
>>>
>>> Third rant, dear Zope-Friends (and I mean it as friends!).
>>>
>>> In an attempt to make the ZODB a small, independant package, ZODB
>>> has been split into many modules.
>>
>> Maybe not as many as you think:
>> persistent, transaction, ZEO, ZODB and BTrees.
>>
>> 5 
>>
>>> I appreciate that, while I think it partially has the opposite effect:
>>>
>>> - splitting BTrees apart is a good idea per se.
>>> But the way as it is, it adds more Namespace-pollution than benefits:
>>>
>>> To make sense of BTrees, you need the ZODB, and only the ZODB!
>>> So, why should then BTrees be a top-level module at all?
>>>
>>> This does not feel natural, but eavesdropping, pretending as
>>> something
>>> that is untrue.
>>>
>>> I think:
>>>
>>>   - BTrees should either be a ZODB sub-package in its current state,
>>>
>>>   - or a real stand-alone package with some way of adding persistence as
>>> an option.
>>
>> I don't agree that because a package depends on ZODB
>> it should be in ZODB.  There are lots of packages that depend
>> on ZODB.
>
>
> This is generally true. In the case of BTrees, I think the ZODB
> is nothing without BTrees, and BTrees make no sense without
> a storage and carry those _p_ which are not optional.

This is true of every class that subclasses Persistent.

>
> BTrees would make more sense as a standalone package if the persistence
> model were pluggable. But that is also theoretical because I don't see
> right now how to split that further with all the C code.

Well, it's definitely possible.  Early in the evolution of BTrees, there we
ifdefs that turned off dependence on persistence.

But even with the dependence on Persistent, their still perfectly usable without
storing them in a database.  Their use is just a lot more compelling
in the presence
of a database.

...

>> I agree with your sentiments about namespace pollution.
>> You and I may be the only ones that care though .3 ;).
>>
>
> Yay, actually I care mainly because just trying 'pip install ZODB'
> spreads out n folders in my site-packages, and 'pip uninstall ZODB' leaves
> n-1
> to pick the names by hand. That's why I want things nicely grouped ;-)



Maybe you should use virtualenv or buildout so as to leave your site-packages
alone.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] make ZODB as small and compact as expected

[Jim Fulton]

On Mon, Jul 22, 2013 at 9:15 AM, Stephan Richter
 wrote:
> On Sunday, July 21, 2013 06:12:34 AM Christian Tismer wrote:
...
>>  ZConfig
>
> In my opinion this is a relic from the times before configparser existed.

IMO, ZConfig is very useful in some specific cases, especially ZODB and logging
configuration.

> It is
> also used by other projects outside of ZODB.
>
>>  ZEO
>
> This is separate for historical reasons. I agree it could be merged into the
> ZODB project these days.

It was separate a long time ago.  It's been part of the ZODB distribution
for a long time until recently.

It makes sense for it to be optional, as it's of no interest to people who
use relstorage.

More importantly, it's more complicated than any other part of ZODB and it
makes a lot of sense for ZODB development to be unburdened of it.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] make ZODB as small and compact as expected

[Stephan Richter]

On Sunday, July 21, 2013 06:12:34 AM Christian Tismer wrote:
>  BTrees

I agree, this could be part of ZODB and it would be fine.

>  persistent
>  transaction
>  zc.lockfile
>  zdaemon
>  zope.interface

These are all very useful outside the context of ZODB and I use them without 
it.

>  zc.zlibstorage

This is an add-on that I do not necessarily use, since I do not deal with 
large amounts of data.

>  ZConfig

In my opinion this is a relic from the times before configparser existed. It is 
also used by other projects outside of ZODB.

>  ZEO

This is separate for historical reasons. I agree it could be merged into the 
ZODB project these days.

>  ZODB3   (zlibstorage)

Well, this package is deprecated. It is available for backward-compatibility.

Regards,
Stephan
-- 
Entrepreneur and Software Geek
Google me. "Zope Stephan Richter"
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] BTrees and ZODB simplicity

[Stephan Richter]

On Monday, July 22, 2013 02:11:04 PM Christian Tismer wrote:
> BTrees would make more sense as a standalone package if the persistence
> model were pluggable. But that is also theoretical because I don't see
> right now how to split that further with all the C code.

I agree with this sentiment. When I wrote mongopersist, which is a MongoDB 
store (*not* ZODB storage!) on top of persistent and transaction, BTrees were 
completely useless, because MongoDB provides its own efficient mapping type.

Regards,
Stephan
-- 
Entrepreneur and Software Geek
Google me. "Zope Stephan Richter"
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] BTrees and ZODB simplicity

[Christian Tismer]

On 22.07.13 13:13, Jim Fulton wrote:

On Sat, Jul 20, 2013 at 11:43 PM, Christian Tismer  wrote:

Third rant, dear Zope-Friends (and I mean it as friends!).

In an attempt to make the ZODB a small, independant package, ZODB
has been split into many modules.

Maybe not as many as you think:
persistent, transaction, ZEO, ZODB and BTrees.

5 


I appreciate that, while I think it partially has the opposite effect:

- splitting BTrees apart is a good idea per se.
But the way as it is, it adds more Namespace-pollution than benefits:

To make sense of BTrees, you need the ZODB, and only the ZODB!
So, why should then BTrees be a top-level module at all?

This does not feel natural, but eavesdropping, pretending as something
that is untrue.

I think:

  - BTrees should either be a ZODB sub-package in its current state,

  - or a real stand-alone package with some way of adding persistence as
an option.

I don't agree that because a package depends on ZODB
it should be in ZODB.  There are lots of packages that depend
on ZODB.


This is generally true. In the case of BTrees, I think the ZODB
is nothing without BTrees, and BTrees make no sense without
a storage and carry those _p_ which are not optional.

BTrees would make more sense as a standalone package if the persistence
model were pluggable. But that is also theoretical because I don't see
right now how to split that further with all the C code.

That made me think it belongs to ZODB, what else could it support,
and who would ever install ZODB without it.


I agree with your sentiments about namespace pollution.
You and I may be the only ones that care though .3 ;).



Yay, actually I care mainly because just trying 'pip install ZODB'
spreads out n folders in my site-packages, and 'pip uninstall ZODB' 
leaves n-1

to pick the names by hand. That's why I want things nicely grouped ;-)

cheers - chris

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] zc.zlibstorage missing from zodb package

[Jim Fulton]

On Sat, Jul 20, 2013 at 11:09 PM, Christian Tismer  wrote:
> Hi friends,
>
> I'm trying to work with ZODB. (!)

Cool.

>
> Coming from durus development since a couple of weeks, I am
> spoiled by simplicity.
>
> Actually, I'm annoyed by durus' incapability to accept patches,
> so I'm considering to put my efforts into ZODB.
>
> On the other hand, ZODB tries to become small and non-intrusive,
> but looking at its imports, this is still not a small package, and I'm
> annoyed of this package as well.
>
> - missing
>
>the zc.zlibstorage module is missing, IMHO.

I don't understand this statement.

>besides that, zc.zlibstorage was not maintained since quite a while
>and imports ZOPE3.

It's still maintained, but hasn't required maintenance in some
time.

This is nonsensical.  It depends on ZODB and zope.interface
(and zope.testing and manuel for tests).

...

> - discussion
>
>zc.zlibstorage requites a wrapper to add it to filestorage.
>I consider this an option, instead, and a simple boolean flag to switch
>it on and off.
>The module is way too simple to add all this config extra complication
>to even think of it.

The layered storage architecture made it very easy and low risk
to add this capability.  Further, some have suggested that we
should use different compression schemes.  Making this pluggable
makes it more flexible.

Having said that though, I agree that compression is something
people almost always want, and I can understand your desire to
make it simpler.

> - proposal:
>let me integrate that with ZODB and add a config option, instead of
>a wrapper.

I don't know what you mean by "integrate".  I suggest, if you want
to make it simpler is to provide new ZConfig tags or Python factories
that make configuration simpler the way you'd like, but that do so
by assembling layers under the hood.

...

> Meant in a friendly, collaborative sense -- Chris

Much appreciated.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] make ZODB as small and compact as expected

[Jim Fulton]

On Sun, Jul 21, 2013 at 12:12 AM, Christian Tismer  wrote:
> This is my last emission for tonight.
>
> I would be using ZODB as a nice little package if it was one.
>
> There should be nothing else but
>
> ZODB.
>
> Instead, there is
>
> BTrees
> persistent
> transaction
> zc.lockfile
> zc.zlibstorage
> ZConfig
> zdaemon
> ZEO
> ZODB
> ZODB3   (zlibstorage)
> zope.interface
>
> and what I might have forgotton.
>
> Exception:
> There is also
> zodbpickle
> which I think is very usefull and general-purpose, and I wan to keep it,
> also I will try to push it into standard CPython.
>
> So, while all the packages are not really large, there are too many
> namespaces
> touched, and things like "Zope Enterprize Objects" are not meant to be here
> as open source pretending modules which the user never asked for.

Despite it's tech-bubblishish acronym expansion, which
few people are aware of, ZEO is the standard client-server
component of ZODB, is widely used, and is certainly open source.


> I think these things could be re-packed into a common namespace
> and be made simpler.

If ZODB had been born much later, it would certainly have used
a namespace package.  Now, it would be fairly disruptive to change
it.

> Even zope.interface could be removed from
> this intended-to-be user-friendly simple package.

I don't understand what you're saying.  It's a dependency
if ZODB.

> So while the amount of code is astonishingly small, the amount of
> abstraction layering tells the reader that this was never really meant to
> be small.
>
> And this makes average, simple-minded users like me shy away and go
> back to simpler modules like Durus.
>
> But the latter has serious other pitfalls, which made me want to re-package
> ZODB into something small, pretty, tool-ish, versatile thing for the pocket.
>
> Actually I'm trying to re-map ZOPE to the simplistic Durus interface,
> without its short-comings and lack of support.
> I think a successfully down-scaled, isolated package with ZODB's
> great implementation, but a more user-oriented interface would
> help ZODB a lot to get widely accepted and incorporated into very
> many projects.
> Right now people are just too much concerned of implicit complication which
> actually does not exist.
>
> I volunteer to start such a project. Proposing the name "david", as opposed
> to "goliath".

ZODB is an old project that has accumulated some cruft over the years,
however:

- I've tried to simplify it and, with the exception of ZEO,
  I think it's pretty straightforward.

- ZODB is used by a lot of people with varying
  needs and tastes.  The fact that it is pretty modular has
  allowed a lot of useful customizations.

- I'm pretty happy with the layered storage architecture.

- With modern package installation tools like buildout and pip,
  having lots of dependencies shouldn't be a problem.
  ZODB uses lots of packages that have uses outside of ZODB.
  I consider this a strength, not a weakness.

  Honestly, I have no interest in catering to users who don't use
  buildout, or pip, or easy_install.

- The biggest thing ZODB needs right now is documentation.
  Unfortunately, this isn't easy. There is zodb.org,
  but much better documentation is needed.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] zc.zlibstorage missing from zodb package

[Christian Tismer]

On 22.07.13 11:54, Adam GROSZER wrote:

On 07/21/2013 05:09 AM, Christian Tismer wrote:


- discussion

zc.zlibstorage requites a wrapper to add it to filestorage.
I consider this an option, instead, and a simple boolean flag to 
switch

it on and off.
The module is way too simple to add all this config extra 
complication

to even think of it.


IMHO the wrapper architecture is a good thing, you can do some handy 
things as:


https://pypi.python.org/pypi/cipher.encryptingstorage



I agree (and congrats for that module!).

Just the compression felt so little (because I come from Durus).
My other comment holds:
Why is zlibstorage not on zopefoundation?

cheers - chris

--
Christian Tismer :^)   
Software Consulting  : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/
14482 Potsdam: PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04   9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
  whom do you want to sponsor today?   http://www.stackless.com/

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] BTrees and ZODB simplicity

[Jim Fulton]

On Sat, Jul 20, 2013 at 11:43 PM, Christian Tismer  wrote:
> Third rant, dear Zope-Friends (and I mean it as friends!).
>
> In an attempt to make the ZODB a small, independant package, ZODB
> has been split into many modules.

Maybe not as many as you think:
persistent, transaction, ZEO, ZODB and BTrees.

5 

>
> I appreciate that, while I think it partially has the opposite effect:
>
> - splitting BTrees apart is a good idea per se.
>But the way as it is, it adds more Namespace-pollution than benefits:
>
>To make sense of BTrees, you need the ZODB, and only the ZODB!
>So, why should then BTrees be a top-level module at all?
>
>This does not feel natural, but eavesdropping, pretending as something
>that is untrue.
>
> I think:
>
>  - BTrees should either be a ZODB sub-package in its current state,
>
>  - or a real stand-alone package with some way of adding persistence as
>an option.

I don't agree that because a package depends on ZODB
it should be in ZODB.  There are lots of packages that depend
on ZODB.

I agree with your sentiments about namespace pollution.
You and I may be the only ones that care though .3 ;).

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] BTrees package problems

[Jim Fulton]

On Sat, Jul 20, 2013 at 11:27 PM, Christian Tismer  wrote:
> The BTrees package is an attempt to isolate certain things from ZODB.
>
> While I appreciate the general intent, I cannot see the advantage at
> this point:
>
> - BTrees can be imported alone, yes. But it has the extensions prepared
>with special ZODB slots, which makes this very questionable.
>
> - BTrees furthermore claims the BTrees global bame for it, all though it
>is not a general BTree package, but for ZODB BTrees, only.

Yeah, I worried about this when we broke it out.

OTOH, there isn't much concern with namespace
pollution in the Python community. :/

> - BTrees has a serious bug, see the following example:
>
>> >>> from BTrees import OOBTree as BT
>> >>> t = BT.BTree()
>> >>> for num in range(100):
>> ...   k = str(num)
>> ...   t[k] = k
>> ...
>> >>> t._firstbucket._next = None
>> >>> len(t)
>> Bus error: 10
>> (tmp)minimax:doc tismer$

Ouch.

>
> So there is either an omission to make t._next() read-only, or a check
> of its validity is missing.

Yup.  OTOH, you're the first person to encounter this
after many years, so while this is bad, and needs to be
fixed, I'm not sure how serious it is as a practical matter.

> Actually, I would like to add a callable-check instead, to allow for more
> flexible derivatives.

I don't understand this.

>
> * this was my second little rant about ZODB. Not finished as it seems.
>
> please, see this again as my kraut way of showing interest in improving
> very good things.

:)

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] zc.zlibstorage missing from zodb package

[Adam GROSZER]

On 07/21/2013 05:09 AM, Christian Tismer wrote:


- discussion

zc.zlibstorage requites a wrapper to add it to filestorage.
I consider this an option, instead, and a simple boolean flag to switch
it on and off.
The module is way too simple to add all this config extra complication
to even think of it.


IMHO the wrapper architecture is a good thing, you can do some handy 
things as:


https://pypi.python.org/pypi/cipher.encryptingstorage

--
Best regards,
 Adam GROSZER
--
Quote of the day:
It's not whether you win or lose, it's how good you look playing!  - 
David Lee Roth

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] slow page requests using relstorage with mysql

[Dylan Jay]

Hi,

I'm not sure if this is a bug or a misconfiguration but I'm getting some really 
slow page generation times. I profiled it and a lot of time is spent in 
rollback which seemed odd.

 Ordered by: internal time
   List reduced from 2017 to 100 due to restriction <100>

Function
   was called by...

   ncalls  tottime  cumtime
{method 'rollback' of '_mysql.connection' objects}  
   <- 1483.6423.642  connmanager.py:89(restart_load)

  1473.6133.613  storage.py:230(_rollback_load_connection)
transform.py:149(transformIterable) 
   <-  406.4706.478  transformer.py:24(__call__)
{method 'query' of '_mysql.connection' objects} 
   <- 1483.6613.661  cursors.py:277(_do_query)
...

The page itself is just a normal page, no writes involved. The mysql server is 
replicated master-slave.



---
Dylan Jay
Technical Solutions Manager
PretaWeb: Multisite Performance Support
P: +612 80819071 | M: +61421477460 | twitter.com/pretaweb | 
linkedin.com/in/djay75



___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev