Re: [ZODB-Dev] BTrees package problems
On Tue, Jul 23, 2013 at 9:12 PM, Christian Tismer tis...@stackless.com wrote: ... What I'm after is a way to over-ride the implementation by user code. I did not yet check it this is implemented already, in the Python way of sub-classing built-ins. BTrees (somewhet to mysurprise, do support subclassing, although you'll need to write custom __getstate__ and __setstate__ methods to handle both the BTree data and instance data. It would be better if the methods provided by the BTree classes handled this automatically (by checking for an instance dictionary or slots). Also, the BTree implementation isn't informed by user provided attributes. It would be better if it was. For example. I'd like bucket and internal node sizes to be controllable via instance attributes (typically defined in a subclass. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTrees package problems
On Mon, Jul 22, 2013 at 9:06 PM, Christian Tismer tis...@stackless.com wrote: ... Actually, I would like to add a callable-check instead, to allow for more flexible derivatives. I don't understand this. Simple: I am writing BTree forests for versioned, read-only databases. For that, I need a way to create a version of Bucket that allows to override the _next field by maybe a callable. Otherwise all the buckets are chained together and I have no way to let frozen BTrees share buckets. In retrospect, it might make more sense to do the chaining a level up. Buckets themselves don't care about chaining. The tree wants buckets to be chained to support iteration. I'm not really sure if that helps your use case. When I played with the structure, I was happy/astonished to see the _next field being writable and thought it was intended to be so. It was not, in the end ;-) It's clearly a bug. The code has a comment right above the attribute definition stating that it's (supposed to be) read only, but the implementation makes them writable. There doesn't seem to be anything that depends on writing this attribute. I verified this by adding a fix and running the tests (in 3.10). For what you're trying to do, I suspect you want to fork BTrees, or start over. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTrees package problems
Hey Jim, On 23.07.13 19:18, Jim Fulton wrote: On Mon, Jul 22, 2013 at 9:06 PM, Christian Tismer tis...@stackless.com wrote: ... Actually, I would like to add a callable-check instead, to allow for more flexible derivatives. I don't understand this. Simple: I am writing BTree forests for versioned, read-only databases. For that, I need a way to create a version of Bucket that allows to override the _next field by maybe a callable. Otherwise all the buckets are chained together and I have no way to let frozen BTrees share buckets. In retrospect, it might make more sense to do the chaining a level up. Buckets themselves don't care about chaining. The tree wants buckets to be chained to support iteration. I'm not really sure if that helps your use case. Yes I know. I was thinking of a minimal-intrusive, minimal-overhead way to get it without forking/re-writing, but I'm not settled, yet. When I played with the structure, I was happy/astonished to see the _next field being writable and thought it was intended to be so. It was not, in the end ;-) It's clearly a bug. The code has a comment right above the attribute definition stating that it's (supposed to be) read only, but the implementation makes them writable. There doesn't seem to be anything that depends on writing this attribute. I verified this by adding a fix and running the tests (in 3.10). I know that it is a serious bug (by definition, since it causes a bus error) but it also is not an urgent bug (because it needed me to find it at all). Actually, I have a tendency to find them; the first time I look intensively into a project that I like, this happens almost all the time. Do you know Mr. Adrian Monk from that wonderful Monk series? On software development, it seems to be just me. :: It's a gift, and a curse. For what you're trying to do, I suspect you want to fork BTrees, or start over. Starting over/forking is the easy but heavy way. Before I do that, I will analyze everything and find out if it makes more sense to share the existing code, which is (after my intense investigation and analysis) very good and a highly optimized implementation. It is my goal to - either add to this quasi-perfect thing in a way that the overhead cycles are below 1.8 percent, or - find out that the benefit of a patched solution is too low to justify a patch and do a re-write fork. What I'm after is a way to over-ride the implementation by user code. I did not yet check it this is implemented already, in the Python way of sub-classing built-ins. -- Christian Tismer :^) mailto:tis...@stackless.com Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/ 14482 Potsdam: PGP key - http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTrees package problems
On Sat, Jul 20, 2013 at 11:27 PM, Christian Tismer tis...@stackless.com wrote: The BTrees package is an attempt to isolate certain things from ZODB. While I appreciate the general intent, I cannot see the advantage at this point: - BTrees can be imported alone, yes. But it has the extensions prepared with special ZODB slots, which makes this very questionable. - BTrees furthermore claims the BTrees global bame for it, all though it is not a general BTree package, but for ZODB BTrees, only. Yeah, I worried about this when we broke it out. OTOH, there isn't much concern with namespace pollution in the Python community. :/ - BTrees has a serious bug, see the following example: from BTrees import OOBTree as BT t = BT.BTree() for num in range(100): ... k = str(num) ... t[k] = k ... t._firstbucket._next = None len(t) Bus error: 10 (tmp)minimax:doc tismer$ Ouch. So there is either an omission to make t._next() read-only, or a check of its validity is missing. Yup. OTOH, you're the first person to encounter this after many years, so while this is bad, and needs to be fixed, I'm not sure how serious it is as a practical matter. Actually, I would like to add a callable-check instead, to allow for more flexible derivatives. I don't understand this. * this was my second little rant about ZODB. Not finished as it seems. please, see this again as my kraut way of showing interest in improving very good things. :) Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTrees package problems
On Jul 20, 2013, at 11:27 PM, Christian Tismer tis...@stackless.com wrote: - BTrees has a serious bug, see the following example: from BTrees import OOBTree as BT t = BT.BTree() for num in range(100): ... k = str(num) ... t[k] = k ... t._firstbucket._next = None len(t) Bus error: 10 (tmp)minimax:doc tismer$ So there is either an omission to make t._next() read-only, or a check of its validity is missing. Maybe you could open an issue on Github? ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTrees package problems
On 22.07.13 16:38, Patrick Strawderman wrote: On Jul 20, 2013, at 11:27 PM, Christian Tismer tis...@stackless.com wrote: - BTrees has a serious bug, see the following example: from BTrees import OOBTree as BT t = BT.BTree() for num in range(100): ... k = str(num) ... t[k] = k ... t._firstbucket._next = None len(t) Bus error: 10 (tmp)minimax:doc tismer$ So there is either an omission to make t._next() read-only, or a check of its validity is missing. Maybe you could open an issue on Github? Yes I can do that (and fix it). I was just telling it here because I'd like to know how it is meant to be. - should the attributes be exposed at all? (I guess yes) - are they meant to be writable? (probably not, although that is handy :) I would actually like to be able to derive from Bucket and implement copy-on-write semantics for FrozenBTree (not yet existing) without re-coding much in C, this was the reason while I played around here. For that purpose (sharing buckets) I need a way to make the _next pointers indirect. cheers - chris -- Christian Tismer :^) mailto:tis...@stackless.com Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/ 14482 Potsdam: PGP key - http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTrees package problems
On 22.07.13 13:08, Jim Fulton wrote: On Sat, Jul 20, 2013 at 11:27 PM, Christian Tismer tis...@stackless.com wrote: The BTrees package is an attempt to isolate certain things from ZODB. While I appreciate the general intent, I cannot see the advantage at this point: - BTrees can be imported alone, yes. But it has the extensions prepared with special ZODB slots, which makes this very questionable. - BTrees furthermore claims the BTrees global bame for it, all though it is not a general BTree package, but for ZODB BTrees, only. Yeah, I worried about this when we broke it out. OTOH, there isn't much concern with namespace pollution in the Python community. :/ - BTrees has a serious bug, see the following example: from BTrees import OOBTree as BT t = BT.BTree() for num in range(100): ... k = str(num) ... t[k] = k ... t._firstbucket._next = None len(t) Bus error: 10 (tmp)minimax:doc tismer$ Ouch. So there is either an omission to make t._next() read-only, or a check of its validity is missing. Yup. OTOH, you're the first person to encounter this after many years, so while this is bad, and needs to be fixed, I'm not sure how serious it is as a practical matter. Actually, I would like to add a callable-check instead, to allow for more flexible derivatives. I don't understand this. Simple: I am writing BTree forests for versioned, read-only databases. For that, I need a way to create a version of Bucket that allows to override the _next field by maybe a callable. Otherwise all the buckets are chained together and I have no way to let frozen BTrees share buckets. When I played with the structure, I was happy/astonished to see the _next field being writable and thought it was intended to be so. It was not, in the end ;-) cheers - Chris -- Christian Tismer :^) mailto:tis...@stackless.com Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/ 14482 Potsdam: PGP key - http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] BTrees package problems
The BTrees package is an attempt to isolate certain things from ZODB. While I appreciate the general intent, I cannot see the advantage at this point: - BTrees can be imported alone, yes. But it has the extensions prepared with special ZODB slots, which makes this very questionable. - BTrees furthermore claims the BTrees global bame for it, all though it is not a general BTree package, but for ZODB BTrees, only. - BTrees has a serious bug, see the following example: from BTrees import OOBTree as BT t = BT.BTree() for num in range(100): ... k = str(num) ... t[k] = k ... t._firstbucket._next = None len(t) Bus error: 10 (tmp)minimax:doc tismer$ So there is either an omission to make t._next() read-only, or a check of its validity is missing. Actually, I would like to add a callable-check instead, to allow for more flexible derivatives. * this was my second little rant about ZODB. Not finished as it seems. please, see this again as my kraut way of showing interest in improving very good things. cheers -- chris -- Christian Tismer :^) mailto:tis...@stackless.com Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 :*Starship* http://starship.python.net/ 14482 Potsdam: PGP key - http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
