Re: [ZODB-Dev] PyTables Storage

2008-11-01 Thread David Pratt
Hi guys.

repoze has package is useful for solving  the transaction management  
issue.

http://svn.repoze.org/repoze.tm2

Regards,
David



On Nov 1, 2008, at 4:39 AM, Andreas Jung wrote:

> On 31.10.2008 22:17 Uhr, Christophe Combelles wrote:
>> Tim Cook a écrit :
>>> Does anyone know of any work being done on a PyTables
>>> http://www.pytables.org storage?
>>
>> No, but I might be interested in such a thing, since I've created  
>> last year a
>> medical data analysis software on which I encountered such memory  
>> issues, while
>> looping over large structures in zodb.
>> I have looked at pytables a few times, but I've never tried it  
>> yet. (I had also
>> looked at SOOM (a component of NetEpi Analysis), which allow to  
>> compute stats on
>> very large datasets.)
>> Is it worth creating a new storage implementation for such cases?  
>> Don't just we
>> need to implement (or use an existing) specific structure in  
>> filestorage?
>>
>> Christophe
>>
>>> I see a future need in my project to be able to manage real time  
>>> storage
>>> of medical data and the author of PyTables lists the memory usage  
>>> of the
>>> ZODB as being a drawback in these situations.
>
> The problem with all such implementations is: they don't support  
> transactions which is a major requirement. I also looked at 10gen  
> lately (something similar like GAE but open-source) and they also  
> provide some nice storage (of course "the cloud") but also lack  
> transaction integration.
>
> Andreas___
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev@zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Amazon SimpleDB Adapter

2008-10-13 Thread David Pratt
Hi guys, I had thought of something along these lines a while back  
for s3 and discovered laurence had already started something - but it  
was too slow which I can understand.  Can't really see things being  
much better with simpledb and I think cost would not be great when  
you start thinking of paying for all the pickles.

I think a better way of interacting with databases these days is to  
use models in a direct way rather than bend the storage to the zodb  
api. Martijn and Laurence have been doing this with megrok.rdb,  
megrok.rdf for illustration. The work to tie transactions together  
with zope is really the important stuff.

There is no lack of storage options out there these days. My opinion  
is that if you want the true characteristics of the storage, you  
won't make it behave like a zodb. Lovely systems came to a similar  
conclusion after trying relstorage and then moving to use storm in a  
more direct way.

Regards,
David


On Oct 12, 2008, at 12:50 AM, Shane Hathaway wrote:

> Laurence Rowe wrote:
>> I'm not sure RelStorage is the best place for it - SimpleDB is very
>> different to relational databases.
>
> RelStorage doesn't use much of a relational database either (except
> during packing).
>
>> A couple of years ago I experimented with s3storage [1]. This  
>> turned out
>> to be very slow due to the number of writes performed every  
>> transaction
>> - one per object, though this could be improved if the writes were
>> parallelized. It reached the point where zope2 would start up.  
>> This took
>> about 10 or 15 mintutes at the time (I did not have access to EC2  
>> at the
>> time and this was over public wifi).
>>
>> It worked by creating it's own indexes in S3. I don't think SimpleDB
>> will give any advantage unless it is shown to be faster to query than
>> S3. You cannot store pickles directly in SimpleDB because it is  
>> limited
>> to an attribute size of 1024 bytes.
>>
>> The challenge in building such a system is in Amazon's eventual
>> consistency model means you cannot know how up to date your view  
>> of the
>> data is. I think it could make a great backend for storing pickles
>> (keyed by oid, tid) but it is probably much easier to have a separate
>> index to consult during loadSerial.
>
> Thanks for the background on S3 and SimpleDB.  Using Amazon's storage
> services as a ZODB backend is sounding ever more like an interesting
> challenge.
>
> Shane
>
> ___
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev@zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] ZODB 3.8.1b2 available and persistent caches

2008-05-13 Thread David Pratt
Hi Jim. Any chance of incorporating Shane's patch for relstorage for 
this release? Many thanks.


Regards,
David

Jim Fulton wrote:


Lately, we've started to use persistent caches. As we've done so, we've 
found and fixed a number of problems.  I'm planning to make a 3.8.1 
release soon.


Jim

--
Jim Fulton
Zope Corporation


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] RelStorage and BLOBs

2008-05-07 Thread David Pratt
Hi Christian. Curious, what were you using for postgres replication? 
Many thanks.


David

Christian Theune wrote:



I'm kind of curious what a match of RelStorage and ZEORaid may give. FWIW my
experiences with Postgresql replication aren't that good so you could still go
that route using two RelStorages with one (or more) ZEORaids in front.

Christian


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: Zeo Server as a single point of failure

2008-03-12 Thread David Pratt
Hi Shane. Not sure if you have been following the whole thread or not. 
In any case, I had committed to maintaining DirectoryStorage a little 
while ago and it will be releasing in the near future under a new 
license.  I think it is a good project too and I am excited about 
bringing it up to date :-). Once I have finished with the packaging, a 
bit of clean up and a first commit done of the updated source to 
sourceforge, I am hoping you may be interested in participating in ts 
development and co-maintaining it with me.


If so, I can provide you with access once the new code has landed in the 
repository. I am happy that there is genuine interest for this storage 
though it is not widely deployed. I have a continued interest in 
RelStorage (formerly PGStorage). Though I cannot commit to svn, I'd be 
happy to help with a buildout configuration that could apply the ZODB 
patch also. Had you though about putting into the z3c namespace. Many 
thanks.


Regards,
David

Shane Hathaway wrote:

Tres Seaver wrote:

What does Shane have to do with DirectoryStorage?


For the record: I am not at all associated with DirectoryStorage, other 
than I think it's a good project.


Shane

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: Zeo Server as a single point of failure

2008-03-12 Thread David Pratt

Tres Seaver wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

David Pratt wrote:
Hi Alan. I'm keeping the thread on the list since you raised the issue 
here. There are several repositories of zope related code that exist 
outside of svn.zope.org. I have made the decision to host 
DirectoryStorage at sourceforge on the basis that I have committed to 
maintaining and contributing to the project.


I volunteered to maintain the project since I saw it falling into 
disrepair following two years without a single commit to the code base. 
I have become the maintainer on the basis of an agreement with Tobi back 
in January. This agreement permits me to relicense the code under my 
choice of a ZPL or MIT license.


I cannot work with sources in zope repository since it would require a 
contributor agreement with Zope Corp. I am unable to enter into this 
agreement for the forseeable future. The zif collective 
(zif.sourceforge.net) is a way that I may contribute back to the zope 
community without the requirement of the agreement.


You don't identify the problem(s) which prevent you from entering into
that agreement:  we might be able to clarify or help resolve them if you
did so.


Hi Tres. Any discussion of the ZCA will have to occur another day. I am 
not seeking a way to enter the agreement or to resolve anything 
publicly. Toby and I made these arrangements a couple months back and 
have committed to ongoing communication.


It should be pointed out that users of the code will be getting releases 
from a package index in future. More important than the location of the 
repository is the fact the code will be eggified, maintained and there 
will be future releases.


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: Zeo Server as a single point of failure

2008-03-11 Thread David Pratt
Hi Martijn. Probably not at this point. Unfortunately, the assignment of 
IP and contribution to open source are not synonymous. Perhaps when the 
relationship between ZC and ZF is clearer it could be reconsidered in 
future.


Regards,

David

Martijn Faassen wrote:

Hey,

David Pratt wrote:
[snip]
I cannot work with sources in zope repository since it would require a 
contributor agreement with Zope Corp. I am unable to enter into this 
agreement for the forseeable future. The zif collective 
(zif.sourceforge.net) is a way that I may contribute back to the zope 
community without the requirement of the agreement.


Just out of interest, would a contributor agreement with the Zope 
Foundation be any better for you?


Regards,

Martijn

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Zeo Server as a single point of failure

2008-03-11 Thread David Pratt
Hi Alan. I'm keeping the thread on the list since you raised the issue 
here. There are several repositories of zope related code that exist 
outside of svn.zope.org. I have made the decision to host 
DirectoryStorage at sourceforge on the basis that I have committed to 
maintaining and contributing to the project.


I volunteered to maintain the project since I saw it falling into 
disrepair following two years without a single commit to the code base. 
I have become the maintainer on the basis of an agreement with Tobi back 
in January. This agreement permits me to relicense the code under my 
choice of a ZPL or MIT license.


I cannot work with sources in zope repository since it would require a 
contributor agreement with Zope Corp. I am unable to enter into this 
agreement for the forseeable future. The zif collective 
(zif.sourceforge.net) is a way that I may contribute back to the zope 
community without the requirement of the agreement.


Anyone interested in DirectoryStorage is encouraged to participate in 
its development since it is an open project that will be getting a more 
liberal license. There will be something more to build upon once the egg 
repackaging is released. It will be released under zif namespaced package.


Shane and I have a relationship since I was also a project admin for 
PGStorage when it was located on sourceforge. There is nothing that 
would prohibit our communication or collaboration on any future 
development of DirectoryStorage. HTH.


Regards,
David


Alan Runyan wrote:
> i understand.  I wanted to see if I could rile the zodb community
> into saying "we want to maintain it".  as in the 'collective' we.
>
> I respect your rights.  But if I could get Shane to say "make it ZPL
> and I will maintain it" -- it would be well worth while to see if
> I could make this happen
>
> alan
>
> On Tue, Mar 11, 2008 at 8:16 AM, David Pratt <[EMAIL PROTECTED]> 
wrote:

>> Hi Alan. I am working on cleanup and will releasing a version with new
>>  licensing at some point in near future. I am working at this with bits
>>  of time I have between other work. Unfortunately, it had been left 
for a
>>  couple of years without attention and my interest is in keeping it 
up to
>>  date with ZODB development. We discussed this about month or so ago 
when

>>  I communicated I would be maintaining it at sourceforge for the reasons
>>  I provided at that time.
>>
>>  Regards,
>>  David
>>
>>
>>
>>
>>  Alan Runyan wrote:
>>  >>  in that vein, DirectoryStorage-behind-ZEO has worked perfectly for
>>  >>  years...  but this new stuff is very interesting!
>>  >
>>  > wishing we could get DirectoryStorage into svn.zope.org
>>  >
>>  > ~alan
>>
>>
>>> ___
>>  > For more information about ZODB, see the ZODB Wiki:
>>  > http://www.zope.org/Wikis/ZODB/
>>  >
>>  > ZODB-Dev mailing list  -  ZODB-Dev@zope.org
>>  > http://mail.zope.org/mailman/listinfo/zodb-dev
>>  >
>>
>
>
>
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Zeo Server as a single point of failure

2008-03-11 Thread David Pratt
Hi Alan. I am working on cleanup and will releasing a version with new 
licensing at some point in near future. I am working at this with bits 
of time I have between other work. Unfortunately, it had been left for a 
couple of years without attention and my interest is in keeping it up to 
date with ZODB development. We discussed this about month or so ago when 
I communicated I would be maintaining it at sourceforge for the reasons 
I provided at that time.


Regards,
David


Alan Runyan wrote:

 in that vein, DirectoryStorage-behind-ZEO has worked perfectly for
 years...  but this new stuff is very interesting!


wishing we could get DirectoryStorage into svn.zope.org

~alan
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread David Pratt

Hi Benji. Have you any settings to recommend or use a default. Many thanks.

Regards,
David

Benji York wrote:

Roché Compaan wrote:

On Tue, 2008-03-04 at 13:27 -0700, Shane Hathaway wrote:
Maybe if you set up a ZODB cache that allows just over 10 million 
objects, the lookup time will drop to microseconds.  You might need a 
lot of RAM to do that, though.


Maybe, but somehow I think that disk IO will prevent this. I'll check.


If you're on Linux, you can tweak swappiness (/proc/sys/vm/swappiness; 
http://lwn.net/Articles/83588/) to affect how much RAM is used for the 
page cache and how much for your process.

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZODB Benchmarks

2008-03-05 Thread David Pratt
Hi Roche. I figured this out once and it was included in PGStorage so it 
should be in relstorage also. Take a look at get_db_size method in 
postgres adapter. relstorage is in the zope repository.


Regards,
David

Roché Compaan wrote:



- How much disk space does each database consume when there are 10M objects?


ZODB: 19GB

How do you check the size of a Postgres database?

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Speedy RelStorage/PostgreSQL

2008-01-31 Thread David Pratt

Ok, great. Woo hoo - 64 bit 2.5. Many thanks Shane.

Regards,
David

Shane Hathaway wrote:

David Pratt wrote:
Hi Shane. Congratulations. Can you advise of compatibility with ZODB 
3.8. Many thanks.


Yes, it's fully compatible now.  A few tests didn't pass at first, but 
they only failed due to the lack of some stub methods.  After I added 
those methods and created a patch that applies cleanly to ZODB 3.8.0, 
all the tests passed (with 64 bit Python 2.5, in fact).


Shane


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Speedy RelStorage/PostgreSQL

2008-01-30 Thread David Pratt
Hi Shane. Congratulations. Can you advise of compatibility with ZODB 
3.8. Many thanks.


Regards,
David

Stephan Richter wrote:

On Wednesday 30 January 2008, Shane Hathaway wrote:

Kudos to the PostgreSQL team for building such a nice database. :-)


I am saying this since at least 8 years now. :-) But everyone went on the 
MySQL bandwagon.


But the storage is great news! I will certainly keep this in mind for the 
current project.


Regards,
Stephan

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] PGStorage

2008-01-24 Thread David Pratt

Hi Andreas. No need to rely on rumors. The current source is available here:

http://pgstorage.cvs.sourceforge.net/pgstorage/

Regards,
David



Andreas Jung wrote:



--On 24. Januar 2008 15:42:01 +0100 Andreas Jung <[EMAIL PROTECTED]> wrote:




--On 23. Januar 2008 18:17:18 +0100 Andreas Jung <[EMAIL PROTECTED]> wrote:




--On 22. Januar 2008 21:17:45 -0500 Stephan Richter
<[EMAIL PROTECTED]> wrote:


On Tuesday 22 January 2008, Dieter Maurer wrote:

"OracleStorage" was abandoned because it was almost an order
or magnitude slower than "FileStorage".


Actually, Lovely Systems uses PGStorage because it is faster for them.



It would be interesting where PGS is faster than filestorage. Ok, I just
tried made a simple benchmark for testing write performance. I created a
Plone site and then created a copy using copy/paste within the ZMI.
(AMD Dualcore 2.6 GHz, Postgres 7.4.7 running on dedicated DB server):

Copy&Paste using Filestorage: 3-4 seconds
Copy&Paste using PGStorage: 30-40 seconds



Alan asked me to so some further testing. I wrote a small script
creating 100 Documents within one transaction (both in CMF and Plone)
and I calculated the transaction time (not included the overhead
for creating the instances)...the numbers are pretty much the
self-speaking:

   CMFPlone
PGStorage10-12 secs   15-18 secs
Filestorage  0.3-0.4 secs 0.4-0.6 secs




I must mention that all tests were done using PGStorage 0.1 - the only
version available to me. Rumors say that Shane might release improved 
versions. So the numbers must be taken with care as they reflect the 
state of PGStorage as of 2006.


Andreas




___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] PGStorage

2008-01-23 Thread David Pratt

Graphical test comparison (open office format)
http://pgstorage.cvs.sourceforge.net/pgstorage/PGStorage/tests/comparison.ods

Regards,
David

Alan Runyan wrote:

Andreas,

Could you try to mount the catalog separately?  My understanding is
the catalog is what makes storages a misery.

CMF/portal_catalog mounted as Filestorage
and CMF could be mounted as PGStorage.

I presume you would see much more reasonable performance?

cheers


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] PGStorage

2008-01-23 Thread David Pratt

Cool.

Jim Fulton wrote:
Berkeley DB storage didnt' work out the first time around.  My 
experience optimizing packing for FileStorage reminded me how much I 
want an alternative to FileStorage for large active databases.  I intend 
to revisit Berkeley DB storage one of these days.

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] PGStorage

2008-01-22 Thread David Pratt
Yes, Shane had done some benchmarking about a year or so ago. PGStorage 
was actually faster with small writes but slower for larger ones. As far 
as packing, as a zodb implementation, packing is still required to 
reduce the size of data in Postgres. BTW Stephan, where is Lovely using 
it - a site example? I had read some time ago that they were exploring 
it but not that it was being used.


Regards,
David

Stephan Richter wrote:

On Tuesday 22 January 2008, Dieter Maurer wrote:

"OracleStorage" was abandoned because it was almost an order
or magnitude slower than "FileStorage".


Actually, Lovely Systems uses PGStorage because it is faster for them.

Regards,
Stephan

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] some interesting benchmarks

2007-09-14 Thread David Pratt

Cool! Thanks Martijn.

Regards,
David

Martijn Faassen wrote:

Hi there,

Recently there were two blog entries which did some simple benchmarks of 
the ZODB and compared to other databases. Possibly others hadn't noticed 
those yet, so here are the references:


ZODB vs Relational Database: a simple benchmark

http://pyinsci.blogspot.com/2007/09/zodb-vs-relational-database-simple.html

ZODB vs Durus

http://pyinsci.blogspot.com/2007/09/zodb-vs-durus.html

Regards,

Martijn

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] zc.recipe.zeo

2007-08-03 Thread David Pratt
Hi Jim. My apologies. I'll move this thread to the other list. Many 
thanks for your reply and help.


Regards,
David

Jim Fulton wrote:


On Aug 3, 2007, at 10:00 AM, David Pratt wrote:

Hi Fred. Working with the new recipe but there does not seem to be a 
way to use the filestorage recipe together with it. I would like to do 
something like this


[zeoscripts]
recipe = zc.recipe.egg:script
eggs = ZODB3

[database]
recipe = zc.recipe.filestorage

[server]
recipe = zc.zodbrecipes:server
database = database
zeo.conf = 
  address 8105
  monitor-address 8106
  transaction-timeout 300
   

to keep the database contained within the buildout and to produce a 
zeo.conf like



   address 8105
   monitor-address 8106
   transaction-timeout 300


   path /my/buildout/path/database/Data.fs


I'll take a look at the recipe since it should be possible to utilize 
some of what is in zc.recipe.zeo which has this behaviour when 
database is defined as an option, otherwise it would pass to do what 
it is currently doing. I use other backend storages also so would like 
to have  it operate automatic with filestorage recipe or manually 
configured. Many thanks.


Use:

  
 path ${database:path}
  

Jim

--
Jim Fultonmailto:[EMAIL PROTECTED]Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporationhttp://www.zope.comhttp://www.zope.org




___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] zc.recipe.zeo

2007-08-03 Thread David Pratt
Hi Fred. Working with the new recipe but there does not seem to be a way 
to use the filestorage recipe together with it. I would like to do 
something like this


[zeoscripts]
recipe = zc.recipe.egg:script
eggs = ZODB3

[database]
recipe = zc.recipe.filestorage

[server]
recipe = zc.zodbrecipes:server
database = database
zeo.conf = 
  address 8105
  monitor-address 8106
  transaction-timeout 300
   

to keep the database contained within the buildout and to produce a 
zeo.conf like



   address 8105
   monitor-address 8106
   transaction-timeout 300


   path /my/buildout/path/database/Data.fs


I'll take a look at the recipe since it should be possible to utilize 
some of what is in zc.recipe.zeo which has this behaviour when database 
is defined as an option, otherwise it would pass to do what it is 
currently doing. I use other backend storages also so would like to have 
 it operate automatic with filestorage recipe or manually configured. 
Many thanks.


Regards,
David












Fred Drake wrote:

On 8/3/07, David Pratt <[EMAIL PROTECTED]> wrote:

Hi there. zc.recipe.zeo has not yet made it to the cheeseshop. It would
be great if someone could put it there. If you want me to do it that is
fine since I have other packages on cheeseshop. The port for zeo on is
also hardcoded for 8100 but I am wanting to run more than a single ZEO
server instance per physical server so it looks like it needs a fix.


I've been using zc.zodbrecipes:server, including

  
address 9000
  

in the value for the zeo.conf key (replacing 9000 with your chosen port).


Question about thread setting in buildout. In zope2's zope.conf you just
set the no of threads. In zc.zope3recipes, there is no mention of
setting threads. I ask this since setting threads to 1 improved
performance according to the recent high performance zope session by
Lovely systems at europython. Many thanks.


You can include the line "threads 1" as part of the value for
zope.conf in the instance part.


  -Fred


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] zc.recipe.zeo

2007-08-03 Thread David Pratt
Hi Fred. The zc.recipe.zeo is a bit old and I seemed to have missed the 
newer zeo server recipe in svn which handles imports also. Thank you 
also for the info on threads.


Regards,
David

Fred Drake wrote:

On 8/3/07, David Pratt <[EMAIL PROTECTED]> wrote:

Hi there. zc.recipe.zeo has not yet made it to the cheeseshop. It would
be great if someone could put it there. If you want me to do it that is
fine since I have other packages on cheeseshop. The port for zeo on is
also hardcoded for 8100 but I am wanting to run more than a single ZEO
server instance per physical server so it looks like it needs a fix.


I've been using zc.zodbrecipes:server, including

  
address 9000
  

in the value for the zeo.conf key (replacing 9000 with your chosen port).


Question about thread setting in buildout. In zope2's zope.conf you just
set the no of threads. In zc.zope3recipes, there is no mention of
setting threads. I ask this since setting threads to 1 improved
performance according to the recent high performance zope session by
Lovely systems at europython. Many thanks.


You can include the line "threads 1" as part of the value for
zope.conf in the instance part.


  -Fred


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] zc.recipe.zeo

2007-08-02 Thread David Pratt
Hi there. zc.recipe.zeo has not yet made it to the cheeseshop. It would 
be great if someone could put it there. If you want me to do it that is 
fine since I have other packages on cheeseshop. The port for zeo on is 
also hardcoded for 8100 but I am wanting to run more than a single ZEO 
server instance per physical server so it looks like it needs a fix.


Question about thread setting in buildout. In zope2's zope.conf you just 
set the no of threads. In zc.zope3recipes, there is no mention of 
setting threads. I ask this since setting threads to 1 improved 
performance according to the recent high performance zope session by 
Lovely systems at europython. Many thanks.


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] ZEO client poll times

2007-05-11 Thread David Pratt
Hi Jim. I understood that you are opening connection. Looks like I 
misunderstood what will happen when max is reached. :-) So after 
reaching the max it will not disconnect -  just attempt to reconnect at 
the max interval. I guess if this is the case, I don't see an issue with 
the shorter interval between retries. Worst case, it will probably just 
mean some extra lines in the log file before it connects (if it can).


Regards,
David

Jim Fulton wrote:


On May 11, 2007, at 4:39 PM, David Pratt wrote:

6 seconds as max time sounds a bit short to me. I think there can be 
latency sometimes for whatever reason over the Internet. I would not 
want it necessarily closing connections as a result. Maybe max of 30 - 
60 sec might be better - and still not too long to wait. If folks use 
the configuration, it adjustable on the other hand.


The timeout I'm talking about isn't for closing connections, it's for 
opening them. It's the length of time to wait before trying again after 
being disconnected or after failing a connection attempt.


Jim

--
Jim Fultonmailto:[EMAIL PROTECTED]Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporationhttp://www.zope.comhttp://www.zope.org




___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] ZEO client poll times

2007-05-11 Thread David Pratt
Hi Jim. Twisted has a reconnecting client with a similar feature. It 
starts at a short interval also but backs off at increasingly longer 
intervals which I think is the way to go when you are unable to reach 
the other side (but when you are obviously running a server).


6 seconds as max time sounds a bit short to me. I think there can be 
latency sometimes for whatever reason over the Internet. I would not 
want it necessarily closing connections as a result. Maybe max of 30 - 
60 sec might be better - and still not too long to wait. If folks use 
the configuration, it adjustable on the other hand. Many thanks.


Regards,
David

Jim Fulton wrote:


When a ZEO Client is disconnected from a ZEO server, it will try to 
reconnect and keep trying until it reconnects or is closed.  It waits 
between attempts.  The waiting period starts at 5 seconds and increases 
to 5 minutes by default.  These times are configurable using the 
min-disconnect-poll and max-disconnect-poll ZConfig options or with the 
min_disconnect_poll and max_disconnect_poll constructor arguments.  
Since most people don't change these defaults, most people's clients 
take too long to reconnect.  I'd like to change these defaults to 
something much smaller.  I'm thinking 1 and 6 seconds.

Anybody have any objections to this change?

Jim

--
Jim Fultonmailto:[EMAIL PROTECTED]Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporationhttp://www.zope.comhttp://www.zope.org



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Storage Interfaces

2007-04-28 Thread David Pratt
Bottom line is that storage methods need to be executed on both sides. I 
don't think the storages need to care what or how objects are 
transported nor what version of zope you are using so long as in the 
end, the object is one that can be used by a storage method - and that 
the storage methods are compatible with the older zopes (which seems to 
be what you are concerned about). At least this is how I am looking at 
it - whether it is zope 2 or 3, it should not matter if they are methods 
that can act on ZODB and are suited to the zodb version. In any case, 
thats the direction I have been going.


On top of it, who wants a transport that does only one thing, that can 
get tired pretty fast. There are many more possibilities than exchanging 
zodb data using a more generic transport. That's the other key reason I 
tossed out my attempt to replicate ZEO directly using twisted - if you 
want to think a bit beyond ZEO.


Regards,
David

Jim Fulton wrote:


On Apr 28, 2007, at 9:13 AM, David Pratt wrote:
...
As far as compatibility, I am not sure the past always needs to equal 
the future. It is probably the methods for zrpc that are important, 
not the transport - at least that is my thinking.


I can use current ZEO servers with ZODB 3.2 clients running in very old 
versions of Zope (2.6 and 2.7).  This is extremely useful to me at this 
point.  I couldn't do this if I changed the wire protocol.


Jim

--
Jim Fultonmailto:[EMAIL PROTECTED]Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporationhttp://www.zope.comhttp://www.zope.org




___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Storage Interfaces

2007-04-28 Thread David Pratt

Jim Fulton wrote:


On Apr 27, 2007, at 9:30 PM, David Pratt wrote:

Hi Jim. I have been quite interested in this also so have experimented 
and continue on this also. The way I am doing the networking it 
similar to ngi I have put this together before you defined these 
interfaces.


I'm not sure which "this" you're referring to. I assume you mean using 
Twisted for ZEO.  I don't see that that has much to do with the new 
storage interfaces.


Yes, Twisted for ZEO. No, I was referring to the interfaces you were 
setting up - the interfaces to ngi.




In what way is your networking similar to ngi?


The interfaces are similar. The system I am putting together is pretty 
symmetrical and I like the language which is more consistent with 
twisted's use of factories. You were using this language before in your 
initial proposal to ngi but many of the interfaces are now name 
differently named. See how nice and symmetrical this looks :-)


clientconnfactoryhandler
clientconnhandler
clientconnfactory
clientreconnfactory
clientmethods
clientstorage
connection
serverconnfactory
serverconnhandler
serverconnfactoryhandler
servermethods
serverstorage



Note that I wrote ngi based on a misunderstanding of Twisted. :(  I 
didn't realize that Twisted application code didn't touch sockets and 
therefore could be tested without using sockets, threads or 
subprocesses.  IMO, this is Twisted's most important feature.  I 
recently finished my first Twisted application and I was very pleased 
with my ability to create sane tests for it, although it's test support 
infrastructure could use improvement, which I plan to do.


I have got a basic transport to use prospective broker and banana 
protocol. So a client and server but I have not yet put together the 
methods for interacting with the storage.


I plan to stick with the existing ZEO protocol, both for compatibility 
and for performance reasons.


Banana protocol is efficient and extensible. It has its own 
producer/consumer so no need to roll you own system as you have for ZEO. 
 I think you will find banana an efficient serialization of objects.


As far as compatibility, I am not sure the past always needs to equal 
the future. It is probably the methods for zrpc that are important, not 
the transport - at least that is my thinking. This is the obvious 
downside of ZEO at the moment is that is too tied to asyncore. In fact, 
 I started by trying to replicate much of what was in ZEO - then I 
threw it away because something simpler I believe is possible.


If a new system can transport the objects and execute the methods, isn't 
that all it is supposed to do? On top of it, what if something better 
comes around - the methods that you are executing on both sides are the 
important part and the transport should be transparent. I think this 
could produce a much cleaner and understandable system.


On top of it, it doesn't upset anything with current zeo for users that 
currently use this. It could in fact be developed and run in parallel, 
not that you would want to run it that way :-). Another obvious 
advantage is that twisted project has tests for most of this already so 
that is also a big plus.


Out of curiosity, does perspective broker
support one-way calls, messages sent without replies?  Scanning the 
docs, I can't tell.  Of course, not waiting for replies is inherent in 
Twisted's defered model I suppose.


Yes

To set up something to work with I create both a TCP and SSL 
clientfactory so I bring them into the multiservice when zope is 
started. This way you do not need a separate loop to run the service 
which is the current way zeo runs. At the same time this does not 
interfere with the current zope setup. I have no proof one way or the 
other that this will not work as efficiently on a single twisted loop 
with the second asyncore loop. I still have to hook up database methods 
and I had put this on a backburner for a while since I have had other 
things to do.


In any case, the client setup in zope requires only minor modification 
of the zope.app.twisted main.py since the script only deals with servers 
and not clients at the moment. The clients are reconnecting clients so 
will continue to try reconnecting if connection is lost in gradually 
increasing time intervals.


On the server side I have the server working using zconfig and zdaemon 
in a similar way to way it is setup for zope itself. I have been looking 
at the zope3recipes since I am working on getting this into a buildout 
to continue since I am quite happy with the way buildouts work.


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Storage Interfaces

2007-04-27 Thread David Pratt
Hi Jim. I have been quite interested in this also so have experimented 
and continue on this also. The way I am doing the networking it similar 
to ngi I have put this together before you defined these interfaces.


I have got a basic transport to use prospective broker and banana 
protocol. So a client and server but I have not yet put together the 
methods for interacting with the storage. Just opening it at the moment 
and executing a couple of methods on the objects on the other side.


Regards,
David


Jim Fulton wrote:



BTW, an upcoming project of mine will be to change ZEO to use Twisted.  
I suppose that will make it easier to support encrypted ZEO connections.

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: Cleaning up storage interfaces

2007-04-24 Thread David Pratt
Hi. I am going to be doing an egg release for pgstorage at some point in 
the next couple of weeks. The current repository is:


http://pgstorage.cvs.sourceforge.net/pgstorage/PGStorage/

getSerial is not used in the source. Many thanks.

Regards,
David



Tres Seaver wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Christian Theune wrote:


Am Dienstag, den 24.04.2007, 13:16 -0400 schrieb Jim Fulton:
I'm finally trying to clean up the storage interfaces.  I have a  
question.  I'll probably have more later, but I'm going to deal with  
them one by one rather than trying to save them up.


Does anyone know if getSerial is used for anything?  I suspect that  
it is an old name for getTid and would like to remove it in favor of  
getTid.


In particular, it's plumbed through the storage and client servers,  
but I can't see any evidence that people are using it.

Hmm. Looks like it. getTid looks like it's used from ZEO in the
verify() method. I didn't find any place where getSerial was used.
Zope 3 uses neither (and it probably shouldn't anyway.)


DirectoryStorage[1] is probably the major "out-of-tree" storage
implementation to check;  PGStorage[2] is likel a distant second in
terms of usage:

[1] http://dirstorage.sourceforge.net/

[2] http://hathawaymix.org/Software/PGStorage/


Tres.
- --
===
Tres Seaver  +1 540-429-0999  [EMAIL PROTECTED]
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGLtS7+gerLs4ltQ4RAvn7AJ98Wgm7D/EbA3YS/A/gbJ5bKjbR5QCgtuyv
tX4is0ClS1zf1Pa6Oc2RwkE=
=I2S0
-END PGP SIGNATURE-

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] ZODB 3.8 and Blobs

2006-11-29 Thread David Pratt

Many thanks to the team for this awesome effort.

Regards
David

Christian Theune wrote:

Hi,

it's been about 21 months since work on blobs for ZODB started. I
finally sat down today and fixed the last broken test for blobs on
windows and merged the branch to the trunk.

Christian





___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Experimental ZODB 3.7 release

2006-11-23 Thread David Pratt
Hi Christian. I have been anticipating the merge of blobs in the trunk 
for a while so also hoping this happens for this release. Many thanks.


Regards,
David

Christian Theune wrote:

Hi,

from the project management perspective: Does that mean we're on a
feature freeze for 3.7?

I still have to deal with one bug on windows (which is likely just a
problem in the test itself that fails, not in the blob code) until I can
merge into the trunk and I'd be very happy to commit this soon.

Christian

Jim Fulton wrote:

I've released a fairly experimental 3.7 release,
http://www.python.org/pypi/ZODB3/3.7.0b3.

Unlike older ZODB releases, this release only installs the
packages that are part of ZODB.  It doesn't include ZConfig
zdaemon, zope.testing, zope.proxy and zope.interface.  These
are now dependencies that need to be installed separately.
If this release is installed with easy_install or zc.buildout
then the dependencies should be installed automatically.

I haven't modified mkzeoinst to reflect the fact that
the dependencies are no-longer included in the distribution.
As a consequence, the instances built by mkzeoinst will only
work if the dependencies are on the python path (e.g. installed
into site-packages.)  I will need to modify mkzeiinst to
deal with this.

Anyway, if people want to play with it, feedback is welcome.
I'm currently using the release in a project in which I install the
release with zc.buildout.

Jim






___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB

2006-08-21 Thread David Pratt

Hi Christian. Yay :-) Looking forwards to it. Many thanks.

Regards,
David

Christian Theune wrote:

David Pratt wrote:
Can you advise whether blobs will make it the trunk any time soon. I 
see an extfile package now for z3 but would rather see filesystem 
storage dealt with at the backend than in the app itself. Many thanks.


Chris McDonough and I have this on our schedule. It's not much to do and 
I hope to get my todo list done during the next week. After that we 
might be able to merge into trunk during September.


Christian


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB

2006-08-20 Thread David Pratt
Can you advise whether blobs will make it the trunk any time soon. I see 
an extfile package now for z3 but would rather see filesystem storage 
dealt with at the backend than in the app itself. Many thanks.


Regards,
David

Gary Poster wrote:


On Apr 29, 2006, at 6:08 AM, Christian Theune wrote:


Hi everybody,

the ZODB team is proud to announce a technical preview of a Blob
implementation for ZODB.


Hey.

This seems like there's no way that it will make it into the next ZODB 
release, right?


I assume no one is using it in production, on the basis of the branch TODO.

Does anyone have a hard need to get this production-ready (address the 
TODOs) by a certain date?


Gary
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Getting clientstorage object in main.py

2006-06-15 Thread David Pratt

Hi Pascal. Yes, this is exactly what I was after. Many thanks for your help!

Regards
David

Pascal Peregrina wrote:

I think that through the DB instance, you can access the storage instance with the 
"_storage" attribute.

Pascal

-Message d'origine-
De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] De la part de David Pratt
Envoyé : jeudi 15 juin 2006 19:29
À : zodb-dev@zope.org
Objet : Re: [ZODB-Dev] Getting clientstorage object in main.py

Perhaps I should qualify this a bit. On realize ClientStorage is 
initialized during ZEO startup. main.py provides a ZODB.DB.DB instance 
as db but I am interested in setting an attribute of the ClientStorage 
instance that was initialized.


Regards,
David

David Pratt wrote:
Hi. I am needing to get hold of my clientstorage obj instance that was 
created on realize in zope.app.twisted.main.py. Can someone advise how 
to get a handle on it in main.py after is has it has been created 
(realized). Many thanks.


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


**
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Getting clientstorage object in main.py

2006-06-15 Thread David Pratt
Perhaps I should qualify this a bit. On realize ClientStorage is 
initialized during ZEO startup. main.py provides a ZODB.DB.DB instance 
as db but I am interested in setting an attribute of the ClientStorage 
instance that was initialized.


Regards,
David

David Pratt wrote:
Hi. I am needing to get hold of my clientstorage obj instance that was 
created on realize in zope.app.twisted.main.py. Can someone advise how 
to get a handle on it in main.py after is has it has been created 
(realized). Many thanks.


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] Getting clientstorage object in main.py

2006-06-15 Thread David Pratt
Hi. I am needing to get hold of my clientstorage obj instance that was 
created on realize in zope.app.twisted.main.py. Can someone advise how 
to get a handle on it in main.py after is has it has been created 
(realized). Many thanks.


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] ZEO Authentication

2006-05-26 Thread David Pratt
Well, you might need a few monkeys for sure :-) I guess it is safe to 
say, that the authentication built into ZEO is not used extensively. You 
never know until you ask. I guess if it were, it might depend on the use 
case you have for ZEO more than anything. Many thanks.


Regards
David



You can use a VLAN, too.



Or a monkey to control each ZEO packet :-)

-aj




___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] ZEO Authentication

2006-05-26 Thread David Pratt
Hi Jens, this is available in ZEO and looks to me that the original idea 
was to support more than a single mode of authentication. I was curious 
of those using it to see how it was generally being used (or whether 
others have worked out something against other authentication schemes). 
Many thanks.


Regards,
David

Jens Vagelpohl wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1


On 26 May 2006, at 05:46, David Pratt wrote:

I am curious how folks are handling authentication for ZEO in a 
general way (client against the server). Is is common to attempt to 
use user/passwords from acl in zope as a means of creating a lists for 
authenticating against a ZEO server? Or is it more typical to manage a 
separate list of user/passwords just for the purpose of authenticating 
the client server connection? I'd like to hear what sort of things are 
being done to handle this generally. Many thanks.


I don't know anyone who uses authenticated ZEO connections, does it even 
work? IMHO most people are in a situation where the traffic between the 
ZEO clients and the ZEO server runs on an internal network, so it simply 
doesn't matter. Or small setups where the communication is on one and 
the same machine.


jens

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.1 (Darwin)

iD8DBQFEdseGRAx5nvEhZLIRApfTAJ9MjasVk9UHp1yvlBP2BNPQl6GXKACeIpIC
Yx5XIHtQzPX9+xQzbL71zh0=
=G8RQ
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] ZEO Authentication

2006-05-25 Thread David Pratt
I am curious how folks are handling authentication for ZEO in a general 
way (client against the server). Is is common to attempt to use 
user/passwords from acl in zope as a means of creating a lists for 
authenticating against a ZEO server? Or is it more typical to manage a 
separate list of user/passwords just for the purpose of authenticating 
the client server connection? I'd like to hear what sort of things are 
being done to handle this generally. Many thanks.


Regards
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB

2006-05-18 Thread David Pratt
Hi Gary. I have been experimenting with it an posed the question to 
Andreas a couple of days ago about when it might get into the trunk. I 
am using the modified ZODB and ZEO files to accomodate it in the interim 
but would be happy if it was in the trunk for sure. When is 3.7 final to 
be tagged? Will this now be for 3.8?


Regards,
David

Gary Poster wrote:


On Apr 29, 2006, at 6:08 AM, Christian Theune wrote:


Hi everybody,

the ZODB team is proud to announce a technical preview of a Blob
implementation for ZODB.


Hey.

This seems like there's no way that it will make it into the next ZODB 
release, right?


I assume no one is using it in production, on the basis of the branch TODO.

Does anyone have a hard need to get this production-ready (address the 
TODOs) by a certain date?


Gary
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Undo differences between Z2 and Z3

2006-05-16 Thread David Pratt
Hi Jim. I managed to fix this by modifying the storage but it seems to 
me strange to differentiate in a storage whether it is a backend to a z2 
or z3 app.


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Undo differences between Z2 and Z3

2006-05-16 Thread David Pratt

Jim Fulton wrote:

David Pratt wrote:
There seems to be enough of a difference between Z2 and Z3 undo to 
have an effect on a backend like PGStorage from working on Z3. I 
believe the objective in the storage api is that it should not matter 
to the backend whether you are connected to Z2 or Z3. Can someone more 
familiar with the history of undo in z2 and z3 explain the current 
situation?


Undo has a fairly complicated API, and I can easily believe that Z2 and
Z3 call that API in very different ways.

BTW, I strongly discourage use of Undo except in emergencies.
Sadly, except when undoing the last (non-undo) transactions in
a database can lead to inconsistency.  Undo should be used with
caution and should generally not be exposed to non-expert users.


Undo has been a part of CMF and Plone for non-expert users for quite 
some time by exposing only the last non-undo transaction. This should be 
fine for Z3 I believe. In any case, my interest is initially to give Z3 
what it is asking for without crapping out so contents view will work 
for a principal through z3 zmi.


In Z3, before bringing up the contents view, there is a check of the 
undo transactions for the principal. It uses z3's undo manager to 
getPrincipalTransactions which eventually results in a lookup in the 
undoLog in the external storage where it dies. Just a brief look at 
the undo code in z3 gives me the feeling the format


Format of what?


I am referring to is this which is the route this is taking to the undoLog :

def _getUndoInfo(self, context, principal, first, last):
specification = {}

if context is not None:
locatable = IPhysicallyLocatable(context, None)
if locatable is not None:
location = Prefix(locatable.getPath())
specification.update({'location': location})

if principal is not None:
# TODO: The 'user' in the transactions is a concatenation
# of 'path' and 'user' (id).  'path' used to be the path
# of the user folder in Zope 2.  ZopePublication currently
# does not set a path, so ZODB lets the path default to
# '/'.  We should change ZODB3 to set no default path at
# some point
path = '/' # default for now
specification.update({'user_name': path + ' ' + principal.id})

entries = self.__db.undoInfo(first, last, specification)



It's possible that there is a bug in PGStorage in handling
extension data.  I can easily believe that Z2 doesn't use extension
data and perhaps z3 does.


Thanks. I have reported this to Shane but am also experimenting with 
storages so have an interest in understanding what is happening.


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] Undo differences between Z2 and Z3

2006-05-16 Thread David Pratt
There seems to be enough of a difference between Z2 and Z3 undo to have 
an effect on a backend like PGStorage from working on Z3. I believe the 
objective in the storage api is that it should not matter to the backend 
whether you are connected to Z2 or Z3. Can someone more familiar with 
the history of undo in z2 and z3 explain the current situation?


In Z3, before bringing up the contents view, there is a check of the 
undo transactions for the principal. It uses z3's undo manager to 
getPrincipalTransactions which eventually results in a lookup in the 
undoLog in the external storage where it dies. Just a brief look at the 
undo code in z3 gives me the feeling the format between z2 and z3 may be 
incompatible which may be the problem.


The relevant portion of the traceback I receive is this:

  File "/usr/local/zope3/z3trunk/src/zope/tales/expressions.py", line 
199, in _eval

return ob()
  File 
"/usr/local/zope3/z3trunk/src/zope/app/publisher/browser/viewmeta.py", 
line 419, in __call__

return meth(*a, **k)
  File "/usr/local/zope3/z3trunk/src/zope/app/undo/browser.py", line 
33, in principalLastTransactionIsUndo

last=1)
  File "/usr/local/zope3/z3trunk/src/zope/app/undo/__init__.py", line 
108, in getPrincipalTransactions

return self._getUndoInfo(context, principal, first, last)
  File "/usr/local/zope3/z3trunk/src/zope/app/undo/__init__.py", line 
129, in _getUndoInfo

entries = self.__db.undoInfo(first, last, specification)
  File "/usr/local/zope3/z3trunk/src/ZODB/UndoLogCompatible.py", line 
37, in undoInfo

return self.undoLog(first, last, filter)
  File "/usr/local/zope3/z3trunk/src/ZODB/PGStorage/pgstorage.py", line 
478, in undoLog

d.update(cPickle.loads(ext))
EOFError

Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Really want to refactor ZEO's networking archiecture

2006-05-04 Thread David Pratt
Hi Jim and Roderigo! This is encouraging news. I have been putting much 
thought into this also. There is a downside to refactoring ZEO in that 
there are many folks heavily dependent on it and also reasonably happy 
with it.


Secondly, zeo has a specific place and relationship. When I initiated 
the twisted zeo discussion about a week ago, it was with the hope of 
drawing interest to some possible changes. I was thinking about 
security, my own positive experiences with twisted, and how things could 
be modified to bring tighter integration. That said, the general 
experience of twisted and zope has been short (since 3.2) and this is a 
departure that has risks that I respect.


I was encouraged to challenge my own thinking on what may be needed. 
Before committing, I am planning to build and test a few ideas to 
explore this further without confining the scope. I don't know where 
this is leading me just yet.


On the notion of multiple independent loops for twisted, I believe there 
are possibilities, some interested folks, and a patch I found that could 
be evaluated. The work would be murky as Glyph has pointed out - but he 
has offered to review tests for this as well.


Many thanks,

Regards,
David


Jim Fulton wrote:


I think that my next big project might be to
refactor ZEO's networking architecture.

Wonder what the main reason is?

- Provide more secure connections? Nope
- Leverage Twisted? Nope
- Get rid if the insane async/sync client madness? Nope
- Get better storage-server performance? Nope
- Allow simultaneous syncronous calls from the same client? Nope

I'd like to do all of these things, but the main reason to refactor
ZEO's networking architecture is to make it testable.

Writing ZEO tests now requires actually starting servers and
clients.  This is nuts.  IMO, application code shouldn't
touch sockets.  Application code, like ZODB and ZEO should
be insulated from actual network APIs.  They use simpler APIs
that are easy to interccept and control.

I've been thinking of a someone general networking API with these aims
in mind, but I realized that the ZEO existing frameworks could probably
be refactored in a more limited way to do this.  We'll see.

If anyone is interested in workingon this with me, let me know.

Jim


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZEO and Twisted

2006-04-25 Thread David Pratt
Hi Rodrigo! Great post on this. You provide an insightful and compelling 
summary of what can be accomplished. Add on top of this, the things that 
twisted "just does" like SSL or SSH out of the box and we can also have 
the security issue beat. Jim, what do you think? Do you think you could 
give us a place to experiment in zope's svn?


Regards,
David

Rodrigo Dias Arruda Senra wrote:

You probably already know all of these, but just in case ...

[ Jim Fulton ]:
---
|
|  I am strongly against linking ZEO to an application's main loop.

Twisted performs better if the protocols build upon it rely on its
reactor (asynchronous loop). That means breaking long computations
to use promises (deferreds in Twisted vocabulary), therefore increasing
the interleaving of data handlers.

Implementing ZEO in Twisted would probably isolate the zeo protocol
from the underlying transport (tcp, udp, whatever). OTOH, the
implementation would be dependent on the typical deferred patterns
(only common in Twisted nowadays). Twisted also support threads, but
that defeats most of the other benefits of using Twisted in the first
place (like not bothering about critical sections and other synch
stuff), so they are just used when blocking calls must be made.
  


|  If twisted supports using multiple independent main loops, then it
| would be an option.

I know it supports multiple implementations of reactors, each
specialized for a particular purpose, for instance to conciliate
two different loops into a single reactor, which is the case of
twisted_networking_loop + GUI_event_loop.
Therefore, it would be possible (my educated guess) to create a
zeoreactor, would that suffice ?  


That was done to marry Twisted to: gtk, gtk2, glib2, wxPython,
win32event loops.

|  Then the possible issues are:
|  
|  - Whether we want Twisted to be a dependency of ZEO


If it comes to that, probably just TwistedCore would be required,
and perhaps even a subset.
 
|  - Performance


There is a lot of this-and-that [1] about what communication pattern
performs better. Twisted is definetely better for long sessions, and
not necessarily much worse for short-termed sessions (where "people"
say thread-pools are much sexier). IMVHO, the strong argument behind
Twisted adoption would be flexibility and not huge speedups. 



[ David Pratt ]:
-
|  > Is there is any strong opposition to using the twistd daemon for a 
|  > twisted zeo service?


I volunteer to help in that effort to the best of my modest skills
and limited (just like everybody else) spare time.

[1] http://www.cs.wustl.edu/~schmidt/patterns-ace.html

cheers,
Senra

-
Rodrigo Senra
GPr Sistemas
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] Re: ZEO and Twisted

2006-04-25 Thread David Pratt
Hi Jim. My experience has been in running a single selectreactor in an 
app together with its own main loop.


I found a patch on google to allow multiple selectreactors but have not 
seen the patch applied to svn or attempted the change in twisted as yet. 
I have emailed the maintainer to ask about it. I was looking at the zeo 
server service first which should be happy to run on its own. There is 
no risk at this stage to experiment with my ideas other than learning 
more than I did before about twisted and ZEO.


Regards,
David

Jim Fulton wrote:

David Pratt wrote:
...
Is there is any strong opposition to using the twistd daemon for a 
twisted zeo service?


I am strongly against linking ZEO to an application's main loop.

If twisted supports using multiple independent main loops, then it would
be an option.

Then the possible issues are:

- Whether we want Twisted to be a dependency of ZEO

- Performance

Switching to twisted would be a big change.  If I was to make such a
change, which I anticipate, it would be great if twised could be an
option.

Jim


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] Re: ZEO and Twisted

2006-04-25 Thread David Pratt

Florent Guillaume wrote:



Huh, I thought you were talking about the ZEO client, ClientStorage, not 
the ZEO server.

For the ZEO server I don't see the point of changing it, it works well.
OTOH a ClientStorage has to integrate with the other servers in Zope, 
and that's the one that would benefit from being moved to the twisted 
event loop if twisted is used.


Hi Florent. Sorry I was meaning the whole not just asyncore and twisted 
integration for instances. I think re-implementing the zrpc protocol for 
ZEO server would clean things up quite a bit and make the security issue 
easier to deal with.


Then we are talking apples and probably less code as well since twisted 
handles a fair amount and its already in the zope distribution. Reading 
posts about about securing zeo etc from as far back as six years ago 
makes me wonder why not? Twisted uses zope interfaces, has all the right 
stuff and makes it easier on top of it. I am no networking guru for 
sure, but at the end of the day, zrpc is a simple custom protocol that 
should be pretty doable in twisted without building custom add ons to 
secure network communication.


Is there is any strong opposition to using the twistd daemon for a 
twisted zeo service?


Regards,
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


Re: [ZODB-Dev] ZEO Twisted and Z3

2006-04-24 Thread David Pratt

Good news. Sorry to reply to own post.

I have found the following interfaces:

twisted.internet.interfaces.IReactorSSL.listenSSL
twisted.internet.interfaces.IReactorSSL.connectSSL

this looks promising. I believe authentication is passed in the protocol 
 so maybe there is not that much to worry about there.


So maybe this is an issue of overriding asnychronous core with twisted.

Regards,
David

David Pratt wrote:
Is there a plan to provide security for ZEO client / server 
communication using Twisted now that it is in Zope3?  I only have not 
read all source of ZEO but wish to use it with encryption and 
authentication and Twisted is an excellent toolbox for network 
communication. Anyone done this or have any recommendations or advice.


Many thanks
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev


[ZODB-Dev] ZEO Twisted and Z3

2006-04-24 Thread David Pratt
Is there a plan to provide security for ZEO client / server 
communication using Twisted now that it is in Zope3?  I only have not 
read all source of ZEO but wish to use it with encryption and 
authentication and Twisted is an excellent toolbox for network 
communication. Anyone done this or have any recommendations or advice.


Many thanks
David
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev