Re: [Zope-dev] Re: AdaptableStorage
Shane Hathaway wrote: That would be pretty cool, since it's easy for scripts to add to the queue, and it gives you plenty of granularity. It wouldn't work under Windows, though, AFAIK. Maybe on Windows we could call back to a network socket that accepts the same kind of data. Oops, I meant fall back to a network socket. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
[EMAIL PROTECTED] wrote: Hi Shane I have been thinking about this last night. I think you should provide some sort of out of band signalling method. But given that Adaptable Storage sits below ZODB, which sit's below Zope, I feel that the signalling method should be directed at the lowest level. This means that method doesn't need to be exposed to ZODB, and if people use ZODB without Zope they can use Adaptable storage and not need Zope to perform the signalling through a URL. (A Zope front end could then use it, or anything else for that matter) I would suggest a something like a named pipe, and do a blocked read from a python thread (would this be portable and can you actually do that ;-). Then use a simple protocol of sending the path to the file that has been changed. (That way you also get a FIFO queue of events.) You could make it that if the path is a directory then everything under could be treated as changed. That would be pretty cool, since it's easy for scripts to add to the queue, and it gives you plenty of granularity. It wouldn't work under Windows, though, AFAIK. Maybe on Windows we could call back to a network socket that accepts the same kind of data. (Sorry for taking so long to get back to you. I got swamped.) Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
seb bacon wrote: I'd prefer just to have a method somewhere that, as Shane suggested, could be hit by URL, etc. A whole seperate server seems like overkill... But what about, for example, databases which don't have an efficient way to do callbacks to external applications? You may have to do a SELECT id FROM tblObjects WHERE timestamp some_time or a similar kludge from a polling server. You may want this server to reside at the same location as the RDBMS, rather than as a thread in Zope. Well, if by server you could mean script that gets run by cron every 1 minute and hits a URL in Zope if something has changed, then I might be in agreement ;-) I'm worrying that if we are not to be restricted to Oracle or bleeding edge kernels, the notification part of the cache invalidation scheme may be (a) kludgy, (b) inefficient, and (c) utterly different in design between different storages. I think this is as such is life problem. Provided AdaptableStorage provides some way (exposed url?) for an external process to say that things have changed, I think that's the best form of flexibility we can provide. cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
On Wed, 22 Jan 2003, Chris Withers wrote: I think this is as such is life problem. Provided AdaptableStorage provides some way (exposed url?) for an external process to say that things have changed, I think that's the best form of flexibility we can provide. I think I'll provide such an URL, then. Thanks for going over this. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Chris Withers wrote: seb bacon wrote: I'd prefer just to have a method somewhere that, as Shane suggested, could be hit by URL, etc. A whole seperate server seems like overkill... Well, if by server you could mean script that gets run by cron every 1 minute and hits a URL in Zope if something has changed, then I might be in agreement ;-) Put it like this, I don't mean managed blade cluster running custom distributed architecture :-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
How about having a seperate process which just watched the files and notifed Zope when they changed? A definite possibility. It might even just poke an URL to send the notification. Since every storage will have its own unique notification scheme, which may be more or less inefficient (worst case scenario, periodic polling of entire storage for recently modified items), it might make sense to have a notification server. It would make it simpler for users to create custom storage transaction alert handlers. For different cache invalidation scenarios, Zope could poll the server as well as get poked. seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
seb bacon wrote: Since every storage will have its own unique notification scheme, which may be more or less inefficient (worst case scenario, periodic polling of entire storage for recently modified items), it might make sense to have a notification server. It would make it simpler for users to create custom storage transaction alert handlers. For different cache invalidation scenarios, Zope could poll the server as well as get poked. I'd prefer just to have a method somewhere that, as Shane suggested, could be hit by URL, etc. A whole seperate server seems like overkill... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Chris Withers wrote: seb bacon wrote: Since every storage will have its own unique notification scheme, which may be more or less inefficient (worst case scenario, periodic polling of entire storage for recently modified items), it might make sense to have a notification server. It would make it simpler for users to create custom storage transaction alert handlers. For different cache invalidation scenarios, Zope could poll the server as well as get poked. I'd prefer just to have a method somewhere that, as Shane suggested, could be hit by URL, etc. A whole seperate server seems like overkill... But what about, for example, databases which don't have an efficient way to do callbacks to external applications? You may have to do a SELECT id FROM tblObjects WHERE timestamp some_time or a similar kludge from a polling server. You may want this server to reside at the same location as the RDBMS, rather than as a thread in Zope. I'm worrying that if we are not to be restricted to Oracle or bleeding edge kernels, the notification part of the cache invalidation scheme may be (a) kludgy, (b) inefficient, and (c) utterly different in design between different storages. A server could offer a layer of indirection which could provide a single API for Zope to see, an opportunity to take the process load somewhere else, and a pluggable interface for writers of storages. On the other hand, I don't know much about RDBMS callbacks or filesystem accounting, so I could be inventing a problem to solve :-) seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Shane Hathaway wrote: I'm thinking about real-time updates. When the underlying data changes, you'd like Zope to see the change immediately. If indefinite delays are OK, then AdaptableStorage already does enough: it raises a ConflictError if you try to write changes based on old data. I think it depends on how you're using it, so I guess thsi wants to be configurable. Is that possible? The idea I like the most for relational databases is to ask the RDBMS what the ID of the last transaction was. If Zope missed a transaction, it should flush all caches. This will work if the database is infrequently changed by external applications, or if Zope is accessed infrequently. Indeed. If external applications make a lot of changes, however, and Zope needs good performance at the same time, then both Zope and the external applications need to update a per-object transaction ID. Then, at the beginning of transactions, Zope would invalidate only the recently updated objects. Hmm, perhaps smarter RDBMSs could make it easy to keep transaction IDs updated using triggers. (This solution could also replace both ZEO and ZRS, BTW. ;-) ) This sounds cool, and the best option, when it's possible... On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. How about having a seperate process which just watched the files and notifed Zope when they changed? For that case, I'm thinking that requiring external apps to touch a special file somewhere might be the right thing. At the beginning of each transaction, if Zope sees a change to the file, it flushes its cache. I don't think you can rely on this :-( I'm hoping to present a complete tutorial on AdaptableStorage at PyCon DC 2003. I hope you'll make this availabel for those of us who can't make it... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Chris Withers wrote: Shane Hathaway wrote: I'm thinking about real-time updates. When the underlying data changes, you'd like Zope to see the change immediately. If indefinite delays are OK, then AdaptableStorage already does enough: it raises a ConflictError if you try to write changes based on old data. I think it depends on how you're using it, so I guess thsi wants to be configurable. Is that possible? I think so. How about having a seperate process which just watched the files and notifed Zope when they changed? A definite possibility. It might even just poke an URL to send the notification. I'm hoping to present a complete tutorial on AdaptableStorage at PyCon DC 2003. I hope you'll make this availabel for those of us who can't make it... Yes, I plan to, assuming they accept my proposal. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Chris Withers wrote: Shane Hathaway wrote: performance. There needs to be a way for applications that modify the database to tell Zope about the modification, so Zope can reset its caches. But, IIRC, the last time this was discussed on a mailing list you had some cool ideas to sovle the problem, right? Yes, but I want to hear other people's ideas first. What do you think? Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
seb bacon wrote: Shane Hathaway wrote: Chris Withers wrote: Shane Hathaway wrote: performance. There needs to be a way for applications that modify the database to tell Zope about the modification, so Zope can reset its caches. But, IIRC, the last time this was discussed on a mailing list you had some cool ideas to sovle the problem, right? Yes, but I want to hear other people's ideas first. What do you think? Isn't this a different problem for each kind of storage, e.g. MD5 hash for ext2, transaction ID for foo...? Or are you referring to a different aspect of the problem? I'm thinking about real-time updates. When the underlying data changes, you'd like Zope to see the change immediately. If indefinite delays are OK, then AdaptableStorage already does enough: it raises a ConflictError if you try to write changes based on old data. The idea I like the most for relational databases is to ask the RDBMS what the ID of the last transaction was. If Zope missed a transaction, it should flush all caches. This will work if the database is infrequently changed by external applications, or if Zope is accessed infrequently. If external applications make a lot of changes, however, and Zope needs good performance at the same time, then both Zope and the external applications need to update a per-object transaction ID. Then, at the beginning of transactions, Zope would invalidate only the recently updated objects. Hmm, perhaps smarter RDBMSs could make it easy to keep transaction IDs updated using triggers. (This solution could also replace both ZEO and ZRS, BTW. ;-) ) On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. For that case, I'm thinking that requiring external apps to touch a special file somewhere might be the right thing. At the beginning of each transaction, if Zope sees a change to the file, it flushes its cache. While reading the referenced thread on the subject, I found your description of the product design here: http://lists.zope.org/pipermail/zope-dev/2002-August/016981.html Could this go in the docs/ directory of the product? The design, while very clean, doesn't lend itself to immediate understanding on a cursory view of the source... I'm hoping to present a complete tutorial on AdaptableStorage at PyCon DC 2003. I'll integrate those notes. Thanks for pointing them out--I'd forgotten about them. The names are changed somewhat, but the basic design is the same. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Shane Hathaway wrote: Chris Withers wrote: Shane Hathaway wrote: performance. There needs to be a way for applications that modify the database to tell Zope about the modification, so Zope can reset its caches. But, IIRC, the last time this was discussed on a mailing list you had some cool ideas to sovle the problem, right? Yes, but I want to hear other people's ideas first. What do you think? Isn't this a different problem for each kind of storage, e.g. MD5 hash for ext2, transaction ID for foo...? Or are you referring to a different aspect of the problem? While reading the referenced thread on the subject, I found your description of the product design here: http://lists.zope.org/pipermail/zope-dev/2002-August/016981.html Could this go in the docs/ directory of the product? The design, while very clean, doesn't lend itself to immediate understanding on a cursory view of the source... seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Shane Hathaway wrote: On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. FWIW, since I had the same problem some time ago (which could be solved in another way), I dug out an url, which might be of interest - probably you already know about it: FAM, used by the two major open source desktop envs: http://oss.sgi.com/projects/fam/ It may at least help to make the whole problem more os independend. They have a lot of related pointers on their homepage. Btw. windows (=nt IIRC) already has the capability to notify on directory alteration events, without polling. cheers, oliver ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Oliver Bleutgen wrote: Shane Hathaway wrote: On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. FWIW, since I had the same problem some time ago (which could be solved in another way), I dug out an url, which might be of interest - probably you already know about it: FAM, used by the two major open source desktop envs: http://oss.sgi.com/projects/fam/ It may at least help to make the whole problem more os independend. They have a lot of related pointers on their homepage. I've seen it before, but I don't think FAM is able to monitor an entire directory tree. It only monitors individual files. I'd really like to be wrong. :-) Btw. windows (=nt IIRC) already has the capability to notify on directory alteration events, without polling. Do you know what API? That would sure help. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Shane Hathaway wrote: Oliver Bleutgen wrote: Shane Hathaway wrote: On the filesystem, the problem seems much more difficult, since there are no transactions. You'd like the kernel to send Zope a message anytime someone modifies a file in a certain hierarchy, but that would require kernel hacking. FWIW, since I had the same problem some time ago (which could be solved in another way), I dug out an url, which might be of interest - probably you already know about it: FAM, used by the two major open source desktop envs: http://oss.sgi.com/projects/fam/ It may at least help to make the whole problem more os independend. They have a lot of related pointers on their homepage. I've seen it before, but I don't think FAM is able to monitor an entire directory tree. It only monitors individual files. I'd really like to be wrong. :-) I think you are wrong, because the manpage (for IRIX) says otherwise. Additionally, it wouldn't be of much use for kde etc. if it only could monitor files. I think a filemanager would mainly be interested in directory changes (files added/deleted). Then there's also dnotify (also reference from the FAM site) - there's hope that the d isn't an acronym for file ;). I remember someting about recent 2.4.x versions having the prerequisites to use that. Btw. windows (=nt IIRC) already has the capability to notify on directory alteration events, without polling. Do you know what API? That would sure help. I don't have any expirience on win32, but just searched google. There's Win32::ChangeNotify for perl, described here http://www.xav.com/perl/site/lib/Win32/ChangeNotify.html and this seems to use ReadDirectoryChangesW, decribed here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/readdirectorychangesw.asp cheers, oliver ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: AdaptableStorage
Oliver Bleutgen wrote: Shane Hathaway wrote: I've seen it before, but I don't think FAM is able to monitor an entire directory tree. It only monitors individual files. I'd really like to be wrong. :-) I think you are wrong, because the manpage (for IRIX) says otherwise. Additionally, it wouldn't be of much use for kde etc. if it only could monitor files. I think a filemanager would mainly be interested in directory changes (files added/deleted). I checked again. It is still limited to 1000 files or directories at a time. It's not meant for entire subtrees, it has to run as root, and it requires portmap, making it less attractive. Then there's also dnotify (also reference from the FAM site) - there's hope that the d isn't an acronym for file ;). I remember someting about recent 2.4.x versions having the prerequisites to use that. Now this one is quite interesting. It requires at least kernel 2.4.19, so I guess I'm at the edge of kernel development. (!) It just might do the trick, and maybe even better than I hoped. Thanks. I don't have any expirience on win32, but just searched google. There's Win32::ChangeNotify for perl, described here http://www.xav.com/perl/site/lib/Win32/ChangeNotify.html and this seems to use ReadDirectoryChangesW, decribed here: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/readdirectorychangesw.asp Yep, that's it. With some unicode update, it even works on Win95. But I'm going to leave the Windows work for someone else. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: AdaptableStorage
seb bacon wrote: Shane, AdaptableStorage is insane and beautiful - congratulations :-) Thanks! I've been working on this for a long time. Two years ago a Digital Creations customer demanded proper object-relational mapping. The customer abandoned us for different reasons, but I feel like the root cause was our inability to fulfill that basic requirement. Ever since then I've focused on this. It took 99% perspiration and 1% inspiration, yet inspiration accounted for 99% of the implementation. :-) It could fit a possible project we have coming up where a requirement is to store content in an XML format inside MS SQL Server. It should fit that use quite well, I think. Do you have a TODO list? Are there any particularly alpha parts? I would need to get an idea of the risk I was taking on, but if the list is sufficiently small, I love to help mature and extend the product. I didn't release the alpha until all the gnarly details were dealt with, like moving objects on the filesystem and implementing complex mappers. The only things left to do before calling it a beta release are: - Implement specific mappers for more Zope 2 object types, like DTML methods, BTreeFolders, etc. - Come up with a good manual cache invalidation scheme. Currently, you can disable the cache (or turn on volatile mode, which is really the same thing) and you'll see immediate updates, but you'll lose a lot of performance. There needs to be a way for applications that modify the database to tell Zope about the modification, so Zope can reset its caches. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: AdaptableStorage
Paul Winkler scrive: On Wed, Jan 15, 2003 at 05:30:58PM +, seb bacon wrote: Shane, AdaptableStorage is insane and beautiful - congratulations :-) It seems to inspire insanity :) kosh and i got into a discussion on #zope about using AdaptableStorage with reiserfs4, mapping zope properties to reiserfs4 properties... finally, a fully really transparent unix filesystem - ZODB solution that isn't a half-assed version of either! didn't get into detail but it seems like it should be doable. uhumm.. nice idea... i'm thinking also about using Subversion (subversiom.tigris.org), a cvs interesting alternative, as a storage for zope objects at some level. Subversion has the concept of versioned properties on a file and, best of all, it has a complete python swig interface. I'm still thinking at what level should i do the mapping, if at Storage level or at user level, like CVSFolder does. After looking at the code, it seems that AdaptableStorage could help on this. azazel ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
