Re: [Zope3-dev] Re: [Zope-dev] Announcement: the Zope Subversion repository will be down for maintenance on December 25th, 2005
On Mon, Dec 19, 2005 at 05:17:04PM -0500, Jim Fulton wrote:
Jim Fulton wrote:
Jens Vagelpihl has graciously offered to convert the subversion
repository to use a file-system back end.
Jim groans, That's Jens Vagelpohl.
Thanks again Jens!
Yes! Yes! Yes!
I can offer a short shell script that I used to convert about 50 small
repositories to fsfs. Perhaps it will come in handy.
Marius Gedminas
--
In short, at least give the penguin a fair viewing. If you still don't
like it, that's ok: that's why I'm boss. I simply know better than you
do.
-- Linus what, me arrogant? Torvalds, on c.o.l.advocacy
#!/bin/sh
# convert a subversion repository to fsfs
# usage: convert-svn-to-fsfs /path/to/repository
repos=${1%/}
test -z $repos {
echo usage: $0 /path/to/repository 12
exit
}
test -f $repos/format || {
echo $0: $repos is not a subversion repository 12
exit 1
}
test -e $repos.old {
echo $0: $repos.old already exists 12
exit 1
}
test `cat \$repos/db/fs-type\ 2/dev/null` = fsfs {
echo $0: $repos already uses fsfs 12
exit 1
}
set -e
mv $repos $repos.old
svnadmin create --fs-type fsfs $repos
svnadmin dump -q $repos.old | svnadmin load -q $repos
echo * $repos converted to fsfs
ls $repos.old/hooks | grep -v [.]tmpl$ | while read fn; do
echo copying $fn hook
cp $repos.old/hooks/$fn $repos/hooks/
done
signature.asc
Description: Digital signature
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: Make HTTP streaming of large data simpler
Sidnei da Silva wrote: On Mon, Dec 05, 2005 at 01:24:54PM -0500, Paul Winkler wrote: | On Mon, Dec 05, 2005 at 11:13:31AM -0500, Jim Fulton wrote: | BTW, your implementation also doesn't work because it doesn't | set the content length. | | Speaking of which, I'd love to see a way to make this work for | situations where you can't precompute a correct content length because | e.g. you're building a lot of data on the fly but you still want to | start streaming it as early as possible. | In this case, the HTTP spec says that you should NOT set the | content-length header but you must close the channel to signal end of | data. Or at least that's what Dieter says it says :-) | http://mail.zope.org/pipermail/zope/2005-November/162714.html | | Apparently, according to Dieter, Zope 2 doesn't let you use this idiom; | does zope 3? (I've very briefly looked at the Zope 2 code in medusa, | and I can't quite tell if Dieter is correct - there seems to be some | support for the chunked transfer coding, which should do the job, but | I'm not sure if/how one can currently take advantage of it.) There are a couple conditions that must be met for 'chunked' to work with Zope 2. 1. A Content-Length header must not be set. 2. The request must be HTTP 1.1 3. You must be streaming 4. No 'Connection: close' header has been set during the request If all of that is met, then it works just fine. The signal for 'end of data' in chunked mode is '0\r\n\r\n', which Zope properly inserts when appropriate. Is this documented anywhere? If not (or perhaps even if so :), and given problems like: Paul Winkler wrote: On Tue, Dec 06, 2005 at 12:01:37PM -0200, Sidnei da Silva wrote: On Tue, Dec 06, 2005 at 08:55:56AM -0500, Jim Fulton wrote: | Interesting. I wasn't aware of this in Z2. Zope 3 definately | doesn't have this. Sindnei, have you verified that this actially | works in Z2? I didn't think the Zope 2 publisher actually started | propducing output until the request was finished. Yes, I can confirm it works like I described. I've been using and relying on this (where this == chunked response producing output before the request is finished) on Enfold Desktop for a while. But, as Sidnei pointed out to me offlist, it's not as modular as a first glance at the code would make one hope :-) From an offlist mesage: Actually, chunked_producer doesn't seem to be used, afaict. Chunking is inlined in the body of HTTPResponse.write(). That could really get some loving. I've recently added 'gzip' content-encoding to streamed content by hacking around the 'ChannelPipe' and it was a real pain to do gzip+chunked. Updating the compressed producer to add the missing zlib header + checksum and actually using those instead of hardcoding it all in ZServer would be very interesting. BUT, in the light of Zope 3 moving to twisted, and possibly Zope 2 at the time of Zope 2.11, maybe that's not worth the effort. I'd really like us to think about how we *want* this to work in Zope (2 and 3). I'm uncomfortable with the subtle cues that seem to change the behavior of request.write from non-streaming to streaming. I'd like to see a proposal for a clean API that supports: - Simple traditional output (ie pages) - Non-streaming large-file output. o Don't hold application thread to do I/O o Don't consume a lot of memory - Streaming output o For applications that tale a lot of time to generate output. o Output is generated over a long period of time. o Perhaps extends life of application thread. (Maybe there are other alternatives.) Note that there are some serious issues being discussed on the Python Web SIG wrt threading and WSGI that have a bearing on this discussion. I'm worried that we won't get adequate threading support from WSGI. I hope that isn't the case. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: Make HTTP streaming of large data simpler
On Tue, Dec 20, 2005 at 07:33:13AM -0500, Jim Fulton wrote: | There are a couple conditions that must be met for 'chunked' to work | with Zope 2. | | 1. A Content-Length header must not be set. | 2. The request must be HTTP 1.1 | 3. You must be streaming | 4. No 'Connection: close' header has been set during the request | | If all of that is met, then it works just fine. The signal for 'end of | data' in chunked mode is '0\r\n\r\n', which Zope properly inserts when | appropriate. | | Is this documented anywhere? In this email and on the source only it seems :( Could write a doctest for sure. | I'd really like us to think about how we *want* this to work in Zope (2 and | 3). | | I'm uncomfortable with the subtle cues that seem to change the behavior of | request.write from non-streaming to streaming. I'd like to see | a proposal for a clean API that supports: | | - Simple traditional output (ie pages) | | - Non-streaming large-file output. | | o Don't hold application thread to do I/O | | o Don't consume a lot of memory | | - Streaming output | | o For applications that tale a lot of time to generate | output. | | o Output is generated over a long period of time. | | o Perhaps extends life of application thread. | | (Maybe there are other alternatives.) If I understand correctly the newly introduced one would be the second bullet? I don't have an idea of how that can be achieved. FWIW, I could live with returning a generator instead of using request.write for all applications where I use request.write today. One thing that I had in mind for long-running streaming output is that 99% of the time it just needs a constant, read-only view of the database at the time the streaming started so instead of using a connection from the zodb connection pool, it could open a new, read-only connection and use that for delivering the content. | Note that there are some serious issues being discussed | on the Python Web SIG wrt threading and WSGI that have a bearing on | this discussion. I'm worried that we won't get adequate threading | support from WSGI. I hope that isn't the case. Me too. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] URGENT RFC: Is anyone using response.write in Zope 3?
Jeff Shell wrote: Yes, it's hurry.file. What's Tramline? http://www.infrae.com/newsitems/tramline_0_4_release http://www.infrae.com/products/tramline http://faassen.n--tree.net/blog/view/weblog/2005/11/11/0 Regards, Martijn ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] right introspector permission
Hi The permission for the introspector.html page is the zope.ManageContent (source zope/app/apidoc/codemodule/browser/introspector.zcml). However the permission zope.app.introspector.Introspect is also defined: perhaps the last one is more appropriated that the current one? Kind regards, Marco ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: Make HTTP streaming of large data simpler
Sidnei da Silva wrote: On Tue, Dec 20, 2005 at 07:33:13AM -0500, Jim Fulton wrote: | There are a couple conditions that must be met for 'chunked' to work | with Zope 2. | | 1. A Content-Length header must not be set. | 2. The request must be HTTP 1.1 | 3. You must be streaming | 4. No 'Connection: close' header has been set during the request | | If all of that is met, then it works just fine. The signal for 'end of | data' in chunked mode is '0\r\n\r\n', which Zope properly inserts when | appropriate. | | Is this documented anywhere? In this email and on the source only it seems :( Could write a doctest for sure. That's what I thought. :) That's good. It means we're not bound. ;) | I'd really like us to think about how we *want* this to work in Zope (2 and | 3). | | I'm uncomfortable with the subtle cues that seem to change the behavior of | request.write from non-streaming to streaming. I'd like to see | a proposal for a clean API that supports: | | - Simple traditional output (ie pages) | | - Non-streaming large-file output. | | o Don't hold application thread to do I/O | | o Don't consume a lot of memory | | - Streaming output | | o For applications that tale a lot of time to generate | output. | | o Output is generated over a long period of time. | | o Perhaps extends life of application thread. | | (Maybe there are other alternatives.) If I understand correctly the newly introduced one would be the second bullet? I don't have an idea of how that can be achieved. What newly introduced one? You mean the IResult business? The IResult api addresses the first use case, not the second. FWIW, I could live with returning a generator instead of using request.write for all applications where I use request.write today. Cool. But that alone doesn't really address the core issue of how the application thread should be managed. (I need to think about this some more.) One thing that I had in mind for long-running streaming output is that 99% of the time it just needs a constant, read-only view of the database at the time the streaming started so instead of using a connection from the zodb connection pool, it could open a new, read-only connection and use that for delivering the content. I don't see how that helps. Avoiding writes doesn't substantially (if at all) reduce resource consumption. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Finished zope/i18n/format.py XXX (was Re: [Zope3-dev] XXX Roundup)
On Dec 19, 2005, at 2:41 PM, Gary Poster wrote: On Dec 17, 2005, at 4:06 PM, Gary Poster wrote: On Dec 16, 2005, at 4:20 PM, Benji York wrote: Gary zope/i18n/format.py:141 and 149 Update, if anyone needs to follow along: ... This is fixed in trunk and 3.2 branch. Thanks to Stuart Bishop for his feedback and help. Gary ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: Make HTTP streaming of large data simpler
On Tue, Dec 20, 2005 at 09:52:09AM -0500, Jim Fulton wrote: | One thing that I had in mind for long-running streaming output is that | 99% of the time it just needs a constant, read-only view of the | database at the time the streaming started so instead of using a | connection from the zodb connection pool, it could open a new, | read-only connection and use that for delivering the content. | | I don't see how that helps. Avoiding writes doesn't substantially | (if at all) reduce resource consumption. I was thinking that it would free a database connection back to the connection pool and use a new, read-only connection with a 'delivery thread', freeing the application thread. But maybe that doesn't really help at all. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] RFC: Make HTTP streaming of large data simpler
On Tue, Dec 20, 2005 at 10:48:24AM -0200, Sidnei da Silva wrote: On Tue, Dec 20, 2005 at 07:33:13AM -0500, Jim Fulton wrote: | I'd really like us to think about how we *want* this to work in Zope (2 and | 3). | | I'm uncomfortable with the subtle cues that seem to change the behavior of | request.write from non-streaming to streaming. Agreed. I had no idea about those cues until Sidnei pointed it out. | I'd like to see | a proposal for a clean API that supports: | | - Simple traditional output (ie pages) | | - Non-streaming large-file output. | | o Don't hold application thread to do I/O | | o Don't consume a lot of memory | - Streaming output | | o For applications that tale a lot of time to generate | output. | | o Output is generated over a long period of time. | | o Perhaps extends life of application thread. | | (Maybe there are other alternatives.) In zope 2 you can also do RESPONSE.setBody(foo), where foo provides ZPublisher.Iterators.IStreamIterator. Works well for what you're calling non-streaming large-file output. However, it doesn't handle the streaming output case: for two reasons: 1) there's a big Don't Do That note in the docstring about not using the database in your iterator code; 2) it mandates a precomputed content-length, which could be fixed pretty easily: http://www.zope.org/Collectors/Zope/1962 One thing that I had in mind for long-running streaming output is that 99% of the time it just needs a constant, read-only view of the database at the time the streaming started so instead of using a connection from the zodb connection pool, it could open a new, read-only connection and use that for delivering the content. I agree that read-only is the more likely scenario. However, the 1% may be more than 1%. Suppose I'm writing a UI for maintaining a gigantic site. Some actions might be inherently slow and inherently write to the ZODB - e.g. rebuilding a massive catalog. I like to give feedback to the user early and often, if possible. Another case that affects me today: using ZSyncer in pull mode. In current ZSyncer cvs, i use response.write() with http 1.1 chunking as Sidnei described, and it works nicely for giving feedback as each item is fetched or pushed. Some of these operations involve writes, and none of them have a known output size, so I can't currently use an IStreamIterator. -- Paul Winkler http://www.slinkp.com ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] buildbot failure in Zope3 branches 3.2 2.4 Linux remy
The Buildbot has detected a failed build of Zope3 branches 3.2 2.4 Linux remy. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 2368 Blamelist: frerich,jim BUILD FAILED: failed svn sincerely, -The Buildbot ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] buildbot failure in Zope3 branches 3.2 2.4 Windows 2000 zc-bbwin
The Buildbot has detected a failed build of Zope3 branches 3.2 2.4 Windows 2000 zc-bbwin. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 2368 Blamelist: frerich,jim BUILD FAILED: failed svn sincerely, -The Buildbot ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] why security proxy the modules for a menu item directive?
zope/app/publisher/browser/menu.py, line 132, security proxies the 'modules' name. This is pretty annoying, and seems unnecessary to me for a zcml filter: trusted filesystem code. Anyone want to defend it? Gary ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
