RE: [Zope-dev] Serving big files thru zope?

[Kapil Thangavelu]

you could play with
http://puggy.symonds.net/~srp/stuff/mod_auth_remote/

and have apache auth off zope.

cheers,

-kapil

On Mon, 2004-03-01 at 11:06, Sandor Palfy wrote:
> > I am also wondering about the security aspects of bypassing Zope to 
> > serve files - acess to some of the files we need to serve is 
> > restricted 
> > to particular users with particular permissions, and I cannot clearly 
> > see how this would be possible to accomplish without Zope's security 
> > machinery.
> 
> Same situation here. Is there any way to provide authentication
> information for Apache from Zope (which works at least with cookie based
> auth)? 
> 
> Regards,
> Sandor
> 
> 
> ___
> Zope-Dev maillist  -  [EMAIL PROTECTED]
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope )
> 


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Marc Lindahl]

On Monday, March 1, 2004, at 02:43  PM, Dieter Maurer wrote:
In order to reduce memory consumption and decouple Zope from
ZServer (and the response delivery), a large file
is spooled via a temporary.
This means, that the file content is read from ZODB, stored
in a temporary file and then delivered from this temporary file.
At least, when your client has a high bandwidth to your server,
this serving through a temporary file is a waste.
I found that putting my /var partition (the default place for zope's 
temp files) on a very fast SCSI disk with plenty of space (GB's) 
improved the situation tremendously

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Paul Winkler]

On Tue, Mar 02, 2004 at 07:06:31PM +0100, Dieter Maurer wrote:
(snip)
> I interpreted these observations in the following way:
> 
>   *  time is dominated by the disc access
> 
>   *  time is largely independent of the object size (if
>  the object is in the OS disc cache)
> 
>   *  ZEO overhead is about 1 to 3 ms per object.

Hmm. I wondered if ZEO overhead per-object was significant,
but this seems not too bad.

If I read OFS/Image.py correctly, the pdata chunks are
65536 bytes.  So a 40 MB file would split up into
about 640 chunks, so ZEO overhead by your calculations
should be roughly 1-2 seconds.
This does not account for the order of magnitude difference
I am seeing (initial load from filestorage: 0:03.87,
initial load from zeo: 0:33.54).

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's PROTO FLASH TROMBONIST!
(random hero from isometric.spaceninja.com)

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Dieter Maurer]

Jeremy Hylton wrote at 2004-3-1 14:01 -0500:
> ...
>It really would be good to get to the bottom of the bad ZEO performance
>for loading an object the first time.  I won't have time to do it until
>after the ZODB 3.3 release, but I think it's a worthwhile effort to
>understand what ZEO is doing.

I instrumented "ClientStorage" to get precise timings
for ZEO operations. The obtained times are not bad:

When the data file was not yet in the operating system's
cache, loading an object took between 3 to 30 ms
with very high variance.

After the relevant parts of the data file have been cached
by the OS, time was between 1 to 3 ms per object, almost
independent from object size.

I obsevered loading of about 1.600 objects between several
hundred bytes and several kilobytes in size
mostly index data for a ZCatalog "FieldIndex".

I had disabled both the ZODB cache as well as
the ZEO client cache for my measurement.

ZEO client and server have been on the same machine (local
communication)

I interpreted these observations in the following way:

  *  time is dominated by the disc access

  *  time is largely independent of the object size (if
 the object is in the OS disc cache)

  *  ZEO overhead is about 1 to 3 ms per object.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Dieter Maurer]

Dario Lopez-Kästen wrote at 2004-3-1 08:24 +0100:
> ...
>Is the current state of ZServer such that it is suboptimal for serving 
>large files? if so,
>
>a) what is the problem with the current implementation?

In order to reduce memory consumption and decouple Zope from
ZServer (and the response delivery), a large file
is spooled via a temporary.

This means, that the file content is read from ZODB, stored
in a temporary file and then delivered from this temporary file.

At least, when your client has a high bandwidth to your server,
this serving through a temporary file is a waste.

Serving static content via Python/Zope is about one order
of magnitude slower than serving the same static content
via Apache.

>b) is there a fix available?

Probably, but it will be a lot of work:

  When your client (or its connection) is slow, you
  want to spool large files indirectly (to free the
  scares Zope worker thread soon).
  Only when you know your client is fast, would you want
  to serve directly.
  An adaptive scheme is possible but much more demanding
  than the simple schema (to use a temporary file for
  all sufficiently large content).

  You could code ZServer in "C" and speed it up thereby.
  Maybe "psyco" helps a bit?

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Chris McDonough]

I think we should work on making Zope perform better when large files
are downloaded at the PyCon sprint in March, as you suggested Paul,
FWIW.

On Mon, 2004-03-01 at 13:41, Paul Winkler wrote:
> On Mon, Mar 01, 2004 at 01:03:24PM +0100, Dario Lopez-K?sten wrote:
> > I am also wondering about the security aspects of bypassing Zope to 
> > serve files - acess to some of the files we need to serve is restricted 
> > to particular users with particular permissions, and I cannot clearly 
> > see how this would be possible to accomplish without Zope's security 
> > machinery.
> 
> Tricky.  The only solution I've heard of (but not tried)
> is to use Squid configured for disk caching and forcing
> revalidation against the proxied server (zope) before allowing
> a download from the cache.
>  
> > So, I am looking for a solution that either utilises Zope+ZServer with 
> > additional fixes or that replaces ZServer altoghether with something 
> > better (Apache, Pound, Twisted?). All this assuming of course that 
> > ZServer is not good at serving large files to many users.
> 
> I'm not sure about that anymore.
> My latest experiments with large files 
> suggest that Zope / ZServer might not be as
> bad as I thought when compared to Apache *over a network*.
> I had previously only tested on localhost and observed
> that Apache is about 10x faster than ZServer for a 40 MB file.
> I had (probably erroneously) thought that this explained
> the poor download times from Zope that I had seen in the field.
> But on a 100 Mbps local network, the download times were pretty
> close - the network becomes the bottleneck and Apache was only
> a little bit faster than Zope.
> 
> However, ZEO is a whole other story.
> The time to load a 40 MB file from ZEO and serve it,
> is about another 10x slower than plain Zope  without ZEO.
> This is painfully bad and readily apparent to users.
> But if your ZEO cache is large enough to hold all the data, 
> subsequent download times are as fast as plain Zope. 
> More investigation needed, I'm not sure about the accuracy
> or relevance of these results.  But almost certainly,
> if using large files with Zope with ZEO, you will want to configure 
> a much larger ZEO cache size than the default (20 MB). 
> 
> Also, this experiment was only for a single download at
> a time.  If you expect a lot of concurrent downloads of
> large files, better do some testing to see if Zope's large
> file download time deteriorates under load.
> I am not especially optimistic ;-)


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Jeremy Hylton]

On Mon, 2004-03-01 at 13:41, Paul Winkler wrote:
> However, ZEO is a whole other story.
> The time to load a 40 MB file from ZEO and serve it,
> is about another 10x slower than plain Zope  without ZEO.
> This is painfully bad and readily apparent to users.
> But if your ZEO cache is large enough to hold all the data, 
> subsequent download times are as fast as plain Zope. 
> More investigation needed, I'm not sure about the accuracy
> or relevance of these results.  But almost certainly,
> if using large files with Zope with ZEO, you will want to configure 
> a much larger ZEO cache size than the default (20 MB). 

It really would be good to get to the bottom of the bad ZEO performance
for loading an object the first time.  I won't have time to do it until
after the ZODB 3.3 release, but I think it's a worthwhile effort to
understand what ZEO is doing.

You're right, too, that a large ZEO cache makes a big difference.  If
you've got the disk space, setting it several hundred MB shouldn't be a
problem.

Jeremy



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Paul Winkler]

On Mon, Mar 01, 2004 at 01:03:24PM +0100, Dario Lopez-K?sten wrote:
> I am also wondering about the security aspects of bypassing Zope to 
> serve files - acess to some of the files we need to serve is restricted 
> to particular users with particular permissions, and I cannot clearly 
> see how this would be possible to accomplish without Zope's security 
> machinery.

Tricky.  The only solution I've heard of (but not tried)
is to use Squid configured for disk caching and forcing
revalidation against the proxied server (zope) before allowing
a download from the cache.
 
> So, I am looking for a solution that either utilises Zope+ZServer with 
> additional fixes or that replaces ZServer altoghether with something 
> better (Apache, Pound, Twisted?). All this assuming of course that 
> ZServer is not good at serving large files to many users.

I'm not sure about that anymore.
My latest experiments with large files 
suggest that Zope / ZServer might not be as
bad as I thought when compared to Apache *over a network*.
I had previously only tested on localhost and observed
that Apache is about 10x faster than ZServer for a 40 MB file.
I had (probably erroneously) thought that this explained
the poor download times from Zope that I had seen in the field.
But on a 100 Mbps local network, the download times were pretty
close - the network becomes the bottleneck and Apache was only
a little bit faster than Zope.

However, ZEO is a whole other story.
The time to load a 40 MB file from ZEO and serve it,
is about another 10x slower than plain Zope  without ZEO.
This is painfully bad and readily apparent to users.
But if your ZEO cache is large enough to hold all the data, 
subsequent download times are as fast as plain Zope. 
More investigation needed, I'm not sure about the accuracy
or relevance of these results.  But almost certainly,
if using large files with Zope with ZEO, you will want to configure 
a much larger ZEO cache size than the default (20 MB). 

Also, this experiment was only for a single download at
a time.  If you expect a lot of concurrent downloads of
large files, better do some testing to see if Zope's large
file download time deteriorates under load.
I am not especially optimistic ;-)

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's THE PREPUBESCENT PROFESSOR!
(random hero from isometric.spaceninja.com)

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

RE: [Zope-dev] Serving big files thru zope?

[Sandor Palfy]

> I am also wondering about the security aspects of bypassing Zope to 
> serve files - acess to some of the files we need to serve is 
> restricted 
> to particular users with particular permissions, and I cannot clearly 
> see how this would be possible to accomplish without Zope's security 
> machinery.

Same situation here. Is there any way to provide authentication
information for Apache from Zope (which works at least with cookie based
auth)? 

Regards,
Sandor


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Dario Lopez-Kästen]

Stefan H. Holek wrote:

Dario,

The best solution for me always has been to serve such files statically 
from Apache, bypassing ZServer altogether. In fact I have added support 
for static URLs to my version of ExtFile 
.

ZEO clusters will need a comon disk share (SAMBA, NFS) for this to work.
Hi, thanks for the reply.

I am a bit hesitant to revert to filesystem based storage at this point 
- we used to use LocalFS but switched from it in favor of DirectoryStorage.

I am also wondering about the security aspects of bypassing Zope to 
serve files - acess to some of the files we need to serve is restricted 
to particular users with particular permissions, and I cannot clearly 
see how this would be possible to accomplish without Zope's security 
machinery.

So, I am looking for a solution that either utilises Zope+ZServer with 
additional fixes or that replaces ZServer altoghether with something 
better (Apache, Pound, Twisted?). All this assuming of course that 
ZServer is not good at serving large files to many users.

Thanks,

/dario

--
-- ---
Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech.
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] Serving big files thru zope?

[Stefan H. Holek]

Dario,

The best solution for me always has been to serve such files statically 
from Apache, bypassing ZServer altogether. In fact I have added support for 
static URLs to my version of ExtFile .

ZEO clusters will need a comon disk share (SAMBA, NFS) for this to work.

HTH,
Stefan
--On Montag, 01. März 2004 08:24 +0100 Dario Lopez-Kästen 
<[EMAIL PROTECTED]> wrote:

Hello,

I need to serve big files from zope - PDF, movies, large PS-files,
Students lab-results, etc. We recently moved from LocalFS to  Dirstorage
to manage ZEO-ability in a clustered enviromnet).
Quite a while ago, I remember having read somewhere on the lists that
there is a pacth/fix for Zope/ZServer that optimises the way zope serves
large files.
Is the current state of ZServer such that it is suboptimal for serving
large files? if so,
a) what is the problem with the current implementation?
b) is there a fix available?
if not b) then I am willing to start looking into it. I need this for my
unis student web-site.
Thanx,

/dario

--
-- ---
Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech.
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -  http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


--
The time has come to start talking about whether the emperor is as well
dressed as we are supposed to think he is.   /Pete McBreen/
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )