Feature Request: dav_fs impersonation

2005-12-05 Thread Phillip Susi
I would like to set up a shared file server using apache and mod_dav_fs 
to share files with windows and linux clients using the dav protocol 
over an SSL connection that is authenticated using client certificates. 
 The one thing it seems I can not do is get apache to impersonate the 
user when accessing the filesystem so that the user will be using their 
permissions to access the filesystem and not 'www-data'.


Does anyone have any thoughts on this?



Re: Deleting the async-dev branch

2005-11-21 Thread Phillip Susi
Having a thread block to page in data that isn't currently in the cache 
is fine, as long as other threads are able to handle all the other 
requests.  What I would really like to see is support for zero copy 
sends without using sendfile().  Right now I believe that apache is 
calling sendfile() on small chunks of data at a time, so that it can 
make sure the transfer is not stalled.  This tends to let the pipeline 
stall for short periods between each sendfile() call.  Also sendfile() 
can not be used at all to send data that is not in a file on disk.  For 
instance, the output of a cgi script, or data that has been encrypted to 
send over SSL.


On NT you can direct the kernel to set the send buffer size to zero ( 
setsockopt I think it was ) and that will prevent it from making the 
user->kernel copy when you send.  To keep the pipeline full you just 
have to overlap multiple async send requests.  This allows you to get 
sendfile() performance without its drawbacks.  I hope to one day see 
linux and apache support this.


I have wondered though, what it would be like to be able to 
asynchronously page fault in the cache pages to send, rather than 
relying on the send() call to synchronously fault in the pages if 
needed.  That would save threads from blocking on disk IO, which might 
be nice.


Brian Pane wrote:

The async write completion stuff that's now in the trunk does nonblocking
network writes, and it uses sendfile to do zero-copy transfer from the 
filesystem
cache to the network on platforms that support it, but it doesn't try to 
do file

I/O asynchronously.  Thus it's possible for the current code to block on a
disk read while trying to do a nonblocking socket write.  (The Event MPM
compensates for this by allowing multiple threads to do sendfile calls on
different sockets concurrently; if one thread happens to block on a
physical disk read, it doesn't delay other connections.)  Having a means
of doing nonblocking filesystem I/O, via either aio or something like
mincore checks, might be a nice addition.

Brian






Re: Deleting the async-dev branch

2005-11-20 Thread Phillip Susi
Actually you have to use a peg revision, not -r.  If you try -r to look 
at the deleted branch, svn tries to find the named directory in the HEAD 
revision, then look up it's state as it existed in rev x.  This will 
fail since it does not exist in the head revision, so instead you have 
to do something like:


svn list http://repos/svn/[EMAIL PROTECTED]

svn list -r x http://repos/svn/deleted fails because deleted does not 
exist anymore.


By the way, what exactly is the async-dev branch and what are the future 
plans for it?  I actually have spent this entire weekend doing research 
on the current state of support of async IO in linux and modified the dd 
program to use overlapped async IO when you pass it the iflag=direct 
option.  It turned out that glibc's posix aio library simply performs 
sync IO in a background thread, so I ended up hacking together a basic 
aio implementation that directly uses the kernel's aio support.  I'm 
still testing it but it looks like it yielded substantial cpu savings 
and throughput increases by keeping the drive directly reading into the 
userspace buffers.  DMA transfers sure beat memcpy()s.


I have been a big proponent of zero copy async IO for several years now 
so I am very interested in projects to further it's support and use.


Max Bowsher wrote:


Definitely not!

You can always look at any deleted thing by looking back into the
history with an appropriate svn --revision (-r) argument.

The only significant effect of deleting a branch is to unequivocally
indicate to everyone that a branch's sole reason for existence is now
historical record, rather than active development.

Max.







Re: pgp trust for https?

2005-11-09 Thread Phillip Susi
Exactly.  It doesn't matter where you live and what government you live 
under, they issue you some form(s) of identification.  A digital 
certificate is simply a digital form of identification.  Why should the 
government keep using ink and paper documents for identification when 
digital ones are much better?


I for one will be glad when I can just get a certificate in my email 
instead of waiting in line for 5 hours at the department of motor 
vehicles to renew my driver's license.


Boyle Owen wrote:

-Original Message-
From: Nick Kew [mailto:[EMAIL PROTECTED]


... Personally, I feel this role belongs in the government.

Any particular government?  A few years ago I'd probably have agreed.
With the most blatently corrupt government in living memory, that has
less appeal.


At first I thought this statment was inappropriate on an international list 
where we don't know where anybody lives. But then I realised it applies equally 
well to every government on the planet...

Except, maybe, Iceland?

Rgds,
Owen Boyle
Disclaimer: Any disclaimer attached to this message may be ignored. 




Re: pgp trust for https?

2005-11-06 Thread Phillip Susi




Nick Kew wrote:

  Huh?  The same person who installs the cert now.  It's just a different
signature.  And for those who want a certificate authority, have such
authorities (the more the better) sign *their* PGP keys.

  
  
>From whomsoever is responsible for it.  Maybe even more than one
individual, in the case of an org with lots of techies.

  
  
I'll sign my server.  Same as I'll sign an httpd tarball if I roll one
for public consumption.  You sign your server.  Where's the problem?

  
  
I don't want to get involved in something where I have nothing substantial
and new to contribute.  "Encourage others to trust you" is a marketing
job of which I would be totally incapable.

  
  
  



Now you seem to be talking about self signed certificates.  There are
sites out there that use them, but the problem is, they provide no
authentication.  Sure, they allow encrypted communications, and if you
save the certificate, then you can be sure that you are connecting to
the same server the next time ( preventing man in the middle attacks )
but there's nothing to prevent a man in the middle the first time.  

Another example you mentioned is signing tarballs you make.  Well if
your certificate is not signed by a trusted third party, such as
verisign or whomever, then how am I to know that it was YOU who signed
it and not some random guy claiming to be you?  Anyone can make a
keypair, the signature of the CA is what authenticates it.  





  Any particular government?  A few years ago I'd probably have agreed.
With the most blatently corrupt government in living memory, that has
less appeal.

  


True, but if things are that bad then you have bigger problems to worry
about.  As long as you still trust them to issue forms of paper
identification, then digital ones should be no different.  Why get a
paper ID from the government to show to a CA to prove you are you, so
they can sign your cert, rather than just have the gov't sign it
directly?


  
  
Sure.  I do trust my own key, and those of quite a number of other people,
including, for example, most of those with whom I would expect to share
an SVN repository for development work.  That's the kind of application
where a PGP-signed server key is a clear winner.
  


How so?  Again, in a small group of people who know each other in real
life and have a secure method of exchanting certificates, that works
fine, but this breaks down when you have a larger group of people who
do not know each other in real life and can swap their certs on disks
when they see each other.  


  
And when my browser indicates that I don't trust a key, I can investigate
in detail by fetching the public key and its signature(s), and make whatever
other checks I see fit.  Exactly the same as when I download a package
from the 'net.

  

Yes... and you can do that with the x.509 certificates that are used
for https/SSL, so what's pgp needed for?






Re: pgp trust for https?

2005-11-06 Thread Phillip Susi

Nick Kew wrote:


Why would anyone have to do that?  I'll trust a server as much as I trust
the PGP key of the person who signed it.  That's the same as trusting
an httpd download because it's signed by someone whose key I trust.

 



The question then is who is going to sign?  You seem to be sugesting 
that a server accept signatures from anyone and everyone, and you would 
only trust the server if someone that you know and trust had already 
decided to trust the server and sign their certificate.  That still 
leaves you in a position of either deciding to implicitly trust the site 
and sign it's certificate, which then causes all of your friends to 
trust it, or trusting the opinion of your friend who already decided ( 
based on what exactly? ) to trust the site and sign their cert. 

That doesn't provide much in the way of security, and is impossible to 
maintain -- a site can't accept a million signatures on their 
certificate from everyone who feels like signing it. 


It's usually signed by verispam, who make a habit of engaging in some
very nasty business practices, from spamming to holding the 'net to
ransom.  They also bought the main competitor (thawte), leaving us
short of competition amongst those widely recognised by browsers.

With PGP it's my own trust, not theirs.

 



You are quite free to set up your own root CA and encorage others to 
trust you.  You are also free to decide to NOT trust certificates signed 
by verispam.  Personally, I feel this role belongs in the government.  
That's where you get your birth certificate, driver's license, social 
security card, and other forms of 'official' ID.  They may as well get 
rid of all the paper ID and just start issuing digital certificates. 


I seldom use pgp for email (and I hate it when people sign messages
posted to a list like this).  But I always use it to verify software I 
download from the 'net.  And, unlike https, it tells me every time

whether or not *I* trust the digital signature.


 



How do you decide that such a signature is trustworthy and valid?  You 
either have to know about their public key a priori, or know ( and trust 
) another one that signed theirs, otherwise, you're just guessing that 
you can trust it. 





Re: pgp trust for https?

2005-11-05 Thread Phillip Susi
The big reason that comes to my mind is that users don't want to have to 
implicitly trust the server from the start, then register on the site by 
uploading their own key before secure communications can begin. 

The big advantage of a public certificate infrastructure is that the 
rest of us can trust someone we have never met before ( i.e. an https 
server you have never visited before ) because they present us with a 
certificate that is signed by a trusted third party.  The server proves 
to the client that it is who it says it is, and the client can 
optionally prove its identity to the server with a client certificate.  
Either one allows all communications to be encrypted and you don't have 
to exchange private information first to do it. 

Personally, I don't understand the need for pgp.  You can sign and 
encrypt email or IPsec just fine using x.509 certificates. 


Nick Kew wrote:


We have grown accustomed to two separate trust mechanisms
on the 'net; server certs signed by some authority, or the PGP
web of trust.

I would like to be able to use PGP trust over the web.  That would
mean (something like) installing a certificate on the server, and
signing it with my PGP key.  My browser, having access to and
trust in, my public key, will then trust the server.  No need for
any dealings with verispam.

Is there any technical reason this shouldn't happen?

 





Re: async write completion prototype

2005-10-10 Thread Phillip Susi
Non blocking is not async IO.  It is not really possible to perform zero 
copy IO with non blocking IO semantics, you must have full async IO to 
issue multiple pending requests.


Brian Akins wrote:

Phillip Susi wrote:

As an alternative, you can bypass the cache and do direct async IO to 
the disk with zero copies.  IIRC, this is supported on linux with the 
O_DIRECT flag.  Doing this though, means that you will need to handle 
caching yourself, which might not be such a good idea.  Does Linux not 
support O_DIRECT on sockets?



Can you not just set the socket to non-blocking using O_NONBLOCK?





Re: async write completion prototype

2005-10-10 Thread Phillip Susi
On NT you can set the kernel buffer size on the socket to 0 ( with 
setockopt() or ioctlsocket()? ), and the NIC can DMA directly from the 
user buffers to send rather than copy to kernel space.  This of course, 
requires that you keep more than one pending async operation so the nic 
allways has a buffer availible so it can keep the line saturated.  If 
you memory map the source file from the disk, then zero copy IO can be 
done entirely from user space.  Is this optimization not availible on 
linux or freebsd?


As an alternative, you can bypass the cache and do direct async IO to 
the disk with zero copies.  IIRC, this is supported on linux with the 
O_DIRECT flag.  Doing this though, means that you will need to handle 
caching yourself, which might not be such a good idea.  Does Linux not 
support O_DIRECT on sockets?


By using this technique I have been able to achieve TransmitFile() 
performance levels entirely from user space, without any of the 
drawbacks of TransmitFile().  Specifically virtually zero cpu time is 
needed to saturate multiple 100 MBps links, pushing 11,820 KB/s, 
progress is known the entire time and can be canceled at any time, and a 
small handfull of threads can service thousonds of clients.


Paul Querna wrote:

Phillip Susi wrote:

On what OS?  Linux?  NT supports async IO on sockets rather nicely, as 
does FreeBSD iirc.



The event MPM doesn't run on NT at all, only Unixes.

Yes, FreeBSD (and linux) support async_write().. But this requires that 
you read the file off of disk, and into a buffer, and then copy it back 
into the kernel.  With a non-blocking sendfile, we can avoid all the 
data copying (aka 'zero copy'), and let the kernel do everything itself.


There is currently no such thing as 'async sendfile', which would be the 
perfect solution for this use case.  There have been various people 
mentioning it as an idea, but no one has gone out and done it.



-Paul





Re: async write completion prototype

2005-10-10 Thread Phillip Susi
On what OS?  Linux?  NT supports async IO on sockets rather nicely, as 
does FreeBSD iirc.


Paul Querna wrote:

Phillip Susi wrote:

Nicely done.  Have you done any benchmarking to see if this improved 
performance as one would expect?  Would it be much more work to use 
true async IO instead of non blocking IO and polling?  What about 
doing the same for reads, as well as writes?




All current async_io methods require you to read the data off disk, ie 
there is no async_sendfile()


Reads are much harder in the current httpd.  There are several core 
functions that would need to be rewritten first.


-Paul





Re: async write completion prototype

2005-10-10 Thread Phillip Susi
Nicely done.  Have you done any benchmarking to see if this improved 
performance as one would expect?  Would it be much more work to use true 
async IO instead of non blocking IO and polling?  What about doing the 
same for reads, as well as writes?


Brian Pane wrote:

With the batch of commits I did this weekend, the Event MPM in
the async-dev Subversion branch now does write completion
in a nonblocking manner.  Once an entire response has been
generated and passed to the output filter chain, the MPM's
poller/listener thread watches the connection for writability
events.  When the connection becomes writable, the poller
thread sends it to one of the worker threads, which writes
some more output.

At this point, the event-handling code is ready for testing and
review by other developers.

The main changes on the async-dev branch (compared
to the 2.3-dev trunk) are:

1. ap_core_output_filter: rewrite to do nonblocking writes
   whenever possible.

2. core, http module, and mod_logio: removed the generation
   of flush buckets where possible.

3. request cleanup and logging: the logger phase and
   subsequent destruction of the request's pool are now
   triggered by the destruction of an End-Of-Request
   bucket in the core output filter.

4. event MPM: asynchronous handling of CONN_STATE_WRITE_COMPLETION.

There are several more things that need to be fixed in order
to make the asynchronous write completion useful in a
production release of httpd-2.x:

- The main pollset in the Event MPM currently is sized to
  hold up to one socket descriptor per worker thread.  With
  asynchronous keepalives and write completion, the pollset
  should accommodate many descriptors per thread.

- The logic for starting more child processes, which Event
  inherited from Worker, is based on assumptions about
  the number of concurrent connections being equal to
  the number of threads.  These assumptions aren't valid
  for a multiple-connections-per-thread MPM.

- Similarly, there may be some changes needed in the
  flow control logic that the listener thread uses to decide
  whether it can do an accept.

- The scoreboard probably needs a redesign.

- It may be valuable to have a separate thread pool to
  run handlers that do arbitrarily lengthy processing, such
  as mod_perl and mod_php.

Brian





Re: Bandwidth Limit on Windows

2005-09-15 Thread Phillip Susi

William A. Rowe, Jr. wrote:


It's called swap-death.  Forcing the system to begin paging both mapped
files (which aren't swapped to the swap file since their contents are
unchanged, e.g. your programs and dll files' code pages) and ram (which
is thrown at the swap file) slows it down considerably.

Bill



This is not true.  Mapping a full view of a large file does not cause 
any data to actually be read, and hence, any memory to be used.  Data is 
only read from disk as the pages are faulted in on access.


William A. Rowe, Jr. wrote:
>
> Yup.  It's definately worth considering your alternatives.  It's also
> worth looking at what socket-level timeouts exist, or if we can start
> plugging in some higher-level timeout within the MPM.  When a process
> does nothing in certain phases of the request cycle after some timeout,
> simply close the socket from underneath APR.
>
> So how to figure out if we are making progress?  I'm unsure and this
> all deserves another look at the current state of the Win32 sockets API,
> I haven't fallen down that well in a good long time :)
>
> Of course async would be wonderful, at some point, as apache 3.0 starts
> to draw interest.
>
> Bill
>

I agree, trying to solve this kind of DoS attack in the httpd is kind of 
futile.  Also async IO is the way to go imho, and I've been spending 
some spare time lately trying to figure out how hard it would be to 
modify apache to take advantage of async IO, which would allow for the 
kind of zero copy IO that sendfile and TransmitFile were created for, 
without the problems they cause.  It also allows a small number of 
threads to scale to serving a very large number of clients.




Re: Bandwidth Limit on Windows

2005-09-15 Thread Phillip Susi
When I asked about this a month or two ago, someone explained that 
Apache uses TransmitFile() to implement sendfile but in a weird way that 
makes it really, really slow.  Disabling sendfile in the apache config, 
and just using the mmap module gives the best throughput on windows.


I don't know if apache mmaps the entire file if it is large, but if it 
is very large, I don't see how it can as you only have a limited amount 
of virtual memory to work with.  Unless you run out of virtual memory ( 
i.e. lots of mapped files, or a single file that is several gigs ) then 
mapping the entire file isn't really a problem.


William A. Rowe, Jr. wrote:

Phillip Susi wrote:

Probably has to do with sendfile, always turn it off on windows.  I 
was seeing low throughput as well until I turned off sendfile, then 
over a 100 MBps network I got over 11,000 KB/s.


Apparently apache has a rather horrible emulation of sendfile on 
windows for some reason or another.



Sendfile != MMap.

In the case of Sendfile, it's using the Win32 API TransmitFile().  In
the case of MMap, the file is mapped into virtual memory, and loaded
into physical memory.  Apache prefers Sendfile, if both are available,
IIRC.

However, the way Apache/APR MMAPs the entire file is quite probably
sub-optimal for some resources.

Bill





Re: Bandwidth Limit on Windows

2005-09-15 Thread Phillip Susi
Probably has to do with sendfile, always turn it off on windows.  I was 
seeing low throughput as well until I turned off sendfile, then over a 
100 MBps network I got over 11,000 KB/s.


Apparently apache has a rather horrible emulation of sendfile on windows 
for some reason or another.


Ivan Barrera A. wrote:

Hi All.

I'm working on a bandwidth module for apache.
My question is as follows :

Recently, a user asked me why on windows, he can get higher bandwidth
only with mod_bw. Without it, he can get about 1.6MB/s. Usind the mod,
and setting unlimited , he gets ~9MB/s.

I'm not on my develop system, so i haven't tested.

The question is , why this happens ? does it have relation with the
EnableMMAP and Senfile options ?

Thanks.







Re: ap_process_connection

2005-08-26 Thread Phillip Susi
Except that it isn't REALLY defined there.  The use of all these macros
is giving me a headache.  There are a number of papers out there on why
this kind of macro (ab)use is bad, mainly because it makes it impossible
to understand the code.

server/connection.c has this:

AP_IMPLEMENT_HOOK_RUN_FIRST(int,process_connection,(conn_rec
*c),(c),DECLINED)

Well what the heck does that do?  There's no function body defined
there.  I'm guessing that macro does some tricks to achieve something
like polymorphism in C, binding an object to a set of functions that
should be called in a given event.  I can't see what function is being
bound to the process_connection event though.

When I looked up the definition of this macro, it is itself defined
using another macro.  My eyes have now gone crossed.  Maybe I'll try to
understand these evil macros later.

C.G. Lee wrote:
>  >Where is this function implemented? I can't seem to find it.
>  
> When I was exploring the Apache2 source code, I couldn't understand the 
> ap_process_connection(). Why is it there? What does it doing?? 
>  
> I don't know what you're doing, but the function ap_process_connection 
> is in the server/connection.c.
>  
> 


ap_process_connection

2005-08-25 Thread Phillip Susi

Where is this function implemented?  I can't seem to find it.

I found this comment in a header:


Note: In the future it will be possible for ap_process_connection
to return to the MPM prior to finishing the entire connection; and
the MPM will proceed with asynchronous handling for the connection;
in the future the MPM may call ap_process_connection again -- but
does not guarantee it will occur on the same thread as the first call.


And I was wondering if this has been done yet, but I can't locate the 
implementation of ap_process_connection to find out.  I'm trying to see 
if I can modify it to make asynchronous socket IO calls and return to 
the MPM before the entire conversation is complete.  That way the 
winnt_mpm can use a small pool of worker threads to perform zero copy 
async IO using the IO completion port and become more fast, efficient, 
and scalable than it currently is.




Re: SSL downloads faster than non SSL?

2005-08-03 Thread Phillip Susi

William A. Rowe, Jr. wrote:


In the APR library, yes, we translate 'apr_sendfile' to TransmitFile()
on win32.  Some other magic occurs to obtain a file handle which can 
be passed to TransmitFile.  But there are enough flaws in the TF() api

that perhaps this would be better defaulted to 'off'.

 

Really?  Are you quite sure?  I wonder what's hosing it all up.  Once 
you hand TransmitFile() the socket and file handles, it should blast the 
file over the network nice and fast. 


That is also available.  As you are aware TransmitFile() lives entirely
in the kernel, so there are far fewer user<->kernel mode transitions.
 



Yes, there are fewer user-kernel transitions, but not that many and they 
are relatively inexpensive.  By far the largest savings that 
TransmitFile() gains is from not having to copy the data from user to 
kernel buffers before it can be sent over the network.  A conventional 
read() and send() call pair ends up making a copy from kernel FS buffer 
memory to user buffer, then back to kernel socket buffer memory.  That's 
where most of the CPU time is wasted.  A few years ago I wrote an FTP 
server and tried using both TransmitFile() and using overlapped IO.  By 
disabling kernel buffering on the socket and memory mapping 2 32 KB 
views of the file at once and overlapping both sends, I was able to 
match both the network throughput and low CPU load of TransmitFile(). 

Specifically, I developed this on a PII-233 system with two fast 
ethernet NICs installed.  Using several other FTP servers popular at the 
time, I was only able to manage around 5500 KB/s though one NIC using 
100% of the CPU.  Using either TransmitFile() or zero copy overlapped 
IO, I was able to push 11,820 KB/s over one NIC and 8,500 KB/s over the 
other ( not as good of a card apparently ) simultaneously using 1% of 
the CPU.  There was no noticeable difference between TransmitFile() and 
the overlapped IO.  Oh, and also I had to find a registry setting to 
make TransmitFile() behave on my NT 4 workstation system the way it does 
by default on NT server to get it to perform well.  By default on 
workstation it was not nearly so good. 


But if you turn off sendfile, and leave mmap on, Win32 (on Apache 2.0,
but not back in Apache 1.3) does use memory mapped I/O.

You suggest this works with SSL to create zero-copy?  That's not quite
correct, since there is the entire translation phase required.

 



My understanding is that the current code will memory map the data file, 
optionally encrypt it with SSL, and then call a conventional send().  
Using send() on a memory mapped file view instead of read() eliminates 
one copy, but there is still another one made when you call send(), so 
you're only half way there.  To eliminate that second copy you have to 
ask the kernel to set the socket buffer size to 0 ( I can't remember if 
that was done with setsockopt or ioctlsocket ) and then use overlapped 
IO ( preferably with IO completion ports for notification ) to give the 
kernel multiple pending buffers to send.  That way you eliminate the 
second buffer copy and the NIC always has a locked buffer from which it 
can DMA. 


:)  We seriously appreciate all efforts.  If you are very familiar with
Win32 internals, the mpm_winnt.c does need work; I hope to change this
mpm to follow unix in setting up/tearing down threads when we hit min
and max thresholds.  Obviously many other things in the (fairly simple)
win32 implementation can be improved.  Support for multiple processes
is high on the list, since a fault in a single thread brings down the
process and many established connections, and introduces a large latency 
until the next worker process is respawned and accepting connections.


Bill
 



Well, ideally you just need a small number of worker threads using an IO 
completion port.  This yields much better results than allocating one 
thread to each request, even if those threads are created in advance.  I 
have been trying to gain some understanding of the Apache 2 bucket 
brigade system, but it's been a bit difficult just perusing the docs on 
the web site in my spare time.  From what I've been able to pick up so 
far though, it looks like the various processing stages have the option 
to either hold onto a bucket to process asynchronously, process the 
bucket synchronously, or simply pass it down to the next layer 
immediately.  What I have not been able to figure out is if any of the 
processing layers tend to make system calls to block the thread.  
Provided that you don't do very much to block the thread while 
processing a request, then if you were to use an IO completion port 
model, a small handful of threads could service potentially thousands of 
requests at once.


Also a fault in one thread does not have to kill the entire process, you 
can catch the fault and handle it more gracefully.  I'd love to dig into 
mpm_winnt but at the moment my plate is a bit full.  Maybe in another 
month or two I'll be able t

Re: SSL downloads faster than non SSL?

2005-08-02 Thread Phillip Susi
That's what it was, thanks.  Say, the name says 'Sendfile' but that 
doesn't exist on win32.  Does apache call TransmitFile() instead or does 
it use some very poor library emulation of sendfile()?  If it were using 
TransmitFile() I would expect it to not have any trouble saturating the 
network. 

FYI, I used wget to download the file without SSL and with sendfile 
disabled and it took 55 seconds and reported a rate of 11 MB/s.  I used 
a copy command to download the file from the command prompt via the SMB 
mapped drive and it took 90 seconds.  I wonder what the heck the deal is 
with all that fancy kernel mode redirector stuff not working out the way 
they designed?


I don't even see why the option for sendfile should exist on the win32 
build, seeing as how you can achieve the same performance using memory 
mapped zero copy overlapped IO, which also would work with non disk 
files, such as an SSL encrypted connection. 

One of these days I'm going to have to take a little vacation from work 
and write an MPM to do just that. 


Rici Lake wrote:


Try setting

EnableSendfile off

in your apache config file and rerunning the tests. There have been 
other reports of surprisingly slow transfer speeds with sendfile 
enabled on Windows. This won't affect SSL, of course, since the SSL 
transfer cannot use sendfile.





SSL downloads faster than non SSL?

2005-08-02 Thread Phillip Susi

I decided to do some informal benchmark comparisons between using
windows SMBFS and apache/webdav for file transfers.  I ended up finding
that apache is actually faster at sending files over an SSL connection
than a plain connection.  I downloaded a ~600 meg test file from the
server using windows explorer webfolders, IE, and firefox.  Firefox
downloads the file in the 4000-5000 KB/s range when using an SSL
connection.  IE gets over 10,000 KB/s downloading over the secure
connection.  Both only are able to download at 300-600 KB/s using the
non SSL connection though.  This is, of course, all done over a 100 Mbps
ethernet network that is minimally loaded, and I repeated the test a few
times, clearing the browsers caches each time.

Needless to say, this is very strange.  My client machine is running
windows 2000 pro, and the server is running windows 2003 server, with
apache 2.0.52 installed.  Does anyone have any ideas to possibly explain
and correct these results?





Re: Single-threaded/non-blocking frontend and multi-threaded/processed backend

2005-05-10 Thread Phillip Susi
Why have a separate 'frontend' and 'backend'?
One of these days when I get some spare time, I plan on rewriting 
mpm_winnt to use a small pool of threads which will use unbuffered zero 
copy overlapped IO and an IO completion port as the notification 
mechanism.  The pool of worker threads will wait on the IOCP for 
incoming data and as data comes in on any of the connected sockets, it 
wakes one thread to process the data.  The MPM would then pass the data 
on down the handler/filter chain, and it is currently my understanding 
that most of the time, the filters in the chain don't make other 
blocking system calls, but rather end up calling back into the apr/mpm 
to get more data, at which point the mpm would be free to handle IO on 
other sockets.

This system would allow fast and efficient IO on the level of 
TransmitFile(), only not restricted to sending unmodified on disk files. 
 I once wrote a simple FTP server using this mechanism and it was able 
to stream a large file over a fast ethernet network at 11,820 KB/s using 
virtually zero cpu time.  I believe that the apache2 bucket brigade 
design is nearly ideal for this system as the various filters and 
handlers can produce and/or modify data as they see fit, and it will 
still be sent and received over the network without the need for the 
user/kernel buffer copies.  It would also allow a small pool of worker 
threads to serve tens or even hundreds of thousands of clients.  I 
believe that BSD has a similar system called kqueue and posix aio.  To 
my knowledge, Linux has no equivalent.

Olaf van der Spek wrote:
On 4/13/05, Paul Querna <[EMAIL PROTECTED]> wrote:
Olaf van der Spek wrote:
Hi,
I originally posted this feature request to the bug tracker, but Joe
Orton suggested it'd post it here instead, so here it is.
I'd like to see a new 'MPM' that basically works like this:
A single or few-threaded, non-blocking frontend that accepts incoming
connections, receives requests and sends responses.
A multi-threaded or multi-processed backend that processes requests, generates
responses and passes the responses to the frontend for sending to the client.
The advantage of this scheme would be that you can handle a (much) bigger
number of clients/persistent connections with less resources.
The event MPM is a step in the right direction, but not 'good' enough.
A scenario with multiple backends may be possible to, for example one
running as user A, one as user B, etc.
Or one multi-threaded backend for static content and one
multi-processed backend for dynamic content (PHP).
http://issues.apache.org/bugzilla/show_bug.cgi?id=32243
I agree with just about everything you said, at least in concept.
I am willing to help out, but I don't have the time or resources to head
up this type of project right now.  If you do the initial work, I would
love to help get such an MPM into the mainline tree.

Wouldn't it be possible to start this 'transition' by writing a module
that forwards the request to a separate backend process?
That way, you could safely run the threaded MPM and still use PHP in a safe way.



Re: Post-2.2 Thoughts

2005-05-09 Thread Phillip Susi
Ben Collins-Sussman wrote:
Actually, there have been jokes in the past about performing  
Subversion's native 'svnserve' protocol buried in HTTP requests.  But  
maybe it's no longer a joke... if this abstraction happens, then  
maybe apache could be taught to speak it natively!


Why bother?  the native protocol is only for connecting to the light 
weight svnserver.  If you are going to use apache as the server anyhow, 
then using webdav/deltav as the protocol is the logical choice. 

Matthieu Estrade wrote:
I am interested by a mod_smtp module, just to be able to have 
http(synchronous) and smtp(asynchronous) possible web service support. 
it could be easy then to code common module for both protocol and 
doing web services.

Neat idea.  If you are implementing a web service using XML-RPC type 
communication, sometimes sending the messages asynchronously is 
beneficial, and smtp is ideal for that. 




Re: Post-2.2 Thoughts

2005-05-08 Thread Phillip Susi
Paul Querna wrote:
So, here is my short-list-made-up-this-afternoon of things I would like
to look at doing after 2.2 is branched/released as GA.  I welcome
additions too.
1) Eliminate the HTTP in HTTPD.  I would like to be able to compile
httpd with --disable-http.  Currently the 'http core' is half mangled
with the 'core core'.
 

I like this idea.  I think one of the best and most important things 
that apache is doing these days is trying to migrate away from being a 
web server and towards a generic server platform.  Eventually I would 
like to be able to run apache to serve web content, email ( smtp as well 
as pop3 ), and subversion repositories and other file shares ( possible 
now ). 

2) Add a major Protocol Module.  I would like to add a major protocol
module, besides http, into the mainline distribution.  I believe adding
a FTP Server Module would be the best one to start with.  Many of the
paradigms are the same, but it is different enough to help with Goal #1,
eliminating the dependence on http.
 

Sounds good.  FTP would offer the path of least resistance and help to 
eliminate the dependence on http.  I do not however, think that it has 
long term use in its own right, because ftp is being replaced with 
http/webdav. 

3) Asynchronous Connection Level Filters.  This one needs much more
thought, but the basic goal is to allow the Event MPM to do all of the
initial HTTP parsing without a worker. (eg, reading the headers).  Once
the headers are read, it would pass the request off to a worker.  I
don't think trying to make the different Hooks or Handlers Asynchronous
will be that helpful, but Connection Level Filters would need a
re-design to handle this idea.  I don't personally have a clear picture
how this would all work yet, and really want the input of everyone else.
 

I'm not quite sure what you mean here, but that is probably because I'm 
new to hacking on apache, so forgive me if this is a stupid question.  
As I understand it now, the MPM receives data from the socket and places 
it into a bucket, attaches the bucket to the brigade, and calls the 
lowest level filter, passing it the bucket.  Each filter in the chain 
then can modify the brigade, and can pass the bucket to the next filter, 
or possibly call back down the chain to the mpm to get more data.  
Eventually the brigade makes it to the handler which forms a reply and 
passes it back down the chain to the mpm to send out on the socket.  Is 
this about correct?  Where do workers fit in?

4) Configuration.  I would like to look at enabling different
configuration systems.  A major change would be to allow more runtime
configuration changes.  Others have mentioned using XML -- Im not sold
on it, and I don't think XML should be an exclusive method, but perhaps
an alternative file format.
I don't think I can do all of these myself.  If we did all of them, it
could be a '3.0'.  I am not sure if the project is ready for a 3.0 yet,
and in some ways, I think after 2.2 we should pick a few key things and
target them for '2.4'. But, it doesn't matter, this is open source, not
some commercial product :)
These are the types of things I would love to talk about at the
Hackathon before ApacheCon this summer. I am seeding them now, in the
hope that other minds will start hacking on new ideas for httpd.
-Paul