Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-28 Thread Thomas Hartman
As far as I can tell, stepcut was probably correct in his diagnosis before.

I was getting two processes started because of a stray cron job.

2010/2/26 Thomas Hartman tphya...@gmail.com:
 Indeed, the error occurs when two processes are running at the same
 time. One process isn't serving anything, and the other is just
 looping, reported that it can't stop because the port is taken, and
 repeating ad infinitum.

 The loop I have is

 ulimit -v 15
 while true; do
  echo starting patchtag: `date`
 ./dist/build/patchtagserver/patchtagserver --port=80 --
  echo patchtag exited: `date`
  echo patchtag exited: `date`  patch-tag.log
  rm -f _local/patchtagserver_state.lock

 The ulimit is so that it will die after consuming a reasonable amount
 of memory. (I seem to have a memory leak somewhere, but it takes many
 hours to get up to 150M.)

 Everything else looks pretty standard to me. It seems to work as
 designed most of the time, but occasionally will result in two
 processes. I'm baffled as to how this happens.

 If anybody has any idea, I would love to hear them.

 2010/2/26 Jeremy Shaw jer...@n-heptane.com:
 Hello,
 It will be interesting to see if that makes any difference -- it shouldn't.
 In happstack-server we use 'listenOn'. According to the documentation
 listenOn already sets ReuseAddr:
 http://hackage.haskell.org/packages/archive/network/2.2.1.7/doc/html/Network.html#v%3AlistenOn
 A quick look at the source code confirms it is calling:
    setSocketOption sock ReuseAddr 1
 It sounds to be like you are getting the socket in use error because you
 have 2 processes running. Seems like the first one hasn't really died, but
 the second one is started anyway. How is it that your loop starts a second
 server when the first one has not finished?
 - jeremy
 On Fri, Feb 26, 2010 at 1:43 PM, Thomas Hartman tphya...@gmail.com wrote:

 Thanks, I altered my top level request handler as follows

 mysmartserver conf h stateProxy = do
      socket - bindPort conf

       I added (setSocketOption socket ReuseAddr 1) here
       Should this be added in a comment, or even in function code,
 in Happstack.Server.SimpleHTTP? What are the tradeoffs, when would you
 *not* want ot use ReuseAddr?)

      webserverTid - forkIO $ simpleHTTPWithSocket socket conf h
      putStrLn . ( starting happs server ++ ) = time

      control - startSystemState stateProxy -- start the HAppS state
 system
      putStrLn . ( happs state started ++ ) = time

      waitForTermination
      killThread webserverTid
      stateShutdown control

 I can't replicate the error reliably so I won't know if it actually
 fixed the problem for a while, therefore just leaving this cookie
 crumb trail in case others may find it helpful.

 2010/2/26 Brandon S. Allbery KF8NH allb...@ece.cmu.edu:
  On Feb 26, 2010, at 04:28 , Thomas Hartman wrote:
 
  me: Like mightybyte, I run my app in a shell loop that will just
  restart it after a crash. But every once in a while it won't restart
  because of the busy socket and I need to do a manual restart, killing
  multiple processes (usually 2).
 
  the error I get is:
 
  bind: resource busy (Address already in use)
 
  This is on application restart?  It's not out of file descriptors, it's
  just
  the system keeping the socket around (netstat will show it in TIME_WAIT,
  or
  possibly in a shutdown negotiation state such as LAST_ACK, FIN_WAIT,
  etc.
   TCP lacks *reliable* socket shutdown negotiation).
 
  You want to configure the socket with SO_REUSEADDR before trying to bind
  it
  (setSocketOption socket ReuseAddr 1).
 
  As to the portable version of sendfile, it's because not all systems
  offer a
  sendfile() system call.  Linux and *BSD do, and can use the native
  implementation; the portable version emulates sendfile() when it doesn't
  exist, at the price of additional CPU usage/system load (sendfile()
  having
  been created specifically to reduce system load in the common case for
  web
  servers).
 
  --
  brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com
  system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu
  electrical and computer engineering, carnegie mellon university    KF8NH
 
 
 



___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-26 Thread Thomas Hartman
I believe this might be causing problems for me with patch-tag.

me: Like mightybyte, I run my app in a shell loop that will just
restart it after a crash. But every once in a while it won't restart
because of the busy socket and I need to do a manual restart, killing
multiple processes (usually 2).

the error I get is:

bind: resource busy (Address already in use)

This sounds like what is being discussed. I have some questions.

 the server will eventually run out of file descriptors

How can I tell if I am running out of file descriptors?

 If I use the portable build of the sendfile package,  everything works fine 
 for hours and hours of this happening.

What is the portable build and how do I use it? Do you eventually have
troubles here as well? What is the reason to use the linux native
build then? speed / more connections per second?

thanks for your help!

2010/2/4 Bardur Arantsson s...@scientician.net:
 Hi all,

 I've been using the sendfile package off Hackage, but it seems to be leaking
 file descriptors when using the Linux-native build.

 What's happening in my specific case is the following:

   1) client requests a range of a file
   2) server starts sending the range
   3) client disconnects before receiving the whole file

 This happens over and over with the client requesting different ranges of
 the file (so the client does make progress).

 If I use the portable build of the sendfile package, everything works fine
 for hours and hours of this happening.

 If I use the Linux-native build of the sendfile package, the server
 will eventually run out of file descriptors. According to lsof the files
 that are being kept open are the data files being sent in 2) above.

 This is on GHC 6.10.x (Ubuntu).

 Is anyone else seeing this? Anyone got any idea what's going on?

 Cheers,

 ___
 Haskell-Cafe mailing list
 Haskell-Cafe@haskell.org
 http://www.haskell.org/mailman/listinfo/haskell-cafe

___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-26 Thread Brandon S. Allbery KF8NH

On Feb 26, 2010, at 04:28 , Thomas Hartman wrote:

me: Like mightybyte, I run my app in a shell loop that will just
restart it after a crash. But every once in a while it won't restart
because of the busy socket and I need to do a manual restart, killing
multiple processes (usually 2).

the error I get is:

bind: resource busy (Address already in use)


This is on application restart?  It's not out of file descriptors,  
it's just the system keeping the socket around (netstat will show it  
in TIME_WAIT, or possibly in a shutdown negotiation state such as  
LAST_ACK, FIN_WAIT, etc.  TCP lacks *reliable* socket shutdown  
negotiation).


You want to configure the socket with SO_REUSEADDR before trying to  
bind it (setSocketOption socket ReuseAddr 1).


As to the portable version of sendfile, it's because not all systems  
offer a sendfile() system call.  Linux and *BSD do, and can use the  
native implementation; the portable version emulates sendfile() when  
it doesn't exist, at the price of additional CPU usage/system load  
(sendfile() having been created specifically to reduce system load in  
the common case for web servers).


--
brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com
system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu
electrical and computer engineering, carnegie mellon universityKF8NH




PGP.sig
Description: This is a digitally signed message part
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-26 Thread Thomas Hartman
Thanks, I altered my top level request handler as follows

mysmartserver conf h stateProxy = do
  socket - bindPort conf

   I added (setSocketOption socket ReuseAddr 1) here
   Should this be added in a comment, or even in function code,
in Happstack.Server.SimpleHTTP? What are the tradeoffs, when would you
*not* want ot use ReuseAddr?)

  webserverTid - forkIO $ simpleHTTPWithSocket socket conf h
  putStrLn . ( starting happs server ++ ) = time

  control - startSystemState stateProxy -- start the HAppS state
system
  putStrLn . ( happs state started ++ ) = time

  waitForTermination
  killThread webserverTid
  stateShutdown control

I can't replicate the error reliably so I won't know if it actually
fixed the problem for a while, therefore just leaving this cookie
crumb trail in case others may find it helpful.

2010/2/26 Brandon S. Allbery KF8NH allb...@ece.cmu.edu:
 On Feb 26, 2010, at 04:28 , Thomas Hartman wrote:

 me: Like mightybyte, I run my app in a shell loop that will just
 restart it after a crash. But every once in a while it won't restart
 because of the busy socket and I need to do a manual restart, killing
 multiple processes (usually 2).

 the error I get is:

 bind: resource busy (Address already in use)

 This is on application restart?  It's not out of file descriptors, it's just
 the system keeping the socket around (netstat will show it in TIME_WAIT, or
 possibly in a shutdown negotiation state such as LAST_ACK, FIN_WAIT, etc.
  TCP lacks *reliable* socket shutdown negotiation).

 You want to configure the socket with SO_REUSEADDR before trying to bind it
 (setSocketOption socket ReuseAddr 1).

 As to the portable version of sendfile, it's because not all systems offer a
 sendfile() system call.  Linux and *BSD do, and can use the native
 implementation; the portable version emulates sendfile() when it doesn't
 exist, at the price of additional CPU usage/system load (sendfile() having
 been created specifically to reduce system load in the common case for web
 servers).

 --
 brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com
 system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu
 electrical and computer engineering, carnegie mellon university    KF8NH



___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-26 Thread Brandon S. Allbery KF8NH

On Feb 26, 2010, at 14:43 , Thomas Hartman wrote:

in Happstack.Server.SimpleHTTP? What are the tradeoffs, when would you
*not* want ot use ReuseAddr?)


In the modern world, any server socket probably should use  
SO_REUSEADDR.  About the only modern use for the default would involve  
servers launched as a result of a port-knock or similar.


--
brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com
system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu
electrical and computer engineering, carnegie mellon universityKF8NH




PGP.sig
Description: This is a digitally signed message part
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-26 Thread Jeremy Shaw
Hello,

It will be interesting to see if that makes any difference -- it shouldn't.
In happstack-server we use 'listenOn'. According to the documentation
listenOn already sets ReuseAddr:

http://hackage.haskell.org/packages/archive/network/2.2.1.7/doc/html/Network.html#v%3AlistenOn

A quick look at the source code confirms it is calling:

setSocketOption sock ReuseAddr 1

It sounds to be like you are getting the socket in use error because you
have 2 processes running. Seems like the first one hasn't really died, but
the second one is started anyway. How is it that your loop starts a second
server when the first one has not finished?

- jeremy

On Fri, Feb 26, 2010 at 1:43 PM, Thomas Hartman tphya...@gmail.com wrote:

 Thanks, I altered my top level request handler as follows

 mysmartserver conf h stateProxy = do
  socket - bindPort conf

   I added (setSocketOption socket ReuseAddr 1) here
   Should this be added in a comment, or even in function code,
 in Happstack.Server.SimpleHTTP? What are the tradeoffs, when would you
 *not* want ot use ReuseAddr?)

  webserverTid - forkIO $ simpleHTTPWithSocket socket conf h
  putStrLn . ( starting happs server ++ ) = time

  control - startSystemState stateProxy -- start the HAppS state
 system
  putStrLn . ( happs state started ++ ) = time

  waitForTermination
  killThread webserverTid
  stateShutdown control

 I can't replicate the error reliably so I won't know if it actually
 fixed the problem for a while, therefore just leaving this cookie
 crumb trail in case others may find it helpful.

 2010/2/26 Brandon S. Allbery KF8NH allb...@ece.cmu.edu:
  On Feb 26, 2010, at 04:28 , Thomas Hartman wrote:
 
  me: Like mightybyte, I run my app in a shell loop that will just
  restart it after a crash. But every once in a while it won't restart
  because of the busy socket and I need to do a manual restart, killing
  multiple processes (usually 2).
 
  the error I get is:
 
  bind: resource busy (Address already in use)
 
  This is on application restart?  It's not out of file descriptors, it's
 just
  the system keeping the socket around (netstat will show it in TIME_WAIT,
 or
  possibly in a shutdown negotiation state such as LAST_ACK, FIN_WAIT, etc.
   TCP lacks *reliable* socket shutdown negotiation).
 
  You want to configure the socket with SO_REUSEADDR before trying to bind
 it
  (setSocketOption socket ReuseAddr 1).
 
  As to the portable version of sendfile, it's because not all systems
 offer a
  sendfile() system call.  Linux and *BSD do, and can use the native
  implementation; the portable version emulates sendfile() when it doesn't
  exist, at the price of additional CPU usage/system load (sendfile()
 having
  been created specifically to reduce system load in the common case for
 web
  servers).
 
  --
  brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com
  system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu
  electrical and computer engineering, carnegie mellon universityKF8NH
 
 
 

___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-26 Thread Thomas Hartman
Indeed, the error occurs when two processes are running at the same
time. One process isn't serving anything, and the other is just
looping, reported that it can't stop because the port is taken, and
repeating ad infinitum.

The loop I have is

ulimit -v 15
while true; do
  echo starting patchtag: `date`
./dist/build/patchtagserver/patchtagserver --port=80 --
  echo patchtag exited: `date`
  echo patchtag exited: `date`  patch-tag.log
  rm -f _local/patchtagserver_state.lock

The ulimit is so that it will die after consuming a reasonable amount
of memory. (I seem to have a memory leak somewhere, but it takes many
hours to get up to 150M.)

Everything else looks pretty standard to me. It seems to work as
designed most of the time, but occasionally will result in two
processes. I'm baffled as to how this happens.

If anybody has any idea, I would love to hear them.

2010/2/26 Jeremy Shaw jer...@n-heptane.com:
 Hello,
 It will be interesting to see if that makes any difference -- it shouldn't.
 In happstack-server we use 'listenOn'. According to the documentation
 listenOn already sets ReuseAddr:
 http://hackage.haskell.org/packages/archive/network/2.2.1.7/doc/html/Network.html#v%3AlistenOn
 A quick look at the source code confirms it is calling:
    setSocketOption sock ReuseAddr 1
 It sounds to be like you are getting the socket in use error because you
 have 2 processes running. Seems like the first one hasn't really died, but
 the second one is started anyway. How is it that your loop starts a second
 server when the first one has not finished?
 - jeremy
 On Fri, Feb 26, 2010 at 1:43 PM, Thomas Hartman tphya...@gmail.com wrote:

 Thanks, I altered my top level request handler as follows

 mysmartserver conf h stateProxy = do
      socket - bindPort conf

       I added (setSocketOption socket ReuseAddr 1) here
       Should this be added in a comment, or even in function code,
 in Happstack.Server.SimpleHTTP? What are the tradeoffs, when would you
 *not* want ot use ReuseAddr?)

      webserverTid - forkIO $ simpleHTTPWithSocket socket conf h
      putStrLn . ( starting happs server ++ ) = time

      control - startSystemState stateProxy -- start the HAppS state
 system
      putStrLn . ( happs state started ++ ) = time

      waitForTermination
      killThread webserverTid
      stateShutdown control

 I can't replicate the error reliably so I won't know if it actually
 fixed the problem for a while, therefore just leaving this cookie
 crumb trail in case others may find it helpful.

 2010/2/26 Brandon S. Allbery KF8NH allb...@ece.cmu.edu:
  On Feb 26, 2010, at 04:28 , Thomas Hartman wrote:
 
  me: Like mightybyte, I run my app in a shell loop that will just
  restart it after a crash. But every once in a while it won't restart
  because of the busy socket and I need to do a manual restart, killing
  multiple processes (usually 2).
 
  the error I get is:
 
  bind: resource busy (Address already in use)
 
  This is on application restart?  It's not out of file descriptors, it's
  just
  the system keeping the socket around (netstat will show it in TIME_WAIT,
  or
  possibly in a shutdown negotiation state such as LAST_ACK, FIN_WAIT,
  etc.
   TCP lacks *reliable* socket shutdown negotiation).
 
  You want to configure the socket with SO_REUSEADDR before trying to bind
  it
  (setSocketOption socket ReuseAddr 1).
 
  As to the portable version of sendfile, it's because not all systems
  offer a
  sendfile() system call.  Linux and *BSD do, and can use the native
  implementation; the portable version emulates sendfile() when it doesn't
  exist, at the price of additional CPU usage/system load (sendfile()
  having
  been created specifically to reduce system load in the common case for
  web
  servers).
 
  --
  brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com
  system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu
  electrical and computer engineering, carnegie mellon university    KF8NH
 
 
 


___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-19 Thread Jeremy Shaw
Hello,

I think to make progress on this bug we really need a failing test case that
other people can reproduce.

I have hacked up small server that should reproduce the error (using fdWrite
instead of sendfile). And a small C client which is intended to reproduce
the error -- but doesn't.

I have attached both.

The server tries to write a whole lot of 'a' characters to the client. The
client does not consume any of them. This causes the server to block on the
threadWaitWrite.

No matter how I kill the client, threadWaitWrite always wakes up. So, we
need to figure out exactly what the PS3 is doing differently that causes
threadWaitWrite to not wakeup.. If we don't know why it is failing, then I
don't think we can properly fix it.

- jeremy
module Main where

import Control.Concurrent (forkIO)
import Control.Monad (forever)
import GHC.Conc		(threadWaitRead, threadWaitWrite)
import Network (PortID(PortNumber), Socket, listenOn, sClose)
import Network.Socket (accept, socketToHandle, send, fdSocket, setSocketOption, SocketOption(KeepAlive))
import System.IO
import System.Posix.IO
import System.Posix.Types

listen' :: PortID - (Socket - IO ()) - IO () 
listen' port handler =
  do socket - listenOn port
 forever $ do (s,sa) - accept socket
  setSocketOption s KeepAlive 1
  forkIO $ handler s
  
main :: IO ()
main =
  listen' (PortNumber (toEnum 2525)) $ \s -
do -- h - socketToHandle s ReadWriteMode
   let fd = Fd (fdSocket s)
   writeLoop fd (10^6)
  where 
writeLoop :: Fd - ByteCount - IO ()
writeLoop fd count | count = 0 = 
  do putStrLn done.
 return ()
writeLoop fd count =
  do putStrLn threadWaitWrite
 threadWaitWrite fd
 putStrLn writing...
 n - fdWrite fd $ take (fromIntegral count) (repeat 'a')
 putStrLn (wrote:   ++ show n)
 writeLoop fd (count - n)
  
   
  #include stdio.h
#include sys/types.h
#include sys/socket.h
#include netinet/in.h
#include netdb.h 

void error(char *msg)
{
perror(msg);
exit(0);
}

int main(int argc, char *argv[])
{
int sockfd, portno, n;
struct sockaddr_in serv_addr;
struct hostent *server;

char buffer[256];
if (argc  3) {
   fprintf(stderr,usage %s hostname port\n, argv[0]);
   exit(0);
}
portno = atoi(argv[2]);
sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd  0) 
error(ERROR opening socket);
server = gethostbyname(argv[1]);
if (server == NULL) {
fprintf(stderr,ERROR, no such host\n);
exit(0);
}
bzero((char *) serv_addr, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
bcopy((char *)server-h_addr, 
 (char *)serv_addr.sin_addr.s_addr,
 server-h_length);
serv_addr.sin_port = htons(portno);
if (connect(sockfd,serv_addr,sizeof(serv_addr))  0) 
error(ERROR connecting);
/*
printf(Please enter the message: );
bzero(buffer,256);
fgets(buffer,255,stdin);
n = write(sockfd,buffer,strlen(buffer));
if (n  0) 
 error(ERROR writing to socket);
bzero(buffer,256);

n = read(sockfd,buffer,2);
if (n  0) 
 error(ERROR reading from socket);
buffer[n] = '\0';
printf(%s\n,buffer);
*/
getchar();
return 0;
}
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


[Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-04 Thread Bardur Arantsson

Hi all,

I've been using the sendfile package off Hackage, but it seems to be 
leaking file descriptors when using the Linux-native build.


What's happening in my specific case is the following:

   1) client requests a range of a file
   2) server starts sending the range
   3) client disconnects before receiving the whole file

This happens over and over with the client requesting different ranges 
of the file (so the client does make progress).


If I use the portable build of the sendfile package, everything works 
fine for hours and hours of this happening.


If I use the Linux-native build of the sendfile package, the server
will eventually run out of file descriptors. According to lsof the 
files that are being kept open are the data files being sent in 2) above.


This is on GHC 6.10.x (Ubuntu).

Is anyone else seeing this? Anyone got any idea what's going on?

Cheers,

___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-04 Thread Thomas Hartman
Do you have a test script to reproduce the behavior?

I am interested in this patch-tag uses sendfile (via happstack) to
serve static data, and I'd like to know if I'm affected by this.

2010/2/4 Bardur Arantsson s...@scientician.net:
 Hi all,

 I've been using the sendfile package off Hackage, but it seems to be leaking
 file descriptors when using the Linux-native build.

 What's happening in my specific case is the following:

   1) client requests a range of a file
   2) server starts sending the range
   3) client disconnects before receiving the whole file

 This happens over and over with the client requesting different ranges of
 the file (so the client does make progress).

 If I use the portable build of the sendfile package, everything works fine
 for hours and hours of this happening.

 If I use the Linux-native build of the sendfile package, the server
 will eventually run out of file descriptors. According to lsof the files
 that are being kept open are the data files being sent in 2) above.

 This is on GHC 6.10.x (Ubuntu).

 Is anyone else seeing this? Anyone got any idea what's going on?

 Cheers,

 ___
 Haskell-Cafe mailing list
 Haskell-Cafe@haskell.org
 http://www.haskell.org/mailman/listinfo/haskell-cafe

___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-04 Thread Jeremy Shaw
Well, that's not good.

I wonder how we are supposed to know that the client has disconnected... We
use withBinaryFile to open the file, so if an uncaught except is being
thrown then that should close the file handle..

If you look in Network.Socket.SendFile.Linux, you will see that when the C
sendfile() function returns -1, we throw an exception, unless it is the
EAGAIN error. So, that should cause us to close the files when an error
occurs.

There is one possible odd case.. if sendfile returns 0, then we assume
nothing went wrong and keep trying to resend the data until everything is
sent. If it keeps returning 0 then we just keep trying even though nothing
is happening. (Or if we keep getting EAGAIN errors..). But I am not sure
that either of these cases would actually happen in practice.

Another cause might be if threadWaitWrite is just never returning after the
client disconnects..

- jeremy



On Thu, Feb 4, 2010 at 1:43 PM, Bardur Arantsson s...@scientician.netwrote:

 Hi all,

 I've been using the sendfile package off Hackage, but it seems to be
 leaking file descriptors when using the Linux-native build.

 What's happening in my specific case is the following:

   1) client requests a range of a file
   2) server starts sending the range
   3) client disconnects before receiving the whole file

 This happens over and over with the client requesting different ranges of
 the file (so the client does make progress).

 If I use the portable build of the sendfile package, everything works fine
 for hours and hours of this happening.

 If I use the Linux-native build of the sendfile package, the server
 will eventually run out of file descriptors. According to lsof the files
 that are being kept open are the data files being sent in 2) above.

 This is on GHC 6.10.x (Ubuntu).

 Is anyone else seeing this? Anyone got any idea what's going on?

 Cheers,

 ___
 Haskell-Cafe mailing list
 Haskell-Cafe@haskell.org
 http://www.haskell.org/mailman/listinfo/haskell-cafe

___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe


Re: [Haskell-cafe] sendfile leaking descriptors on Linux?

2010-02-04 Thread Jeremy Shaw
Actually,

We should start by testing if native sendfile leaks file descriptors even
when the whole file is sent. We have a test suite, but I am not sure if it
tests for file handle leaking...

- jeremy


On Thu, Feb 4, 2010 at 7:58 PM, Jeremy Shaw jer...@n-heptane.com wrote:

 Well, that's not good.

 I wonder how we are supposed to know that the client has disconnected... We
 use withBinaryFile to open the file, so if an uncaught except is being
 thrown then that should close the file handle..

 If you look in Network.Socket.SendFile.Linux, you will see that when the C
 sendfile() function returns -1, we throw an exception, unless it is the
 EAGAIN error. So, that should cause us to close the files when an error
 occurs.

 There is one possible odd case.. if sendfile returns 0, then we assume
 nothing went wrong and keep trying to resend the data until everything is
 sent. If it keeps returning 0 then we just keep trying even though nothing
 is happening. (Or if we keep getting EAGAIN errors..). But I am not sure
 that either of these cases would actually happen in practice.

 Another cause might be if threadWaitWrite is just never returning after the
 client disconnects..

 - jeremy



 On Thu, Feb 4, 2010 at 1:43 PM, Bardur Arantsson s...@scientician.netwrote:

 Hi all,

 I've been using the sendfile package off Hackage, but it seems to be
 leaking file descriptors when using the Linux-native build.

 What's happening in my specific case is the following:

   1) client requests a range of a file
   2) server starts sending the range
   3) client disconnects before receiving the whole file

 This happens over and over with the client requesting different ranges of
 the file (so the client does make progress).

 If I use the portable build of the sendfile package, everything works fine
 for hours and hours of this happening.

 If I use the Linux-native build of the sendfile package, the server
 will eventually run out of file descriptors. According to lsof the files
 that are being kept open are the data files being sent in 2) above.

 This is on GHC 6.10.x (Ubuntu).

 Is anyone else seeing this? Anyone got any idea what's going on?

 Cheers,

 ___
 Haskell-Cafe mailing list
 Haskell-Cafe@haskell.org
 http://www.haskell.org/mailman/listinfo/haskell-cafe



___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe