Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive
On Wed, Feb 2, 2011 at 1:01 PM, Felipe Almeida Lessa felipe.le...@gmail.com wrote: And what about connection limits? We shouldn't create a thousand connections to the same host =). For what it's worth, I wrote a connection pool manager for the riak package that has to solve some of the same problems. https://github.com/mailrank/riak-haskell-client/blob/master/src/Network/Riak/Connection/Pool.hs ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive
On Wed, Feb 2, 2011 at 11:57 AM, Michael Snoyman mich...@snoyman.com wrote: As far as keep-alive goes, I still need to do a bit more research, but my basic idea (with credit to Bryan O'Sullivan): * http (and family) will all take an extra argument, Maybe Manager. * Manager will be an abstract type that will keep an MVar (Map (Host, Port, IsSecure) Socket). * If http is provided with a Manager, then it uses the Socket available in the Manager. If none is available, it creates a new Socket and places it in the Manager. * If http is *not* provided with a Manager, then it creates a new socket and closes it before returning. * There will be a newManager :: IO Manager, and a closeManager :: Manager - IO (), which closes all Sockets in the Manager and empties out the inner Map. How about concurrent use of Manager? Should we do A) do m - newManager forM xs $ forkIO $ doSomething m B) forM xs $ forkIO $ do m - newManager doSomething m While B) should work with any sane Manager implementation, it is not optimal. If all your connections are to the same host, than both approaches are the same. But if access hosts O and P, for example, than it is possible that Manager m1 has an open connection to O, but you try connect to O using another Manager m2. That means that ideally we should support approach A) as well. However, to support A a simple Map inside an MVar isn't sufficient. Cheers! =) -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive
On Wed, Feb 2, 2011 at 10:15 PM, Felipe Almeida Lessa felipe.le...@gmail.com wrote: On Wed, Feb 2, 2011 at 11:57 AM, Michael Snoyman mich...@snoyman.com wrote: As far as keep-alive goes, I still need to do a bit more research, but my basic idea (with credit to Bryan O'Sullivan): * http (and family) will all take an extra argument, Maybe Manager. * Manager will be an abstract type that will keep an MVar (Map (Host, Port, IsSecure) Socket). * If http is provided with a Manager, then it uses the Socket available in the Manager. If none is available, it creates a new Socket and places it in the Manager. * If http is *not* provided with a Manager, then it creates a new socket and closes it before returning. * There will be a newManager :: IO Manager, and a closeManager :: Manager - IO (), which closes all Sockets in the Manager and empties out the inner Map. How about concurrent use of Manager? Should we do A) do m - newManager forM xs $ forkIO $ doSomething m B) forM xs $ forkIO $ do m - newManager doSomething m While B) should work with any sane Manager implementation, it is not optimal. If all your connections are to the same host, than both approaches are the same. But if access hosts O and P, for example, than it is possible that Manager m1 has an open connection to O, but you try connect to O using another Manager m2. That means that ideally we should support approach A) as well. However, to support A a simple Map inside an MVar isn't sufficient. Good point: it should be a MVar (Map HostInfo (MVar Socket)) I think* Thanks, Michael * It's late here, just make sure I'm not saying something stupid ;). ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive
On Wed, Feb 2, 2011 at 2:28 PM, Michael Snoyman mich...@snoyman.com wrote: On Wed, Feb 2, 2011 at 10:15 PM, Felipe Almeida Lessa felipe.le...@gmail.com wrote: On Wed, Feb 2, 2011 at 11:57 AM, Michael Snoyman mich...@snoyman.com wrote: As far as keep-alive goes, I still need to do a bit more research, but my basic idea (with credit to Bryan O'Sullivan): * http (and family) will all take an extra argument, Maybe Manager. * Manager will be an abstract type that will keep an MVar (Map (Host, Port, IsSecure) Socket). * If http is provided with a Manager, then it uses the Socket available in the Manager. If none is available, it creates a new Socket and places it in the Manager. * If http is *not* provided with a Manager, then it creates a new socket and closes it before returning. * There will be a newManager :: IO Manager, and a closeManager :: Manager - IO (), which closes all Sockets in the Manager and empties out the inner Map. How about concurrent use of Manager? Should we do A) do m - newManager forM xs $ forkIO $ doSomething m B) forM xs $ forkIO $ do m - newManager doSomething m While B) should work with any sane Manager implementation, it is not optimal. If all your connections are to the same host, than both approaches are the same. But if access hosts O and P, for example, than it is possible that Manager m1 has an open connection to O, but you try connect to O using another Manager m2. That means that ideally we should support approach A) as well. However, to support A a simple Map inside an MVar isn't sufficient. Good point: it should be a MVar (Map HostInfo (MVar Socket)) I think* Or you could remove the socket from the map while it's in use. Antoine ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive
On Wed, Feb 2, 2011 at 6:30 PM, Antoine Latter aslat...@gmail.com wrote: Or you could remove the socket from the map while it's in use. And what about connection limits? We shouldn't create a thousand connections to the same host =). -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive
On Wed, Feb 2, 2011 at 3:01 PM, Felipe Almeida Lessa felipe.le...@gmail.com wrote: On Wed, Feb 2, 2011 at 6:30 PM, Antoine Latter aslat...@gmail.com wrote: Or you could remove the socket from the map while it's in use. And what about connection limits? We shouldn't create a thousand connections to the same host =). Not a bad idea, but that may be creeping outside the scope of the bug as reported. If you're writing a web-scraper/spider, it is true you might need some sort of higher-level manager to handle concurrent access. I'm not sure what that would look like, though -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive
On Thu, Feb 3, 2011 at 12:00 AM, Antoine Latter aslat...@gmail.com wrote: On Wed, Feb 2, 2011 at 3:01 PM, Felipe Almeida Lessa felipe.le...@gmail.com wrote: On Wed, Feb 2, 2011 at 6:30 PM, Antoine Latter aslat...@gmail.com wrote: Or you could remove the socket from the map while it's in use. And what about connection limits? We shouldn't create a thousand connections to the same host =). Not a bad idea, but that may be creeping outside the scope of the bug as reported. If you're writing a web-scraper/spider, it is true you might need some sort of higher-level manager to handle concurrent access. I'm not sure what that would look like, though -- Felipe. I think Felipe's point is that using the approach I outlined, we will never spawn more than one connection to a single host. By simply taking the socket out of the map, there's no limit to the number of sockets that will be created. Michael ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [web-devel] http-enumerator: redirects, streaming and keep-alive
On Wed, Feb 2, 2011 at 11:06 PM, Michael Snoyman mich...@snoyman.com wrote: On Thu, Feb 3, 2011 at 12:00 AM, Antoine Latter aslat...@gmail.com wrote: On Wed, Feb 2, 2011 at 3:01 PM, Felipe Almeida Lessa felipe.le...@gmail.com wrote: On Wed, Feb 2, 2011 at 6:30 PM, Antoine Latter aslat...@gmail.com wrote: Or you could remove the socket from the map while it's in use. And what about connection limits? We shouldn't create a thousand connections to the same host =). Not a bad idea, but that may be creeping outside the scope of the bug as reported. If you're writing a web-scraper/spider, it is true you might need some sort of higher-level manager to handle concurrent access. I'm not sure what that would look like, though -- Felipe. I think Felipe's point is that using the approach I outlined, we will never spawn more than one connection to a single host. By simply taking the socket out of the map, there's no limit to the number of sockets that will be created. Ah, then you would block one thread on another if the host info is in the map? I get it. Strictly one socket per host might be a tricky invariant to maintain - you'd have to insert an empty socket MVar in the map on lookup failure, and the shuffling of the nested MVars seems tricky off the top of my head. But even without all of that I don't think you'd be flooding the host any more than without any keepalive support, which was what I was getting at. This is just speculation on my part - I don't have a stake in this, it just sounded like an interesting problem. I'd hate to be steering this into a design black hole. Antoine Michael ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
