Re: [Haskell-cafe] External system connections
On 07/10/11 15:37, Felipe Almeida Lessa wrote: You don't need to do it, it is already done =). See the pool package by Michael Snoyman on Hackage [1]. More specifically, see createPoolCheckAlive [2]. Cheers, [1] http://hackage.haskell.org/package/pool [2] http://hackage.haskell.org/packages/archive/pool/0.1.0.2/doc/html/Data-Pool.html#v:createPoolCheckAlive How do people use this stuff? The README is empty, there's no documentation, and none of the functions are commented. I'm forced to conclude that there are people capable of looking at this, createPoolCheckAlive :: MonadControlIO m = IO a - (a - IO ()) - Int - (Pool a - m b) - (a - IO Bool) - m b and knowing immediately what it does. I'm still a beginner, and I don't think I have a use for this package (what does it do?), but I've run into similar situations with other packages and would genuinely like to know (as opposed to just complain about it). ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Mon, Jul 11, 2011 at 11:27, Michael Orlitzky mich...@orlitzky.com wrote: How do people use this stuff? The README is empty, there's no documentation, and none of the functions are commented. I'm forced to conclude that there are people capable of looking at this, I can *almost* work it out just from the type and knowing what it should be doing. But in this case, the main problem is that it was split off of yesod (http://www.yesodweb.com/), and it was only documented in the context of that. -- brandon s allbery allber...@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Mon, Jul 11, 2011 at 12:27 PM, Michael Orlitzky mich...@orlitzky.com wrote: How do people use this stuff? The README is empty, there's no documentation, and none of the functions are commented. I'm forced to conclude that there are people capable of looking at this, createPoolCheckAlive :: MonadControlIO m = IO a - (a - IO ()) - Int - (Pool a - m b) - (a - IO Bool) - m b and knowing immediately what it does. Yes, there's a documentation problem. But you can guess by the types =). IO a: opens a new resource a. a - IO (): disposes the given resource. Int: maximum number of open resources. Pool a - m b: the action you want to execute; after it executes, the pool is destroyed. a - IO Bool: check if the resource is still alive. m b: the result of running your given action with a new pool. The key to understanding all of this is looking backwards from the result. 1) The result is m b. Where did this come from? 2) Well, there's a 'Pool a - m b'. So it runs the action for me. So this is like a 'withFile :: FilePath - (Handle - IO a) - IO a' function. 3) So if it is a 'Pool a', then the resource is of type 'a'. 4) The only function returning 'a' is 'IO a', so this creates the resource. 5) By the same reason, 'a - IO ()' disposes it. 6) 'a - IO Bool' is confusing, but there's a doc for that. 7) 'Int' is also confusing, but since this is a pool, this must be the maximum number of open resources. HTH, -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
Have you looked at Bryan O'Sullivan's resource-pool[1] library? I've used it with MySQL, Redis and Riak connection pooling. It's been robust and easy to use. It seems like it would be a good fit for your application. Cheers, Ozgun [1] http://hackage.haskell.org/package/resource-pool On Sun, Jul 10, 2011 at 3:04 PM, Richard Wallace rwall...@thewallacepack.net wrote: Hello all, I'm trying to understand how to properly structure a connection to an external system. I'm writing an application that processes requests (it's an IRC bot - ya, I've looked at lambabot but it's a bit beyond my current understanding and I'm really trying to learn this stuff and find the best way to do that is to write it myself) and sends requests to an external system, via SOAP. The SOAP requests should be sent on separate threads. The SOAP server requires an initial request for authentication be made to obtain a token used on subsequent requests. There are three ways I can structure this. One is to do the authentication with the SOAP server when the application starts up and just use that token for all subsequent requests. This will fall over if the token times out and I'd have to restart the application. Another way is to do authenticate with the server and get a new token for each request. This is obviously really inefficient. What I'd like to do is something a bit smarter. When a request to the SOAP server is to be made, if we have a token then we try and use it. If it fails, we reauthenticate and get a new token. When establishing a new token, other threads trying to send SOAP requests should block until a new token is available. There are other conditions that should be handled - such as an exponential back-off when the SOAP server can't be reached - but I feel like reestablishing the authentication token is really the key concern here. Unfortunately, I have no idea right now how to write this in Haskell. Any pointers would be awesome. Thanks! Rich ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On 07/11/11 11:41, Felipe Almeida Lessa wrote: Yes, there's a documentation problem. But you can guess by the types =). IO a: opens a new resource a. a - IO (): disposes the given resource. Int: maximum number of open resources. Pool a - m b: the action you want to execute; after it executes, the pool is destroyed. a - IO Bool: check if the resource is still alive. m b: the result of running your given action with a new pool. The key to understanding all of this is looking backwards from the result. 1) The result is m b. Where did this come from? 2) Well, there's a 'Pool a - m b'. So it runs the action for me. So this is like a 'withFile :: FilePath - (Handle - IO a) - IO a' function. 3) So if it is a 'Pool a', then the resource is of type 'a'. 4) The only function returning 'a' is 'IO a', so this creates the resource. 5) By the same reason, 'a - IO ()' disposes it. 6) 'a - IO Bool' is confusing, but there's a doc for that. 7) 'Int' is also confusing, but since this is a pool, this must be the maximum number of open resources. Magic, got it =) ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Mon, Jul 11, 2011 at 6:27 PM, Michael Orlitzky mich...@orlitzky.com wrote: On 07/10/11 15:37, Felipe Almeida Lessa wrote: You don't need to do it, it is already done =). See the pool package by Michael Snoyman on Hackage [1]. More specifically, see createPoolCheckAlive [2]. Cheers, [1] http://hackage.haskell.org/package/pool [2] http://hackage.haskell.org/packages/archive/pool/0.1.0.2/doc/html/Data-Pool.html#v:createPoolCheckAlive How do people use this stuff? The README is empty, there's no documentation, and none of the functions are commented. I'm forced to conclude that there are people capable of looking at this, createPoolCheckAlive :: MonadControlIO m = IO a - (a - IO ()) - Int - (Pool a - m b) - (a - IO Bool) - m b and knowing immediately what it does. I'm still a beginner, and I don't think I have a use for this package (what does it do?), but I've run into similar situations with other packages and would genuinely like to know (as opposed to just complain about it). Author of the package speaking. I agree, that's a problem :). I've just uploaded a new version[1] that is properly documented, though it will take a bit for Hackage to generates the Haddocks. As Brandon said, the reason this wasn't documented in the first place is that it was an internally used module for Persistent that I decided after the fact to release separately. As for Bryan's resource-pool: currently I would strongly recommend *against* using it for any purpose. It is based on MonadCatchIO-transformers[2], which is a subtly broken package. In particular, when I tried using it for pool/persistent in the first place, I ended up with double-free bugs from SQLite. As an abridged history, I ended up writing a replacement called MonadInvertIO, which later was superceded by MonadPeelIO[3] and MonadControlIO[4]. The latter two packages are both much more correct that MonadCatchIO, and either one should be used in its place. I did email Bryan about this a bit ago, but he didn't get back (he *is* a very busy guy). Ideally, I think that we don't need to have both pool and resource-pool, but such is the situation right now. Michael [1] http://hackage.haskell.org/package/pool-0.1.0.3 [2] http://hackage.haskell.org/package/MonadCatchIO-transformers-0.2.2.2 [3] http://hackage.haskell.org/package/monad-peel [4] http://hackage.haskell.org/package/monad-control-0.2.0.1 ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On 07/11/11 13:49, Michael Snoyman wrote: Author of the package speaking. I agree, that's a problem :). I've just uploaded a new version[1] that is properly documented, though it will take a bit for Hackage to generates the Haddocks. As Brandon said, the reason this wasn't documented in the first place is that it was an internally used module for Persistent that I decided after the fact to release separately. Thanks for humoring me. I guess I'll begin tracking down other authors and making them feel guilty! ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Mon, Jul 11, 2011 at 10:49 AM, Michael Snoyman mich...@snoyman.comwrote: I did email Bryan about this a bit ago, but he didn't get back [...] Thanks for jogging my memory. I've released an updated version of resource-pool that drops the dependency on that package. http://hackage.haskell.org/package/resource-pool ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
12 July 2011 05:49, Michael Snoyman mich...@snoyman.com wrote: As for Bryan's resource-pool: currently I would strongly recommend *against* using it for any purpose. It is based on MonadCatchIO-transformers[2], which is a subtly broken package. In particular, when I tried using it for pool/persistent in the first place, I ended up with double-free bugs from SQLite. Do you have a reference explaining this brokenness? e.g. a mailing list message? I wasn't aware of this. Are the other MonadCatchIO-* packages also broken? Alistair ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Jul 11, 2011, at 10:48 PM, Alistair Bayley wrote: 12 July 2011 05:49, Michael Snoyman mich...@snoyman.com wrote: As for Bryan's resource-pool: currently I would strongly recommend *against* using it for any purpose. It is based on MonadCatchIO-transformers[2], which is a subtly broken package. In particular, when I tried using it for pool/persistent in the first place, I ended up with double-free bugs from SQLite. Do you have a reference explaining this brokenness? e.g. a mailing list message? I wasn't aware of this. Are the other MonadCatchIO-* packages also broken? http://www.haskell.org/pipermail/haskell-cafe/2010-October/084890.html ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] External system connections
Hello all, I'm trying to understand how to properly structure a connection to an external system. I'm writing an application that processes requests (it's an IRC bot - ya, I've looked at lambabot but it's a bit beyond my current understanding and I'm really trying to learn this stuff and find the best way to do that is to write it myself) and sends requests to an external system, via SOAP. The SOAP requests should be sent on separate threads. The SOAP server requires an initial request for authentication be made to obtain a token used on subsequent requests. There are three ways I can structure this. One is to do the authentication with the SOAP server when the application starts up and just use that token for all subsequent requests. This will fall over if the token times out and I'd have to restart the application. Another way is to do authenticate with the server and get a new token for each request. This is obviously really inefficient. What I'd like to do is something a bit smarter. When a request to the SOAP server is to be made, if we have a token then we try and use it. If it fails, we reauthenticate and get a new token. When establishing a new token, other threads trying to send SOAP requests should block until a new token is available. There are other conditions that should be handled - such as an exponential back-off when the SOAP server can't be reached - but I feel like reestablishing the authentication token is really the key concern here. Unfortunately, I have no idea right now how to write this in Haskell. Any pointers would be awesome. Thanks! Rich ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
You don't need to do it, it is already done =). See the pool package by Michael Snoyman on Hackage [1]. More specifically, see createPoolCheckAlive [2]. Cheers, [1] http://hackage.haskell.org/package/pool [2] http://hackage.haskell.org/packages/archive/pool/0.1.0.2/doc/html/Data-Pool.html#v:createPoolCheckAlive -- Felipe. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Sun, Jul 10, 2011 at 15:04, Richard Wallace rwall...@thewallacepack.net wrote: What I'd like to do is something a bit smarter. When a request to the SOAP server is to be made, if we have a token then we try and use it. If it fails, we reauthenticate and get a new token. When establishing a new token, other threads trying to send SOAP requests should block until a new token is available. There are other conditions that Typically you'd use a Chan or TChan for communication with the SOAP thread. Other threads write to the Chan and block if the reading thread is busy. There's also a non-blocking version. (TChan uses STM, which provides atomic transactions; in this case it may be overkill, especially if you're just getting started.) I wouldn't be surprised if there's already a (or several) package on Hackage for it -- brandon s allbery allber...@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Sun, Jul 10, 2011 at 2:54 PM, Brandon Allbery allber...@gmail.com wrote: On Sun, Jul 10, 2011 at 17:34, Richard Wallace rwall...@thewallacepack.net wrote: Rather than a single separate thread that makes requests I was hoping to make several soap requests concurrently, rather than have them be made serially. I would only want to block other threads making soap request if one of them returns a response indicating the token they were using is no longer valid. Is there one shared token, or one per requesting thread, or some other arrangement? There should be one shared token. In either case, separation of concerns makes me think the token handling belongs on the other end: a thread writes a request down a Chan, a queue dispatcher thread reads it if there are available workers and the token is available (with callers blocking otherwise), dispatcher sends request to available worker. If the worker finds the token is invalid, it calls back in to the dispatcher with a token renewal request, which causes anything else that needs that token to block as above, and itself blocks until a valid token is returned. Ahhh... I see. That sounds like a much better plan. It also keeps the fact that the token is mutable confined to a single thread, the dispatcher thread. One thing I'd have to be careful to handle is iIf multiple worker threads find that the token is now invalid. In that case they would all send the token renewal request to the dispatcher and they would each try and do the re-auth. To avoid that, part of the token renewal request could be the used token. When the dispatcher does the token renewal, it checks if the used token is the same as the current token. If so, it does the token renewal. If not, it dispatches to the SOAP worker thread with the new token. Another advantage of this is that adding more workers is done in an obvious place (the queuer thread). That said, I still don't know enough details to say if that token management implementation actually makes sense. I'd still go with the worker/queuer thread setup, which is I think what you were pointed to (the pool hackage). Ok, I still don't see how the Pool package helps. I had assumed the pooled resource would be the token, so the size of the pool would be 1. Now that I rethink it, it seems like you are suggesting the resource is the worker threads. Is that right? When I read what you were talking about above with the dispatcher thread I had thought it would use forkIO to start the worker threads. The worker threads would read from a Chan that the dispatcher wrote to.In that way, the dispatcher doesn't have to worry about worker threads being available. Am I misunderstanding something? Thanks, Rich -- brandon s allbery allber...@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Sun, Jul 10, 2011 at 19:16, Richard Wallace rwall...@thewallacepack.net wrote: On Sun, Jul 10, 2011 at 2:54 PM, Brandon Allbery allber...@gmail.com wrote: One thing I'd have to be careful to handle is iIf multiple worker threads find that the token is now invalid. In that case they would all send the token renewal request to the dispatcher and they would each try and do the re-auth. To avoid that, part of the token renewal One of the reasons to send the request back to the dispatcher instead of doing it inline is so that the dispatcher can note that a renewal request is already in flight (which it needs to know anyway, so it can block other requests) and wake all threads waiting on it when it's done, instead of having multiple renewals in flight. 1. Now that I rethink it, it seems like you are suggesting the resource is the worker threads. Is that right? When I read what you were talking about above with the dispatcher thread I had thought it would use forkIO to start the worker threads. The worker threads would read from a Chan that the dispatcher wrote to. In that way, the dispatcher doesn't have to worry about worker threads being available. Am I misunderstanding something? The point of a pool is so (a) you can throttle it in special cases, such as when you need to renew the token, and (b) so you don't find yourself juggling a couple thousand threads if things get unexpectedly busy (or buggy). You can limit the pool to something sensible (probably something like 4 during development and debugging so things can't get too out of hand) and increased later; plus, the pool manager will provide the primitives to deal with managing shared resources (such as your token) within the pool. -- brandon s allbery allber...@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Sun, Jul 10, 2011 at 4:38 PM, Brandon Allbery allber...@gmail.com wrote: One of the reasons to send the request back to the dispatcher instead of doing it inline is so that the dispatcher can note that a renewal request is already in flight (which it needs to know anyway, so it can block other requests) and wake all threads waiting on it when it's done, instead of having multiple renewals in flight. Ok, I can see that. Though I was thinking that the worker threads would send the request back to the dispatcher by way of the same Chan the dispatcher reads requests from. Obviously I was thinking the dispatcher would use the original token to filter requests in the Chan. If I understand what you are talking about, the dispatcher would do the token renewal in a separate thread, continuing to process it's incoming Chan while that is going on, accumulating incoming requests somewhere, perhaps in another Chan. Then, when the token renewal is complete, the dispatcher stops forwarding incoming requests to the secondary Chan, processes the accumulated requests using the new token, and, when done with those, the switches back to processing the incoming Chan. Is that something like what you had in mind, or did I make it more complicated than necessary? The point of a pool is so (a) you can throttle it in special cases, such as when you need to renew the token, and (b) so you don't find yourself juggling a couple thousand threads if things get unexpectedly busy (or buggy). You can limit the pool to something sensible (probably something like 4 during development and debugging so things can't get too out of hand) and increased later; plus, the pool manager will provide the primitives to deal with managing shared resources (such as your token) within the pool. Hmm. I'm still not entirely seeing it and I think the problem is just my lack of knowledge of Haskell concurrency. If the pool is of threads, how do you start the threads? How do you submit work to the threads? The only way I know of in Haskell of creating threads to do work is forkIO. That takes a function and runs to completion. Would a worker thread just be one that loops forever and checks a MVar for something to do? So the pool would really consist of (MVar, ThreadId) pairs. When we speak of getting a thread out of a pool and giving it work to do, are we really talking about sticking a value in an MVar that the thread is blocking on, waiting for data to be available to do something with, so that it can go and do some work? And when we are done with the thread we return it to the pool? Rich P.S. Thanks for talking me through this! I'm learning a ton about concurrency in Haskell. -- brandon s allbery allber...@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
On Sun, Jul 10, 2011 at 20:19, Richard Wallace rwall...@thewallacepack.net wrote: On Sun, Jul 10, 2011 at 4:38 PM, Brandon Allbery allber...@gmail.com wrote: Ok, I can see that. Though I was thinking that the worker threads would send the request back to the dispatcher by way of the same Chan the dispatcher reads requests from. Obviously I was thinking the dispatcher would use the original token to filter requests in the Chan. If I understand what you are talking about, the dispatcher would do the token renewal in a separate thread, continuing to process The same Chan is used to send a request; the thread processing token renewal might or might not be otherwise a normal worker thread, but it's separated out as a Maybe ThreadId instead of being in a pool, because (a) there can only be zero or one of them, and (b) if it's not Nothing then the dispatcher thread accepts only token renewals. (This actually requires either multiple Chans or something more complex than a normal Chan, since you can't filter a Chan based on the type of message.) You also need some way to block the sender, which suggests that a message written down a Chan must include an MVar which will be signaled when the operation is complete. This suggests to me something along the lines of data WorkRequest = SOAPData ... (MVar Bool) | TokenRequest Token (MVar Token) -- | ReadyForWork (MVar WorkRequest) where the requestor allocates an MVar, writes it as part of the WorkRequest, and then does a takeMVar to wait for the response. The dispatcher reads WorkRequests, dispatches any it can to available workers, and queues the rest internally; if it's a TokenRequest then it's queued separately and all SOAPData requests get queued regardless of whether there are free workers. When the single token processor returns, all entries in the TokenRequest queue get awakened (putMVar threadMVar newToken) and normal processing of the standard request queue resumes. Or you can see if the pool hackage handles the ugly details here automatically; I haven't looked. If the pool is of threads, how do you start the threads? How do you submit work to the threads? The only way I know of in Haskell of creating threads to do work is forkIO. That takes a function and runs to completion. Would a worker thread just be one that loops forever Yes; the dispatcher keeps a list of workers, which are forkIO-d threads that are waiting on an MVar or Chan for work to do. When they receive something, they go off and do it, write the result into another MVar or Chan which was specified in the request, and go back to waiting on the initial MVar/Chan for something to do. If the list is shorter than the maximum, more workers are forkIO-d to fill it as needed; if longer, idle workers are sent shut down requests. (The latter is polite handling of program shutdown, and also allows for the pool size to be modified dynamically if needed.) I think doing this right also requires that a worker that's ready for more work explicitly check in, so the dispatcher knows it's available; that could be handled by an additional WorkRequest type (see commented-out line above, where a worker that's ready to handle another request passes its input MVar to the dispatcher)... but there may be better ways; I have some grasp of concurrency, but my Haskell library fu is still somewhat weak. Hopefully someone else will jump in if appropriate. (You can see how quickly this becomes complex, though; if the canned solution does what you need, you might want to avoid reinventing this particular wheel unless you're doing it for educational purposes.) -- brandon s allbery allber...@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] External system connections
Alright, I'll have to think on this some more but I think we're speaking the same language now - and what's more I even understand it! Thanks again for all your help, Rich On Sun, Jul 10, 2011 at 6:46 PM, Brandon Allbery allber...@gmail.com wrote: On Sun, Jul 10, 2011 at 20:19, Richard Wallace rwall...@thewallacepack.net wrote: On Sun, Jul 10, 2011 at 4:38 PM, Brandon Allbery allber...@gmail.com wrote: Ok, I can see that. Though I was thinking that the worker threads would send the request back to the dispatcher by way of the same Chan the dispatcher reads requests from. Obviously I was thinking the dispatcher would use the original token to filter requests in the Chan. If I understand what you are talking about, the dispatcher would do the token renewal in a separate thread, continuing to process The same Chan is used to send a request; the thread processing token renewal might or might not be otherwise a normal worker thread, but it's separated out as a Maybe ThreadId instead of being in a pool, because (a) there can only be zero or one of them, and (b) if it's not Nothing then the dispatcher thread accepts only token renewals. (This actually requires either multiple Chans or something more complex than a normal Chan, since you can't filter a Chan based on the type of message.) You also need some way to block the sender, which suggests that a message written down a Chan must include an MVar which will be signaled when the operation is complete. This suggests to me something along the lines of data WorkRequest = SOAPData ... (MVar Bool) | TokenRequest Token (MVar Token) -- | ReadyForWork (MVar WorkRequest) where the requestor allocates an MVar, writes it as part of the WorkRequest, and then does a takeMVar to wait for the response. The dispatcher reads WorkRequests, dispatches any it can to available workers, and queues the rest internally; if it's a TokenRequest then it's queued separately and all SOAPData requests get queued regardless of whether there are free workers. When the single token processor returns, all entries in the TokenRequest queue get awakened (putMVar threadMVar newToken) and normal processing of the standard request queue resumes. Or you can see if the pool hackage handles the ugly details here automatically; I haven't looked. If the pool is of threads, how do you start the threads? How do you submit work to the threads? The only way I know of in Haskell of creating threads to do work is forkIO. That takes a function and runs to completion. Would a worker thread just be one that loops forever Yes; the dispatcher keeps a list of workers, which are forkIO-d threads that are waiting on an MVar or Chan for work to do. When they receive something, they go off and do it, write the result into another MVar or Chan which was specified in the request, and go back to waiting on the initial MVar/Chan for something to do. If the list is shorter than the maximum, more workers are forkIO-d to fill it as needed; if longer, idle workers are sent shut down requests. (The latter is polite handling of program shutdown, and also allows for the pool size to be modified dynamically if needed.) I think doing this right also requires that a worker that's ready for more work explicitly check in, so the dispatcher knows it's available; that could be handled by an additional WorkRequest type (see commented-out line above, where a worker that's ready to handle another request passes its input MVar to the dispatcher)... but there may be better ways; I have some grasp of concurrency, but my Haskell library fu is still somewhat weak. Hopefully someone else will jump in if appropriate. (You can see how quickly this becomes complex, though; if the canned solution does what you need, you might want to avoid reinventing this particular wheel unless you're doing it for educational purposes.) -- brandon s allbery allber...@gmail.com wandering unix systems administrator (available) (412) 475-9364 vm/sms ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
