Hi James, Thank you for the thorough explanation! So if I understand you correctly, if no calls are made to the executor thread, it will just rest silently until queue-email is called again - not consuming any server resources. Are there any recommendations for the number of worker threads when each does IO?
Thanks, Brjánn On Thu, 25 Oct 2018 at 19:45, James Reeves <ja...@booleanknot.com> wrote: > The Java executor classes are essentially a queue backed by a thread pool > of workers. > > So suppose we create a new executor: > > (def executor > (java.util.concurrent.Executors/newFixedThreadPool 32)) > > This creates an ExecutorService object with a pool of 32 worker threads. > We can pass it a zero-argument function to execute: > > (.execute executor (fn [] (prn "Hello World"))) > > This function is placed on a queue until one of the 32 worker threads is > available to process it. In this case the function prints a message, but > you could write something to send an email instead: > > (defn queue-email [email] > (.execute executor (fn [] (send-email email))) > > If we execute "queue-email" 10,000 times, then 10,000 "send-email" jobs > will be passed to the executor. The executor will pass them onto its 32 > workers, effectively processing 32 at a time until it exhausts the queue. > > If you want fewer emails to be sent at once, you can reduce the worker > thread pool size. In my earlier example, I used an executor with only one > worker thread. > > On Thu, 25 Oct 2018 at 18:17, <brj...@gmail.com> wrote: > >> Hi James, >> >> Thanks! How would one create a thread that continuously monitors a mail >> queue and sends any mails in it? Or do you mean that I would create one >> thread per mail that I want to send? Wouldn't that create a problem if I'm >> about to send 10000 mails in one go? >> >> And thanks for the heads up regarding indefinite blocking - I'll make >> sure to guard against that. >> >> Thanks, >> Brjánn >> >> On Thu, 25 Oct 2018 at 18:57, James Reeves <ja...@booleanknot.com> wrote: >> >>> Hi Brjánn, >>> >>> Executing queued jobs in a background thread is a common task, and as it >>> happens there's a set of Java classes to do just that. >>> >>> (let [e (java.util.concurrent.Executors/newSingleThreadExecutor)] >>> (.execute e #(prn "foo")) >>> (.execute e #(prn "bar")) >>> (Thread/sleep 1000) >>> (.execute e #(prn "baz"))) >>> >>> Don't worry about cleaning up a single thread. An inactive thread takes >>> up a minimal amount of VM memory and no CPU. You're going to use up more OS >>> resources deleting and recreating the thread. >>> >>> You do need to be careful that sending an email doesn't block >>> indefinitely, but usually I/O operations come with timeouts. Just ensure >>> you set a reasonable timeout and do something sensible when you can't send. >>> >>> On Thu, 25 Oct 2018 at 16:46, <brj...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> First, I would like to briefly present myself to the list. I'm a >>>> psychologist and researcher at the Karolinska Institutet in Stockholm, >>>> Sweden. I do research within the field of internet-based psychological >>>> treatment - which is a field that has grown a lot during the last 10-15 >>>> years. I have developed my own web platform to deliver these treatments to >>>> patients with different mental and medical conditions. I wrote the first >>>> version in PHP, and not being a professional programmer, the code base is a >>>> big mess after > 10 years of development. >>>> >>>> Two years ago, a friend introduced me to Clojure. I had never worked >>>> with a lisp before, but I quickly fell completely in love with it. So many >>>> of the issues of programming in PHP completely disappeared! Words cannot >>>> express my gratitude to Rich for inventing Clojure, the core team, all >>>> developers out there who write extremely useful libraries, and my friend >>>> for introducing Clojure to me. >>>> >>>> I decided to completely rewrite my web platform in Clojure. I've chosen >>>> to do a gradual move from PHP to Clojure, meaning that I replace PHP code >>>> with Clojure code component by component and continuously test in >>>> production. Like I said, I'm not a professional programmer, and this >>>> venture poses a lot of challenges. I try to read everything I find in books >>>> and on the web, but when it comes to more complex issues, such as threads >>>> and async programming, I feel that I end up almost guessing and with a lot >>>> of trial and error. I have no idea how cautious one should be when >>>> launching a new thread (while it completely occupy the server???) and am >>>> often surprised when my go blocks suddenly freeze. I feel that I am at the >>>> mercy of the Clojure community if I want to understand these (and many >>>> other!) issues. >>>> >>>> This leads me to the subject of this email. I've decided to migrate my >>>> mail queue from PHP to Clojure. In PHP, it's just a cron job that executes >>>> every five minutes to send all emails (and actually also SMS-messages, but >>>> not really relevant) that have been queued. >>>> >>>> I've written a basic mock Clojure implementation with the following >>>> goals >>>> - All messages should be sent async, i..e, the web user should not have >>>> to wait while the email is being sent. -> I'm sending them in a separate >>>> thread. >>>> - I have a fear that if I have a thread dedicated only to sending >>>> emails, I'm wasting server resources. -> I want the thread to die 5 seconds >>>> after the last email in the queue has been sent. >>>> >>>> My implementation basically consists of >>>> - An eternal go loop that receives a notification through a channel if >>>> new messages have been queued >>>> - The go loop checks if the mail sender thread is running. If not, it >>>> starts it. >>>> - The mail sender thread dies 5 secs after the last email was sent >>>> - The state of the thread (running / not running) is stored in an agent >>>> to avoid race conditions (i.e., starting multiple threads or not starting a >>>> thread because it is running when its status is checked but stops right >>>> after). >>>> >>>> My code is here >>>> https://gist.github.com/brjann/2aef16849b9bd445374cb6b31efece60 >>>> >>>> If any of you have had the time and energy to read this far (including >>>> the code), I would be very grateful for your input. >>>> - Is there a risk that my go block will hang? >>>> - Have I eliminated the risk for race conditions? >>>> - Do I really need to kill the thread or is there no risk for thread >>>> starvation on the server (I will probably >>>> - Could I use send instead of send-off? I guess that I am now using two >>>> threads, one for the sender and one each time I send a message using >>>> send-off. >>>> - Any newbie mistakes / coding style issues? >>>> - Could this be done in a better/simpler way??? >>>> >>>> (Btw, I would be very grateful for feedback on the form of my question >>>> if you have any. Are there better/other forums? Could I present the >>>> question or code in a another manner to help you understand better?) >>>> >>>> I am happy for any replies - on the list or backchannel. And I hope >>>> that you feel that I have not misused your inbox (I may repeat this >>>> behavior...). >>>> >>>> Thanks, >>>> Brjánn >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clojure@googlegroups.com >>>> Note that posts from new members are moderated - please be patient with >>>> your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+unsubscr...@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/clojure?hl=en >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to clojure+unsubscr...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> James Reeves >>> booleanknot.com >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clojure@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > James Reeves > booleanknot.com > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
