Re: [ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
I've got a fresh experience of refactoring my project to use web workers. Here is the story: I'm building a code editor using web technologies[1]. The initial design was pretty simple and sweet. I decided to go with re-frame (thanks Mike!), which means UI driven by reagent and react with nice&clean event processing/subscription system. The editor is doing basically three things: 1) UI rendering and responding to user actions 2) analyzing source code (for syntax highlighting, etc.) 3) layouting source code for renderer to generate markup I observed that analyzing and layouting is blocking my rendering tasks, so I decided to offload them to another thread using webworkers. What seemed like an easy change at first, introduced a lot of complexity later. Problems encountered so far: 1) cljs-devtools[2] does not work from web workers, this is a major problem for my development workflow 2) data must be marshalled as JSON => wasn't that hard, started using transmit 3) instead of one app-db state and one queue of events, now I have it split in two, one in the worker and one in the main thread. Keeping them in sync can be tricky due to async worker nature. imagine undo/redo operations - those have to be executed on both databases "at once". To solve problem #1. I had to implement special development version of the app, which does not run a worker thread, but emulates it on the main thread. It does not sound that hard, except re-frame wasn't ready to provide me with two independent re-frame event queues in single javascript context. I had to rewrite it: https://github.com/Day8/re-frame/pull/107. To make the long story short. One month later and wish I'd chosen regexs instead ;) happy web-working! :) Antonin [1] https://github.com/darwin/plastic [2] https://github.com/binaryage/cljs-devtools On Wednesday, August 26, 2015 at 10:27:34 AM UTC+2, Atamert Ölçgen wrote: > On Wed, Aug 26, 2015 at 2:46 AM, Mike Thompson wrote: > On Tuesday, August 25, 2015 at 7:45:28 PM UTC+10, Thomas Heller wrote: > > > On Tuesday, August 25, 2015 at 10:59:53 AM UTC+2, Daniel Kersten wrote: > > > > "The browser does not support threads so neither can core.async." > > > > > > > > > > > > To expand on that, core.async uses cooperative multitasking, which means > > > you have to give control back every so often so it can schedule other go > > > blocks to be run. Calls like > > why timeout works). > > > > > > > > > > While that is correct let me emphasize that timeout is not a solution! > > > > > > Do you always know how long task X will run or whether you are going to > > need to chunk it? Is it even possible to split up? A "task" that may > > complete in 10ms on your machine might take 100ms on another one or even > > 500ms on yours if the computer is doing something else. > > > > > > If you need to do CPU intensive work in the browser use a WebWorker. It is > > their purpose. While not perfect it is far better than trying to be > > "cooperative" in your code. > > > > > > My 2 cents, > > > /thomas > > > > > > You have a problem, and you decide to solve it via webworkers. > > > > Later you'll WISH you'd chosen regexs instead :-) > > > > Why the cynicism? Isn't it true that the only way to make use of multiple > cores in browser is to use WebWorkers? > > > > > > -- > > Mike > > > > > > -- > > Note that posts from new members are moderated - please be patient with your > first post. > > --- > > You received this message because you are subscribed to the Google Groups > "ClojureScript" group. > > To unsubscribe from this group and stop receiving emails from it, send an > email to clojurescrip...@googlegroups.com. > > To post to this group, send email to clojur...@googlegroups.com. > > Visit this group at http://groups.google.com/group/clojurescript. > > > > > > -- > > > > > > > Kind Regards, > Atamert Ölçgen > > > ◻◼◻ > ◻◻◼ > ◼◼◼ > www.muhuk.com -- Note that posts from new members are moderated - please be patient with your first post. --- You received this message because you are subscribed to the Google Groups "ClojureScript" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscr...@googlegroups.com. To post to this group, send email to clojurescript@googlegroups.com. Visit this group at http://groups.google.com/group/clojurescript.
Re: [ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
On Wednesday, August 26, 2015 at 1:29:38 PM UTC+2, Daniel Kersten wrote: > > If you can use newer browser, you've got Transferrable Object to transfer > binary data with zero-copy. > True but I was talking about CLJS data which isn't binary. > > Agreed, its probably a bit of work. I guess that since each one has its own > JS context, the simplest way is probably to have two cljs projects: your main > code and your web worker code. > Actually you want to use closure modules so some code can be shared. With 2 projects the client will download 2 separate instances of cljs.core which is quite heavy. I have an example here: https://github.com/thheller/shadow-build/tree/master/cljs-data/workers/src https://github.com/thheller/shadow-build/blob/master/dev/build.clj#L6-L30 It is not a good example but the idea is to have one common module (cljs) and one for each "client" (ie. the page itself, worker1 and worker). worker1 (and 2) will automatically load the cljs module which the browser already has available because the page did already load it. But this is going too far off-topic. If anyone is actually interested in using WebWorkers I'd be happy to go into further detail on how that works in shadow-build. Cheers, /thomas -- Note that posts from new members are moderated - please be patient with your first post. --- You received this message because you are subscribed to the Google Groups "ClojureScript" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscr...@googlegroups.com. To post to this group, send email to clojurescript@googlegroups.com. Visit this group at http://groups.google.com/group/clojurescript.
Re: [ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
On Wed, 26 Aug 2015 at 11:36 Thomas Heller wrote: > Well, while we are on the topic perhaps we should mention the downside of > using Web Workers. For me it has been Data Transfer. > > Since you can only transfer JSON type data between workers that usually > meant you had to pr-str/read-string to roundtrip CLJS data. If you only > have very small amounts of data that is no problem but read-string is not > very performant and even something like 1mb of data will eat up enough CPU > time to drop you below 60fps. > If you can use newer browser, you've got Transferrable Object to transfer binary data with zero-copy. > > That might now be worked around with transit though since that is much > faster. But still if you need to transfer a lot of data between workers you > might be better of not using them. IE8/9 might also be an issue but even > Microsoft suggest that you should not support those versions. > > Oh and so far I have not seen support for workers in lein-cljsbuild and I > haven't documented anything in shadow-build so building web worker > javascript files might actually be something of a mystery to people. To be > honest I never used them for anything other than toy projects either. > Agreed, its probably a bit of work. I guess that since each one has its own JS context, the simplest way is probably to have two cljs projects: your main code and your web worker code. There is Servant: https://github.com/MarcoPolo/servant What it seems to do is run the same compiled cljs both as the main code and the webworker code (with some setup/detection code to manage how it should run). This way both your webworker and main code can call the same functions. Communicating between them still happens through messages. I've never actually used it, so my summary may not be great :) > > Anyways, I like them. Use them. It is not that hard. :) > > Cheers, > /thomas > > > > > > > Why the cynicism? Isn't it true that the only way to make use of > multiple cores in browser is to use WebWorkers? > > > > > > -- > Note that posts from new members are moderated - please be patient with > your first post. > --- > You received this message because you are subscribed to the Google Groups > "ClojureScript" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojurescript+unsubscr...@googlegroups.com. > To post to this group, send email to clojurescript@googlegroups.com. > Visit this group at http://groups.google.com/group/clojurescript. > -- Note that posts from new members are moderated - please be patient with your first post. --- You received this message because you are subscribed to the Google Groups "ClojureScript" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscr...@googlegroups.com. To post to this group, send email to clojurescript@googlegroups.com. Visit this group at http://groups.google.com/group/clojurescript.
Re: [ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
Well, while we are on the topic perhaps we should mention the downside of using Web Workers. For me it has been Data Transfer. Since you can only transfer JSON type data between workers that usually meant you had to pr-str/read-string to roundtrip CLJS data. If you only have very small amounts of data that is no problem but read-string is not very performant and even something like 1mb of data will eat up enough CPU time to drop you below 60fps. That might now be worked around with transit though since that is much faster. But still if you need to transfer a lot of data between workers you might be better of not using them. IE8/9 might also be an issue but even Microsoft suggest that you should not support those versions. Oh and so far I have not seen support for workers in lein-cljsbuild and I haven't documented anything in shadow-build so building web worker javascript files might actually be something of a mystery to people. To be honest I never used them for anything other than toy projects either. Anyways, I like them. Use them. It is not that hard. :) Cheers, /thomas > > Why the cynicism? Isn't it true that the only way to make use of multiple > cores in browser is to use WebWorkers? > > -- Note that posts from new members are moderated - please be patient with your first post. --- You received this message because you are subscribed to the Google Groups "ClojureScript" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscr...@googlegroups.com. To post to this group, send email to clojurescript@googlegroups.com. Visit this group at http://groups.google.com/group/clojurescript.
Re: [ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
On Wed, Aug 26, 2015 at 2:46 AM, Mike Thompson wrote: > On Tuesday, August 25, 2015 at 7:45:28 PM UTC+10, Thomas Heller wrote: > > On Tuesday, August 25, 2015 at 10:59:53 AM UTC+2, Daniel Kersten wrote: > > > "The browser does not support threads so neither can core.async." > > > > > > > > > To expand on that, core.async uses cooperative multitasking, which > means you have to give control back every so often so it can schedule other > go blocks to be run. Calls like why timeout works). > > > > > > > While that is correct let me emphasize that timeout is not a solution! > > > > Do you always know how long task X will run or whether you are going to > need to chunk it? Is it even possible to split up? A "task" that may > complete in 10ms on your machine might take 100ms on another one or even > 500ms on yours if the computer is doing something else. > > > > If you need to do CPU intensive work in the browser use a WebWorker. It > is their purpose. While not perfect it is far better than trying to be > "cooperative" in your code. > > > > My 2 cents, > > /thomas > > > You have a problem, and you decide to solve it via webworkers. > > Later you'll WISH you'd chosen regexs instead :-) > Why the cynicism? Isn't it true that the only way to make use of multiple cores in browser is to use WebWorkers? > > -- > Mike > > -- > Note that posts from new members are moderated - please be patient with > your first post. > --- > You received this message because you are subscribed to the Google Groups > "ClojureScript" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojurescript+unsubscr...@googlegroups.com. > To post to this group, send email to clojurescript@googlegroups.com. > Visit this group at http://groups.google.com/group/clojurescript. > -- Kind Regards, Atamert Ölçgen ◻◼◻ ◻◻◼ ◼◼◼ www.muhuk.com -- Note that posts from new members are moderated - please be patient with your first post. --- You received this message because you are subscribed to the Google Groups "ClojureScript" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscr...@googlegroups.com. To post to this group, send email to clojurescript@googlegroups.com. Visit this group at http://groups.google.com/group/clojurescript.
Re: [ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
On Tuesday, August 25, 2015 at 7:45:28 PM UTC+10, Thomas Heller wrote: > On Tuesday, August 25, 2015 at 10:59:53 AM UTC+2, Daniel Kersten wrote: > > "The browser does not support threads so neither can core.async." > > > > > > To expand on that, core.async uses cooperative multitasking, which means > > you have to give control back every so often so it can schedule other go > > blocks to be run. Calls like > timeout works). > > > > While that is correct let me emphasize that timeout is not a solution! > > Do you always know how long task X will run or whether you are going to need > to chunk it? Is it even possible to split up? A "task" that may complete in > 10ms on your machine might take 100ms on another one or even 500ms on yours > if the computer is doing something else. > > If you need to do CPU intensive work in the browser use a WebWorker. It is > their purpose. While not perfect it is far better than trying to be > "cooperative" in your code. > > My 2 cents, > /thomas You have a problem, and you decide to solve it via webworkers. Later you'll WISH you'd chosen regexs instead :-) -- Mike -- Note that posts from new members are moderated - please be patient with your first post. --- You received this message because you are subscribed to the Google Groups "ClojureScript" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscr...@googlegroups.com. To post to this group, send email to clojurescript@googlegroups.com. Visit this group at http://groups.google.com/group/clojurescript.
Re: [ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
On Tuesday, August 25, 2015 at 10:59:53 AM UTC+2, Daniel Kersten wrote: > "The browser does not support threads so neither can core.async." > > > To expand on that, core.async uses cooperative multitasking, which means you > have to give control back every so often so it can schedule other go blocks > to be run. Calls like works). > While that is correct let me emphasize that timeout is not a solution! Do you always know how long task X will run or whether you are going to need to chunk it? Is it even possible to split up? A "task" that may complete in 10ms on your machine might take 100ms on another one or even 500ms on yours if the computer is doing something else. If you need to do CPU intensive work in the browser use a WebWorker. It is their purpose. While not perfect it is far better than trying to be "cooperative" in your code. My 2 cents, /thomas -- Note that posts from new members are moderated - please be patient with your first post. --- You received this message because you are subscribed to the Google Groups "ClojureScript" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojurescript+unsubscr...@googlegroups.com. To post to this group, send email to clojurescript@googlegroups.com. Visit this group at http://groups.google.com/group/clojurescript.
Re: [ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
"The browser does not support threads so neither can core.async."
To expand on that, core.async uses cooperative multitasking, which means
you have to give control back every so often so it can schedule other go
blocks to be run. Calls like wrote:
> This has nothing to do with core.async or the go macro. If you block the
> CPU you will block the UI. The browser does not support threads so neither
> can core.async.
>
> What the timeout achieves is that you give some control back to the
> browser so instead of 1sec "blocking" you get 5x200ms blocking. Still far
> from 60fps but better than 0.
>
> If you want to do CPU intensive work without blocking the UI you'll need
> to use one or more WebWorkers. You could layer core.async on top of that to
> communicate between the Workers and the UI but core.async itself does
> nothing in that regard.
>
> WebWorkers are pretty easy to get going but carry their own set of
> drawbacks so it is very dependent on your use case whether they make sense
> or not.
>
> HTH,
> /thomas
>
>
>
> On Monday, August 24, 2015 at 5:33:48 PM UTC+2, Timothy Pratley wrote:
> > I have an issue where work inside a 'go block' is blocking my UI.
> > Inserting a ( >
> > ** Is there a more idiomatic way? **
> >
> > Here is a cut down version built from a fresh template:
> > If you run this code, you don't see the thinking deeply countdown, and
> you cannot click the other button.
> >
> > If you uncomment the timeout 1, it works better.
> >
> >
> >
> > (defn hello-world []
> > [:div
> >[:h1 (:text @app-state)]
> >[:button
> > {:on-click
> > (fn [e]
> >(swap! app-state assoc :thinking true :thinking-deeply 5)
> >(go
> > (while (pos? (:thinking-deeply @app-state))
> >;( >(apply * (rest (range 1000)))
> >;( >(swap! app-state update :thinking-deeply dec))
> > (swap! app-state dissoc :thinking :thinking-deeply)))}
> > "The button!"]
> >(when (:thinking-deeply @app-state)
> > [:h2 "Thinking deeply " (:thinking-deeply @app-state)])
> >(when (:thinking @app-state)
> > [:h2 "Thinking"])
> >[:button "The other button"]])
> >
> > Full source code is attached
> >
> >
> > Regards,
> > Timothy
>
> --
> Note that posts from new members are moderated - please be patient with
> your first post.
> ---
> You received this message because you are subscribed to the Google Groups
> "ClojureScript" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojurescript+unsubscr...@googlegroups.com.
> To post to this group, send email to clojurescript@googlegroups.com.
> Visit this group at http://groups.google.com/group/clojurescript.
>
--
Note that posts from new members are moderated - please be patient with your
first post.
---
You received this message because you are subscribed to the Google Groups
"ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojurescript+unsubscr...@googlegroups.com.
To post to this group, send email to clojurescript@googlegroups.com.
Visit this group at http://groups.google.com/group/clojurescript.
[ClojureScript] Re: Clojurescript async not yeilding for CPU intensive work
This has nothing to do with core.async or the go macro. If you block the CPU
you will block the UI. The browser does not support threads so neither can
core.async.
What the timeout achieves is that you give some control back to the browser so
instead of 1sec "blocking" you get 5x200ms blocking. Still far from 60fps but
better than 0.
If you want to do CPU intensive work without blocking the UI you'll need to use
one or more WebWorkers. You could layer core.async on top of that to
communicate between the Workers and the UI but core.async itself does nothing
in that regard.
WebWorkers are pretty easy to get going but carry their own set of drawbacks so
it is very dependent on your use case whether they make sense or not.
HTH,
/thomas
On Monday, August 24, 2015 at 5:33:48 PM UTC+2, Timothy Pratley wrote:
> I have an issue where work inside a 'go block' is blocking my UI.
> Inserting a (
> ** Is there a more idiomatic way? **
>
> Here is a cut down version built from a fresh template:
> If you run this code, you don't see the thinking deeply countdown, and you
> cannot click the other button.
>
> If you uncomment the timeout 1, it works better.
>
>
>
> (defn hello-world []
> [:div
>[:h1 (:text @app-state)]
>[:button
> {:on-click
> (fn [e]
>(swap! app-state assoc :thinking true :thinking-deeply 5)
>(go
> (while (pos? (:thinking-deeply @app-state))
>;((apply * (rest (range 1000)))
>;((swap! app-state update :thinking-deeply dec))
> (swap! app-state dissoc :thinking :thinking-deeply)))}
> "The button!"]
>(when (:thinking-deeply @app-state)
> [:h2 "Thinking deeply " (:thinking-deeply @app-state)])
>(when (:thinking @app-state)
> [:h2 "Thinking"])
>[:button "The other button"]])
>
> Full source code is attached
>
>
> Regards,
> Timothy
--
Note that posts from new members are moderated - please be patient with your
first post.
---
You received this message because you are subscribed to the Google Groups
"ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojurescript+unsubscr...@googlegroups.com.
To post to this group, send email to clojurescript@googlegroups.com.
Visit this group at http://groups.google.com/group/clojurescript.
