Re: core.async/close! locks on chans with pending transforms

2017-07-04 Thread Vitalie Spinu 

>> On Mon, Jul 03 2017 21:18, Timothy Baldridge wrote:

> This means, if you want to execute take correctly you must ensure that only
> one thread executes the take instance at one time, since channels already
> operate via a channel-level lock, it makes sense to run the transducer
> inside the channels' lock.

I looked at the code and got somewhat surprised that transducer can be
potentially called within all 3 methods - take!, put! and close!. Not sure if
this could be done differently, but given this implementation it indeed makes
sense to block all 3 methods.

I think the docs regarding the blocking behavior and the timing of transducer's
execution could be improved. I always naively assumed that transducers are
executed on the same thread within put!, but in fact xform is called only when
the value is added to the buffer and can happen both on current thread and on
the thread pool.

This is what I dug so far regarding put!:

1) (half-full buf and pending takes)
 -> transform and add to buf on the same thread

2) (nil buf or full buf) and pending takes
 -> dispatch to pending take on the thread pool
   
3) (half full buf and no pending takes)
 -> transform and add to buf on the same thread
 
4) (nil of full buf)
 -> append to puts list


I find 2 troublesome. It means that this line:

  
https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/channels.clj#L135

can be reached in the corner case when buffer is full and there are pending
takes. If so then the current val is dispatched to the leading pending take
bypassing buf completely. I think that line is intended to be reached only for
unbuffered chans. Am I mistaken here?


  Vitalie

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async/close! locks on chans with pending transforms

2017-07-03 Thread Timothy Baldridge 
A big reason they have to be run inside the lock is that they have to
operate in the context of the channel.

For example:

(chan (take 10))

Firstly we must recognize that transducer instances are *not* thread safe.
They should only ever be executed by one thread at a time. But channels
allow multiple puts and takes. Also channels are not processes, they are
simply rather complex lists with locks around them.

This means, if you want to execute take correctly you must ensure that only
one thread executes the take instance at one time, since channels already
operate via a channel-level lock, it makes sense to run the transducer
inside the channels' lock.

But that's also why it's always been recommended to stick with simple, non
cpu-intensive, non blocking operations in channel transducers.

Timothy

On Mon, Jul 3, 2017 at 4:30 PM, Kevin Downey  wrote:

> On 07/03/2017 03:12 PM, Vitalie Spinu wrote:
>
>>
>>
>>
>> On Monday, 3 July 2017 22:48:40 UTC+2, red...@gmail.com wrote:
>>
>>
>> Discussion of locks aside, doing blocking operations like io or >   >!! or basically anything that looks like it blocks and isn't >!
>> or > is a very bad idea in a transducer on a channel. You will (eventually)
>> block the threadpool and get deadlocks.
>>
>>
>>
>> Is this the limitation in general or only when I use the chan with
>> blocking transducer withing a go block?  Transducers are not run on a go
>> block, are they?
>>
>>
> It is kind of complicated, and I haven't traced through everything in
> channels.clj recently, but assume transducers are not magic, the
> computation has to occur on some real thread some where, and the best
> threads available to do that are the consumer thread and the producer
> thread. If those threads are both executing go blocks, and you have a
> transducer doing blocking stuff on a channel they operate on, you may not
> be technically "blocking" in the go block, but the channel machinery is now
> blocking before it yields control back to the go  block. Putting blocking
> operations in a transducer then operating on the channel from go blocks
> turns `!>` and ' blocks), in to operations that do block a whole OS thread, of which the
> core.async threadpool that go blocks execute on only has 8 by default.
>
> There is this sort of dance where control is handed back and forth between
> channels and go blocks without tying up a thread. When you introduce
> blocking in to the channel machinery (by making the channel execute a
> blocking operation as things are produced or consumed) then the dance stops
> and the thread waits.
>
> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com > clojure+unsubscr...@googlegroups.com>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> And what is good, Phaedrus,
> And what is not good—
> Need we ask anyone to tell us these things?
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to

Re: core.async/close! locks on chans with pending transforms

2017-07-03 Thread Kevin Downey 

On 07/03/2017 03:12 PM, Vitalie Spinu wrote:




On Monday, 3 July 2017 22:48:40 UTC+2, red...@gmail.com wrote:


Discussion of locks aside, doing blocking operations like io or !! or basically anything that looks like it blocks and isn't >!
or Is this the limitation in general or only when I use the chan with 
blocking transducer withing a go block?  Transducers are not run on a go 
block, are they?




It is kind of complicated, and I haven't traced through everything in 
channels.clj recently, but assume transducers are not magic, the 
computation has to occur on some real thread some where, and the best 
threads available to do that are the consumer thread and the producer 
thread. If those threads are both executing go blocks, and you have a 
transducer doing blocking stuff on a channel they operate on, you may 
not be technically "blocking" in the go block, but the channel machinery 
is now blocking before it yields control back to the go  block. Putting 
blocking operations in a transducer then operating on the channel from 
go blocks turns `!>` and '(just park go blocks), in to operations that do block a whole OS thread, 
of which the core.async threadpool that go blocks execute on only has 8 
by default.


There is this sort of dance where control is handed back and forth 
between channels and go blocks without tying up a thread. When you 
introduce blocking in to the channel machinery (by making the channel 
execute a blocking operation as things are produced or consumed) then 
the dance stops and the thread waits.



--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with 
your first post.

To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google 
Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to clojure+unsubscr...@googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.



--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async/close! locks on chans with pending transforms

2017-07-03 Thread Vitalie Spinu 



On Monday, 3 July 2017 22:48:40 UTC+2, red...@gmail.com wrote:
>
>
> Discussion of locks aside, doing blocking operations like io or   >!! or basically anything that looks like it blocks and isn't >! or  is a very bad idea in a transducer on a channel. You will (eventually) 
> block the threadpool and get deadlocks. 
>


Is this the limitation in general or only when I use the chan with blocking 
transducer withing a go block?  Transducers are not run on a go block, are 
they?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async/close! locks on chans with pending transforms

2017-07-03 Thread Vitalie Spinu 


> the side-effect of this means that no other operation (puts, takes or 
closes)

Is there a deeper reason for this beside the ease of implementation?

 If chan is buffered I still fail to see why should close and take block. 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async/close! locks on chans with pending transforms

2017-07-03 Thread Kevin Downey 

On 07/03/2017 11:03 AM, Vitalie Spinu wrote:


Hi,

Async/close! causes deadlocks if its reducer is stalled (e.g. waits for 
an event from

  another chan).

Consider:

   (let [d (chan)
 s (chan 1 (map (fn [v]
  (println "this:" v)
  (println "from d:" (

Discussion of locks aside, doing blocking operations like io or >!! or basically anything that looks like it blocks and isn't >! or is a very bad idea in a transducer on a channel. You will (eventually) 
block the threadpool and get deadlocks. If you must use a transducer 
that does blocking operations, you should checkout pipeline-blocking. 
pipeline-blocking has a slightly convoluted internal structure exactly 
to avoid blocking operations running on the go block threadpool.


The transducer will be executed when publishing to a channel(this is a 
little bit squishy, sometimes publishing to a channel doesn't actually 
publish because the channel is full and a callback is registered to do 
the publishing the next time a consumer takes), if you publish to a 
channel with a transducer that blocks from a go-block, then you are 
blocking that go block which is bad.



  v)))]
 (go (>! s 1))
 (Thread/sleep 100)
 (println "closing s")
 (async/close! s))

   ;; =>
   ;; this: 1
   ;; closing s
   ;; .. [lock]

This is caused by (.lock mutex) in close! method here:


https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/channels.clj#L247


I wonder if it is there "by design" or a bug. IMO it makes little sense 
to lock
external code simply because chan's reducer function is stalled. Just as 
close!

doesn't lock on pending puts it shouldn't stall on pending "half-puts".

I need this for systems with inter-dependent chans. In the above example
component 'd' is a dependency of 's', then system will halt in reverse 
order of

dependencies closing `s` first.

Thank you,

Vitalie

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with 
your first post.

To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google 
Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to clojure+unsubscr...@googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.



--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async/close! locks on chans with pending transforms

2017-07-03 Thread Timothy Baldridge 
Transducers on channels lock the channel while they are running. This is by
design. So yes, the side-effect of this means that no other operation
(puts, takes or closes) can succeed while the transducer is running.

So the short answer is: If you have code that can take awhile to run, don't
put it in a channel transducer, put it in `async/pipeline(-blocking)`
instead.

On Mon, Jul 3, 2017 at 12:03 PM, Vitalie Spinu  wrote:

>
> Hi,
>
> Async/close! causes deadlocks if its reducer is stalled (e.g. waits for an
> event from
>  another chan).
>
> Consider:
>
>   (let [d (chan)
> s (chan 1 (map (fn [v]
>  (println "this:" v)
>  (println "from d:" (  v)))]
> (go (>! s 1))
> (Thread/sleep 100)
> (println "closing s")
> (async/close! s))
>
>   ;; =>
>   ;; this: 1
>   ;; closing s
>   ;; .. [lock]
>
> This is caused by (.lock mutex) in close! method here:
>
>https://github.com/clojure/core.async/blob/master/src/
> main/clojure/clojure/core/async/impl/channels.clj#L247
>
> I wonder if it is there "by design" or a bug. IMO it makes little sense to
> lock
> external code simply because chan's reducer function is stalled. Just as
> close!
> doesn't lock on pending puts it shouldn't stall on pending "half-puts".
>
> I need this for systems with inter-dependent chans. In the above example
> component 'd' is a dependency of 's', then system will halt in reverse
> order of
> dependencies closing `s` first.
>
> Thank you,
>
>Vitalie
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
“One of the main causes of the fall of the Roman Empire was that–lacking
zero–they had no way to indicate successful termination of their C
programs.”
(Robert Firth)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

core.async/close! locks on chans with pending transforms

2017-07-03 Thread Vitalie Spinu 

Hi,

Async/close! causes deadlocks if its reducer is stalled (e.g. waits for an 
event from
 another chan). 

Consider:

  (let [d (chan)
s (chan 1 (map (fn [v]
 (println "this:" v)
 (println "from d:" (! s 1))
(Thread/sleep 100)
(println "closing s")
(async/close! s))

  ;; =>
  ;; this: 1
  ;; closing s
  ;; .. [lock]

This is caused by (.lock mutex) in close! method here:

   
https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async/impl/channels.clj#L247

I wonder if it is there "by design" or a bug. IMO it makes little sense to 
lock
external code simply because chan's reducer function is stalled. Just as 
close!
doesn't lock on pending puts it shouldn't stall on pending "half-puts".

I need this for systems with inter-dependent chans. In the above example
component 'd' is a dependency of 's', then system will halt in reverse 
order of
dependencies closing `s` first.

Thank you,

   Vitalie

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.