Re: Using transducers in a new transducing context

Thanks so much for your input Alex! It was a very helpful confirmation of the key conclusions arrived at in this thread, and I appreciate the additional elaborations you gave, especially the insight you passed on about the stateful transducers using `ArrayList`. I'm glad that I wasn't the only

Re: Using transducers in a new transducing context

Hey all, just catching up on this thread after the weekend. Rich and I discussed the thread safety aspects of transducers last fall and the intention is that transducers are expected to only be used in a single thread at a time, but that thread can change throughout the life of the transducing

Re: Using transducers in a new transducing context

I do have one last question for you on the topic. I'm thinking of using the below function within a core.async context — specifically within a `go` block: (defn reduce-indexed [f init xs] (reduce (let [i (mutable-long -1)] (fn [ret x] (f ret (mutable-swap! i inc) x)))

Re: Using transducers in a new transducing context

You make a very good point. I had been under the misimpression that you could make an `r/folder` out of any thread-safe transducer like so, and it would work out of the box: (defn broken-reducers-map-indexed [f coll] (r/folder coll (map-indexed-transducer-concurrently-multi-threaded f))) and

Re: Using transducers in a new transducing context

In your example transducer, the problem is with the `result` parameter. The specification of transducers is that the result of `(rf result x)` should be fed into the next call to `rf`. In other words: (-> result (rf x1) (rf x2) (rf x3))` trying to do that in a parallel context is next to

Re: Using transducers in a new transducing context

That makes sense about them not being designed for that use case. I would add, though, that transducers could certainly be used in a parallel context *if* the current transducer implementations were abstracted such that you could pass internal state generator and modifier functions and use the

Re: Using transducers in a new transducing context

Transducers were never designed to work in parallel context. So I'd define any behavior that arises from using the same transducers in multiple threads *at the same time*, as undefined behavior. On Sun, Apr 9, 2017 at 4:39 PM, Alexander Gunnarson < alexandergunnar...@gmail.com> wrote: > I should

Re: Using transducers in a new transducing context

I should add that, as Timothy pointed out, if multiple threads mutate and read the value but only one ever does so at a time, as is the case in `core.async`, then a volatile is sufficient. My preliminary conclusions above about volatiles apply only to concurrent mutation via e.g. `fold` or the

Re: Using transducers in a new transducing context

It looks that way to me too, Seth, though I'd have to comb over the details of the locks implemented there to give a reasoned opinion of my own. But yes, if that's the case, the volatile isn't adding anything. Anyway, I'm not trying to poke holes in the current implementation of transducers —

Re: Using transducers in a new transducing context

I'll defer to Timothy on the particulars of core.async but it looks like [1] the transducer in channel is protected by a lock. If that's the case volatile isn't adding anything in terms memory barriers. 1:

Re: Using transducers in a new transducing context

Thanks so much for your well-considered reply, Timothy! That makes sense about volatiles being used in e.g. core.async or core.reducers contexts where the reducing function that closes over the mutable value of the stateful transducer is called in different threads. Why, then, are

Re: Using transducers in a new transducing context

My guess is that partition-all and partition use non-volatile references because none of the built-in stuff will return control back to the caller at a finer resolution than output value (AFAIK). That's why take needs volatile but partition-all doesn't (because for take the state persists

Re: Using transducers in a new transducing context

The volatile! is needed for the case where a transducer is only used by one thread at a time, but the thread executing the transducer may change from one call to the next. This happens fairly often with core.async. If you used a non-atomic, non-volatile mutable field, the JVM would be free to

Re: Using transducers in a new transducing context

EDIT: Transducers are actually not safe in `fold` contexts as I thought: (let [f (fn [i x] (println (str "i " i " " (Thread/currentThread))) (flush) x) r-map-indexed #(r/folder %2 (map-indexed %1))] (->> [6 7 8 9 10] (r-map-indexed f) (r/fold 1 (fn ([] (vector)) ([x] x) ([a

Re: Use of volatile! in stateful transducers

EDIT: Transducers are not safe in `fold` contexts either: (let [f (fn [i x] (println (str "i " i " " (Thread/currentThread))) (flush) x) r-map-indexed #(r/folder %2 (map-indexed %1))] (->> [6 7 8 9 10] (r-map-indexed f) (r/fold 1 (fn ([] (vector)) ([x] x) ([a b] (into a

Re: Use of volatile! in stateful transducers

I was wondering the same thing, Jörg. This thread talks about it as well. I posted a note there which I will reproduce here for your convenience: I think it's safe to assume that since `ArrayList` uses unsynchronized mutability

Re: Using transducers in a new transducing context

I was wondering the same thing, shintotomoe. This thread talks about it as well. I think it's safe to assume that since `ArrayList` uses unsynchronized mutability internally (a quick review of the GrepCode entry for `ArrayList`