Re: core.async consumer + producer working by chunk?

[Gary Verhaegen]

On 5 January 2018 at 19:44, Rob Nikander  wrote:

> Hi,
>
> I’m wondering if there is a core.async design idiom for this situation...
>
> - A buffered channel
> - One producer feeding it
> - A bunch of consumers pulling from it.
> - Producer should wake up and fill the channel only when it’s empty. In
> other words, the producer should work in chunks.
>
> My first idea is to have two channels. The second will be used by
> consumers to signal the producer that the primary channel is empty. But I'm
> wondering if there is a better way.
>
> The motive for this is that the producer is doing a DB query that is more
> efficient in bulk. `select ... limit 50` rather than `select ... limit 1`
> 50 times.
>
> Rob
>

What about simply having the producer put items one by one on the channel?

(ns t.core
  (:require [clojure.core.async :as async]))
(defn ap
  "atomic print"
  [m]
  (print (str (pr-str m) "\n")))
(defn produce-next-batch
  [s]
  (let [m (+ s 10)]
[(range s m) m]))
(defn chunked-producer
  [init-state]
  (let [result-chan (async/chan)]
(async/go
  (loop [[batch cursor] (produce-next-batch init-state)]
(ap {:produced batch})
(doseq [elem batch]
  (async/>! result-chan elem))
(recur (produce-next-batch cursor
result-chan))
(defn run-consumer
  [ch id n]
  (async/go
(dotimes [_ n]
  (ap {:id id :received (async/> [c1 c2 c3]
 (mapv async/

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Spec - is there any way to tell which keys are not described by a spec?

[Lucas Wiener]

Ah, yes that seems to be what I want exactly.

Thank you, I'll give it a shot.

Den fredag 5 januari 2018 kl. 20:53:55 UTC+1 skrev Josh Tilles:
>
> I think Stu Halloway’s proof-of-concept is at least close to what you 
> want: 
> https://gist.github.com/stuarthalloway/f4c4297d344651c99827769e1c3d34e9.
>
> (Here’s the context 
>  for that 
> code, in case you were curious.)
>
> btw, you might want to take a look at the spec-provider 
>  library, too. It’s not 
> exactly what you were asking for, but I think it could help you quickly 
> create (rough, approximate) specs that incorporate all witnessed keys.
>
> On Friday, January 5, 2018 at 11:49:15 AM UTC-5, Lucas Wiener wrote:
>>
>> Hi,
>>
>> I'm writing a spec for a fairly complex data structure. One thing that I 
>> have identified troublesome is that I currently have no clue how "well" my 
>> spec describes my data. I keep iterating the spec, thinking that I have 
>> described all fields but then it turns out later that I've missed 
>> something. I would love some kind of functionality that tells me which keys 
>> are present in my data that are not described in my spec. I'm aware of the 
>> design principle that a spec should not be limiting to having extra data 
>> and I totally support that. However, at development time I think it would 
>> be useful to have something that tells me "keys :x, :y, :z are not in the 
>> spec" or "the spec describes a subset of the given data". Is this possible?
>>
>> Kind Regards
>> Lucas Wiener
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async consumer + producer working by chunk?

[Brian J. Rubinton]

I don’t know; I don’t fully understand the implementation differences of >!! 
and offer!. The behavior of offer! makes me think the buffer is not empty until 
all the outputs of the transducer are consumed, but the behavior of >!! makes 
me think otherwise.

Moritz - is the buffer cleared if:
- it’s size is 1
- you’ve put 1 item onto the channel 
- that 1 item is transformed into 3 by the channel’s transducer
- and only 1 of the 3 items are taken from the channel?

Per the docstrings and the behavior it’s both not possible to put another value 
onto the channel immediately (so offer! returns nil) yet buffer space is 
available after only 1 of the 3 values on the channel are taken (so >!! doesn't 
block).
https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj#L138
 

https://github.com/clojure/core.async/blob/master/src/main/clojure/clojure/core/async.clj#L391
 


This seems like a weird corner case made possible by transducers. Maybe the 
collection put into the buffer is removed from the buffer and passed to the 
transducer on the first take, so the buffer is empty and that’s all >!! cares 
about, yet there are still values to be taken from the channel (from the 
transducer — not the buffer) and somehow that affects offer!’s behavior.


> On Jan 5, 2018, at 4:07 PM, Rob Nikander  wrote:
> 
> 
> 
> On Friday, January 5, 2018 at 4:00:25 PM UTC-5, Moritz Ulrich wrote:
> 
> You have a channel with a buffer-size of one. You clear the buffer by 
> taking one item from it, making room for another one. Therefore the put 
> succeeds. Try just `(async/chan nil xform)` to create a channel without 
> a backing buffer (a rendezvouz channel) where puts only succeed if 
> there's a matching consumer. 
> 
> Then why does Brian's code work the way it does? See how he takes 2 things 
> off the channel, and offer! still fails and returns nil. If offer! is 
> returning nil, shouldn't >!! be blocking?
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ClojureScript] Re: durable datascript experiments

[Christian Weilbach]

It now has its own repo as datahike and is on clojars to play around.

https://github.com/replikativ/datahike

We are also working on replication of the hitchhiker-tree here:

https://gitlab.com/replikativ/index-sync

So hopefully we will be able to stream replicate a datahike db soon.

Best,
Christian

On 31.12.2017 00:13, Laurens Van Houtven wrote:
> Whoa; this looks awesome. Thanks for publishing this.
> 
> On Sat, Dec 30, 2017 at 2:24 PM, Christian Weilbach
> > wrote:
> 
> Hi,
> 
> performance is now a lot better, ~3-5x slower than datascript for
> queries on the fractal index trees on my machine. This still needs
> tuning ofc., but I have tried to make it fast with a lot of profiling
> during idle times at 34C3. The fractal tree causes some overhead though.
> Inserts are a lot more expensive, but I expect them to be written to
> disk anyway. Therefore comparing inserts/deletes with datascript is a
> bit pointless, if you do everything in memory then it will always be
> more performant.
> 
> So it should be usable for small test projects now. GC is still TODO,
> but I would be really happy for feedback! We are developing this as a
> toolbox for distributed dataprocessing architectures.
> 
> Best,
> Christian
> 
> On 26.12.2017 12:18, Christian Weilbach wrote:
> > Hi,
> >
> > I have finally taken the time yesterday in a good end-of-year
> tradition
> > of ambitious hacks to bring the hitchhiker-tree (1) and datascript (2)
> > together. I have only touched the db.cljc namespace in datascript and
> > replaced all calls to the in-memory balanced-tree set (btset) with
> calls
> > to the hitchhiker-tree, the rest is vanilla DataScript. All tests pass
> > and I can store and load databases sufficiently with the three
> trees for
> > the indices (no datascript serialization used).
> >
> > I would like to have early feedback, so feel free to play and throw
> > ideas around. You need to install my version of the
> hitchhiker-tree (1)
> > first. Atm. it is a drop-in replacement for DataScript, but this is a
> > prototype, don't use it for anything serious yet.
> >
> > Here is an example interaction with the store:
> >
> >
> 
> https://github.com/whilo/datascript/blob/hitchhiker_tree_support/test/datascript/test/store.cljc
> 
> 
> >
> > Caveats:
> > 1. There is no working GC for the hitchhiker-trees yet.
> > 2. Performance penalty against DataScript is in the 20-100x range,
> but I
> > haven't done serious profiling yet.
> > 3. There are probably bugs that are not covered by the tests, please
> > open an issue.
> >
> > Goals:
> > This is more of an intermediary goal, although it is very useful
> to have
> > a durable datascript around in itself. The hitchhiker-tree is a
> generic
> > persistent durable datastructure that can be efficiently synched and
> > hence replication is the goal. Next step is a "web-after-tomorrow" (3)
> > architecture similar to datsync, that we are discussing about here:
> >
> > https://gitter.im/metasoarous/datsync
> 
> > https://gitter.im/replikativ/replikativ
> 
> >
> > In the longer run we can allow multiple writers with different
> > conflict-resolution scheme similar to CRDTs and the bloom language
> > (dedalus).
> >
> >
> > Happy hacking :),
> > Christian
> >
> > (1) https://github.com/whilo/hitchhiker-tree
> 
> > (2)
> https://github.com/whilo/datascript/tree/hitchhiker_tree_support
> 
> > (3) http://tonsky.me/blog/the-web-after-tomorrow/
> 
> >
> 
> 
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> 
> Note that posts from new members are moderated - please be patient
> with your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> 
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> 
> ---
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to 

Re: core.async consumer + producer working by chunk?

[Rob Nikander]

On Friday, January 5, 2018 at 4:00:25 PM UTC-5, Moritz Ulrich wrote:
>
>
> You have a channel with a buffer-size of one. You clear the buffer by 
> taking one item from it, making room for another one. Therefore the put 
> succeeds. Try just `(async/chan nil xform)` to create a channel without 
> a backing buffer (a rendezvouz channel) where puts only succeed if 
> there's a matching consumer. 
>

Then why does Brian's code work the way it does? See how he takes 2 things 
off the channel, and offer! still fails and returns nil. If offer! is 
returning nil, shouldn't >!! be blocking?



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async consumer + producer working by chunk?

[Moritz Ulrich]

Rob Nikander  writes:

> Thanks for the explanation! This is very close to what I want. I see some 
> confusing behavior though. See below.
>
> On Friday, January 5, 2018 at 2:40:14 PM UTC-5, Brian J. Rubinton wrote:
>>
>>
>> The work-queue channel has a fixed buffer size of 1. A collection (range 
>> 50) is put on the channel. While consumers can take items off the channel — 
>> note the individual contents of (range 50) are returned — a producer cannot 
>> put another value onto the channel until the entire initial collection is 
>> consumed. 
>>
>
> I tried your example code works for me, but then I tried using `>!!` 
> instead of `offer!` and it behaves differently.
>
> user> (def c (chan 1 (mapcat identity)))
> => #'user/c
> user> (do 
>(async/thread (async/>!! c [1 2 3]) (println "put #1"))
>(async/thread (async/>!! c [4 5 6]) (println "put #2"))
>(async/thread (async/>!! c [7 8 9]) (println "put #3")))
> put #1
>
> As expected it prints "put #1" right away, and other threads are blocked. 
> But...
>
> user> (poll! c)
> => 1
> put #2
> user> (poll! c)
> => 2
> put #3
> 
> As soon as your read anything from it, other puts succeed. Shouldn't they 
> be blocked?

You have a channel with a buffer-size of one. You clear the buffer by
taking one item from it, making room for another one. Therefore the put
succeeds. Try just `(async/chan nil xform)` to create a channel without
a backing buffer (a rendezvouz channel) where puts only succeed if
there's a matching consumer.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Git Deps for Clojure!

[Alex Miller]

On Friday, January 5, 2018 at 2:08:50 PM UTC-6, Gary Trakhman wrote:
>
> Specifically, I'm asking about how to resolve duplicate transitive 
> dependencies to actual code, possibly referring to the same lib repo at 
> different git hashes, or different git repos.
>

Ah, good question. The answer is that you need some way to detect version 
differences and resolve them. Fortunately, this is the identical problem 
you have in Maven and the solution is effectively the same - pick the newer 
one. In a git context this means "the commit that has the other as an 
ancestor". With a growth mindset 
, this is a safe choice. If 
two versions do not share a ancestor/descendant lineage, then your build is 
broken and you will be required to fix it.
 

> I think you answered, 'how do you get a single graph of deps nodes given 
> different kinds of deps'?
>
> For example:
> A->B->C#some_tag
> A->D->C#other_tag
>
> So, assuming you use a lot of git deps, which I think is the long-term 
> intent here based on the messaging, resolving those pointers to actual 
> loaded code is a potential problem that needs to be addressed somehow.  I 
> mentioned one way, which is to have multiple copies of a lib loaded 
> automatically.
>

Not doing that.
 

>
> From what I can tell so far, it seems like you can override a dep's 
> specific transitive deps, but that sounds painful when scaled out over 
> number of deps and how many more commit hashes exist than maven release 
> tags for each lib.
>

Either the descendant-most version will be chosen or your deps are broken 
and will fail to compute a classpath.

Additionally, most Maven artifacts (certainly most in Maven central or 
clojars) contain SCM metadata about the git url and rev that they 
represent. This gives us the ability to compare git deps with Maven deps 
(via their git rev) and apply the same logic. Not yet done, but coming 
soon...
 

> Another way to do it might be to make the git repo url+branch itself could 
> be part of the namespace var mapping, but that seems far off based on the 
> way clojure namespaces currently work, and doesn't cover the common case 
> (at least it's what we're used to with maven) of just resolving a single 
> version for each dep.
>
> Is there some existing design thought along these lines that I've missed?
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async consumer + producer working by chunk?

[Rob Nikander]

Thanks for the explanation! This is very close to what I want. I see some 
confusing behavior though. See below.

On Friday, January 5, 2018 at 2:40:14 PM UTC-5, Brian J. Rubinton wrote:
>
>
> The work-queue channel has a fixed buffer size of 1. A collection (range 
> 50) is put on the channel. While consumers can take items off the channel — 
> note the individual contents of (range 50) are returned — a producer cannot 
> put another value onto the channel until the entire initial collection is 
> consumed. 
>

I tried your example code works for me, but then I tried using `>!!` 
instead of `offer!` and it behaves differently.

user> (def c (chan 1 (mapcat identity)))
=> #'user/c
user> (do 
   (async/thread (async/>!! c [1 2 3]) (println "put #1"))
   (async/thread (async/>!! c [4 5 6]) (println "put #2"))
   (async/thread (async/>!! c [7 8 9]) (println "put #3")))
put #1

As expected it prints "put #1" right away, and other threads are blocked. 
But...
   
user> (poll! c)
=> 1
put #2
user> (poll! c)
=> 2
put #3

As soon as your read anything from it, other puts succeed. Shouldn't they 
be blocked?

Rob 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Git Deps for Clojure!

[Gary Trakhman]

Specifically, I'm asking about how to resolve duplicate transitive
dependencies to actual code, possibly referring to the same lib repo at
different git hashes, or different git repos.

I think you answered, 'how do you get a single graph of deps nodes given
different kinds of deps'?

For example:
A->B->C#some_tag
A->D->C#other_tag

So, assuming you use a lot of git deps, which I think is the long-term
intent here based on the messaging, resolving those pointers to actual
loaded code is a potential problem that needs to be addressed somehow.  I
mentioned one way, which is to have multiple copies of a lib loaded
automatically.

>From what I can tell so far, it seems like you can override a dep's
specific transitive deps, but that sounds painful when scaled out over
number of deps and how many more commit hashes exist than maven release
tags for each lib.

Another way to do it might be to make the git repo url+branch itself could
be part of the namespace var mapping, but that seems far off based on the
way clojure namespaces currently work, and doesn't cover the common case
(at least it's what we're used to with maven) of just resolving a single
version for each dep.

Is there some existing design thought along these lines that I've missed?

On Fri, Jan 5, 2018 at 2:39 PM Alex Miller  wrote:

> On Fri, Jan 5, 2018 at 1:20 PM, Gary Trakhman 
> wrote:
>
>> Congrats on the release! It's exciting to see this vision move forward.
>> Wondering if the logical next step is the npm style of trees of duplicate
>> transitive git deps.  In general, how is this going to work for transitive
>> deps?
>>
>
> tools.deps has the ability to read a file-based manifest in a git dep (or
> a local dep) and traverse its transitive dependencies. This works now for
> deps.edn based libraries and can be extended for other manifest types.
>
> I have started work on a pom.xml manifest reader. lein is harder to do
> right, but maybe easy to do badly. :)  There are still some things to work
> out wrt how manifest readers like this are dynamically found and loaded;
> currently they have to be "in the box" but I see that's not the final end
> state.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Spec - is there any way to tell which keys are not described by a spec?

[Josh Tilles]

I think Stu Halloway’s proof-of-concept is at least close to what you want: 
https://gist.github.com/stuarthalloway/f4c4297d344651c99827769e1c3d34e9.

(Here’s the context 
 for that 
code, in case you were curious.)

btw, you might want to take a look at the spec-provider 
 library, too. It’s not 
exactly what you were asking for, but I think it could help you quickly 
create (rough, approximate) specs that incorporate all witnessed keys.

On Friday, January 5, 2018 at 11:49:15 AM UTC-5, Lucas Wiener wrote:
>
> Hi,
>
> I'm writing a spec for a fairly complex data structure. One thing that I 
> have identified troublesome is that I currently have no clue how "well" my 
> spec describes my data. I keep iterating the spec, thinking that I have 
> described all fields but then it turns out later that I've missed 
> something. I would love some kind of functionality that tells me which keys 
> are present in my data that are not described in my spec. I'm aware of the 
> design principle that a spec should not be limiting to having extra data 
> and I totally support that. However, at development time I think it would 
> be useful to have something that tells me "keys :x, :y, :z are not in the 
> spec" or "the spec describes a subset of the given data". Is this possible?
>
> Kind Regards
> Lucas Wiener
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async consumer + producer working by chunk?

[Brian J. Rubinton]

The `mapcat` transducer takes a collection as its input and outputs each of its 
items individually. This example might be helpful:

user> (use ‘[clojure.core.async])
nil
user> (def work-queue (chan 1 (mapcat identity)))
#’user/work-queue
user> (offer! work-queue (range 50))
true
user> ( ( (offer! work-queue (range 50))
nil
user> (dotimes [_ 48] ( (offer! work-queue (range 50))
true
user> ( On Jan 5, 2018, at 2:10 PM, Rob Nikander  wrote:
> 
> 
> 
> On Friday, January 5, 2018 at 2:03:00 PM UTC-5, Brian J. Rubinton wrote:
> 
> What is the buffered channel’s buffer used for? If that’s set to 1 and the 
> channel’s transducer is `(mapcat identity)` then the producer should be able 
> to continuously put chunks of work onto the channel with the puts only 
> completing when the previous chunk is completely consumed.
> 
> The buffer is sort of a work queue, used to break up and distribute one 
> "chunk" to a pool of consumers. I was imaging the buffer would be size 50. So 
> the producer would grab 50 rows (one "chunk") and put them all in the 
> channel. Consumers would pick them out one by one.
> 
> I'm not familiar with the transducer concept so I need to go read a bit 
> before I can understand your point about the channel's transducer being 
> `(mapcat identity)`.
> 
>  
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Git Deps for Clojure!

[Alex Miller]

On Fri, Jan 5, 2018 at 1:20 PM, Gary Trakhman 
wrote:

> Congrats on the release! It's exciting to see this vision move forward.
> Wondering if the logical next step is the npm style of trees of duplicate
> transitive git deps.  In general, how is this going to work for transitive
> deps?
>

tools.deps has the ability to read a file-based manifest in a git dep (or a
local dep) and traverse its transitive dependencies. This works now for
deps.edn based libraries and can be extended for other manifest types.

I have started work on a pom.xml manifest reader. lein is harder to do
right, but maybe easy to do badly. :)  There are still some things to work
out wrt how manifest readers like this are dynamically found and loaded;
currently they have to be "in the box" but I see that's not the final end
state.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Git Deps for Clojure!

[Gary Trakhman]

Congrats on the release! It's exciting to see this vision move forward.
Wondering if the logical next step is the npm style of trees of duplicate
transitive git deps.  In general, how is this going to work for transitive
deps?

On Fri, Jan 5, 2018 at 1:49 PM Alex Miller  wrote:

> Pleased to announce some new functionality for clj and tools.deps!
>
> https://clojure.org/news/2018/01/05/git-deps
>
> Additionally, there have been new releases of:
> - Brew clojure formula (to get it: brew upgrade clojure)
> - Linux clojure installer (see https://clojure.org/guides/getting_started
> for info)
> - tools.deps.alpha 
> - NEW tools.gitlibs 
>
> Other than git deps, "clj -Spom" for pom generation has some fixes and an
> addition to add Maven repositories if needed.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async consumer + producer working by chunk?

[Rob Nikander]

On Friday, January 5, 2018 at 2:03:00 PM UTC-5, Brian J. Rubinton wrote:
>
>
> What is the buffered channel’s buffer used for? If that’s set to 1 and the 
> channel’s transducer is `(mapcat identity)` then the producer should be 
> able to continuously put chunks of work onto the channel with the puts only 
> completing when the previous chunk is completely consumed.
>

The buffer is sort of a work queue, used to break up and distribute one 
"chunk" to a pool of consumers. I was imaging the buffer would be size 50. 
So the producer would grab 50 rows (one "chunk") and put them all in the 
channel. Consumers would pick them out one by one.

I'm not familiar with the transducer concept so I need to go read a bit 
before I can understand your point about the channel's transducer being 
`(mapcat identity)`.

 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: core.async consumer + producer working by chunk?

[Brian J. Rubinton]

Hi Rob,

What is the buffered channel’s buffer used for? If that’s set to 1 and the 
channel’s transducer is `(mapcat identity)` then the producer should be able to 
continuously put chunks of work onto the channel with the puts only completing 
when the previous chunk is completely consumed.

That would make the producer block/park until the previous chunk is consumed.

For your use case I think an interesting result of this approach (maybe a 
downside?) is initially 2 db queries would be performed, 1 whose result makes 
it onto the channel while 2’s result will be parked until the previous query’s 
result is consumed.

Brian

> On Jan 5, 2018, at 1:44 PM, Rob Nikander  wrote:
> 
> Hi,
> 
> I’m wondering if there is a core.async design idiom for this situation...
> 
> - A buffered channel 
> - One producer feeding it 
> - A bunch of consumers pulling from it.
> - Producer should wake up and fill the channel only when it’s empty. In other 
> words, the producer should work in chunks.
> 
> My first idea is to have two channels. The second will be used by consumers 
> to signal the producer that the primary channel is empty. But I'm wondering 
> if there is a better way.
> 
> The motive for this is that the producer is doing a DB query that is more 
> efficient in bulk. `select ... limit 50` rather than `select ... limit 1` 50 
> times.
> 
> Rob
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Git Deps for Clojure!

[Alex Miller]

Pleased to announce some new functionality for clj and tools.deps!

https://clojure.org/news/2018/01/05/git-deps

Additionally, there have been new releases of:
- Brew clojure formula (to get it: brew upgrade clojure)
- Linux clojure installer (see https://clojure.org/guides/getting_started 
for info)
- tools.deps.alpha 
- NEW tools.gitlibs 

Other than git deps, "clj -Spom" for pom generation has some fixes and an 
addition to add Maven repositories if needed.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

core.async consumer + producer working by chunk?

[Rob Nikander]

Hi,

I’m wondering if there is a core.async design idiom for this situation...

- A buffered channel 
- One producer feeding it 
- A bunch of consumers pulling from it.
- Producer should wake up and fill the channel only when it’s empty. In 
other words, the producer should work in chunks.

My first idea is to have two channels. The second will be used by consumers 
to signal the producer that the primary channel is empty. But I'm wondering 
if there is a better way.

The motive for this is that the producer is doing a DB query that is more 
efficient in bulk. `select ... limit 50` rather than `select ... limit 1` 
50 times.

Rob

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Spec - is there any way to tell which keys are not described by a spec?

[Lucas Wiener]

Hi,

I'm writing a spec for a fairly complex data structure. One thing that I 
have identified troublesome is that I currently have no clue how "well" my 
spec describes my data. I keep iterating the spec, thinking that I have 
described all fields but then it turns out later that I've missed 
something. I would love some kind of functionality that tells me which keys 
are present in my data that are not described in my spec. I'm aware of the 
design principle that a spec should not be limiting to having extra data 
and I totally support that. However, at development time I think it would 
be useful to have something that tells me "keys :x, :y, :z are not in the 
spec" or "the spec describes a subset of the given data". Is this possible?

Kind Regards
Lucas Wiener

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Advice on Shell Scripting with new "clojure" binary

[Alan Thompson]

Cool!  I had somehow missed that in the announcement.  Quick test on linux:

 > echo '(println "Yes!!!")' | time clojure
Clojure 1.9.0
user=> Yes!!!
nil
user=>
clojure  1.38s user   0.07s system   198% cpu   *0.730 total*


Well under a second, even with a full-power JVM to load.  Very convenient!

Alan

On Thu, Jan 4, 2018 at 6:14 PM, Sean Corfield  wrote:

> The `clojure` command just loads your script – it doesn’t call -main – if
> you had
>
>
>
>   ;; test.clj
>
>   (println “Loaded!”)
>
>
>
> And you ran `clojure test.clj` then it would print Loaded!
>
>
>
> So you could do:
>
>
>
>   #!/usr/bin/env clojure
>
>   (println (str “Hello, “ (first *command-line-args*)))
>
>
>
> (the clojure.main/main function binds the command line arguments to that
> var)
>
>
>
> This works on OS X – I haven’t tried it on Linux.
>
>
>
> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
> An Architect's View -- http://corfield.org/
>
> "If you're not annoying somebody, you're not really alive."
> -- Margaret Atwood
>
>
> --
> *From:* clojure@googlegroups.com  on behalf of
> Delon Newman 
> *Sent:* Thursday, January 4, 2018 3:26:43 PM
> *To:* Clojure
> *Subject:* Advice on Shell Scripting with new "clojure" binary
>
> How do I get command line arguments in a Clojure shell script using the
> new "clojure" binary?
>
> So for a file like:
>
>
> # file-name: hello
> #!/usr/bin/env clojure
>
> (defn -main [name]
>(println (str "Hello, " name)))
>
> and execute it like:
>
> ./hello John
>
> the "-main" function is not executed. Is there another method?
>
> Also, any additional advice with respect to using Clojure for shell
> scripting would be appreciated.
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Advice on Shell Scripting with new "clojure" binary

[Delon Newman]

Thanks, but I do need the JVM.

> On Jan 5, 2018, at 1:33 AM, Gary Verhaegen  wrote:
> 
> On 4 January 2018 at 23:26, Delon Newman  wrote:
>> Also, any additional advice with respect to using Clojure for shell
>> scripting would be appreciated.
> 
> Unless you have a specific reason to prefer the JVM to Node as a
> platform for your script, I'd encourage you to take a look at Planck
> and Lumo for shell scripting. Being based on Node allows them to start
> up much faster, which may or may not be important to your use-case.
> 
> http://planck-repl.org
> https://github.com/anmonteiro/lumo
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to a topic in the Google 
> Groups "Clojure" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/clojure/A3ied9_joLA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

New Year - New Functional Gig..

[Alex Mesropians]

Hey All,

Happy New Year! With a new year comes the opportunity to start your year 
with a new new venture!

If you haven't had a look yet then do check out our new and improved 
platform with all the latest and greatest Functional Jobs across the globe. 
Also with our blog having contributions from leading engineers across the 
globe its a great place to learn and share your experience within the space.

Check it out... Functional Works  

Thanks,
Alex 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[JOB] Telia Norge needs people interested in working with Clojure and ClojureScript (Oslo, Norway)

[Jakub Holý]

Hello! We here at Telia Norge are starting a new project using Clojure and 
ClojureScript and need people that know or are interested in learning and 
using these technologies.

So if you could consider becoming our employee and joining the team, let us 
know as soon as possible (you don't need to be decided yet, interest is 
enough).

We need to verify the viability of hiring employees to work with these 
technologies pretty soon to be able to go forward with the project. So if 
you are interested, let us know by January 12th! (The deadline for 
expressing interest might be extended but it isn't guaranteed. However 
hiring will go on for quite a while.)

Let me know at jakub.h...@telia.no if you are interested!

(PS: You can read some more about this at our blog http://techblog.telia.no/
)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Advice on Shell Scripting with new "clojure" binary

[Gary Verhaegen]

On 4 January 2018 at 23:26, Delon Newman  wrote:
> Also, any additional advice with respect to using Clojure for shell
> scripting would be appreciated.

Unless you have a specific reason to prefer the JVM to Node as a
platform for your script, I'd encourage you to take a look at Planck
and Lumo for shell scripting. Being based on Node allows them to start
up much faster, which may or may not be important to your use-case.

http://planck-repl.org
https://github.com/anmonteiro/lumo

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.