[ANN] find-deps - quickly find and add dependencies to your deps.edn file

[Luke Burton]

find-deps helps you add dependencies to your deps.edn file by searching the 
Clojars and Maven APIs. By default, the top match is picked using a simple 
ranking algorithm that considers fuzzy text match, newness, and popularity of 
the coordinate's group ID. 

https://github.com/hagmonk/find-deps

deps.edn has encouraged me to break my larger projects into many smaller parts. 
Which means I now spend more time in the following loop:

0. … hmm but to do that I'll need a JSON parser.
1. Oh, I should use that fancy JSON library from metosin
2. Can I remember the coordinate? Uh, metosin/jsonista? Version something?
3. I should just pillage one of the deps.edn / project.clj files on my system.
4. Wait, in the time it will take me to type this I could just Google it.
5. I wonder what the browser tab limit is, because it took surprisingly long to 
open that one.
6. Ah ha, here's https://github.com/metosin/jsonista
7. Son-of-a- … coordinates are encoded as an image! I guess I'll just remember 
the version number.
8. Ok, I've updated the deps.edn file.
9. Time to blow away the REPL and all my application state before I can use it.
10. So, now that I'm ready to parse JSON … why did I want to parse JSON again?

In the course of writing this tool, I added no less than 7 dependencies to my 
deps.edn, which got easier and easier as I added more functionality. Right now, 
most of those steps can be consolidated down to:

clj -A:find-deps -F:save jsonista

Adding add-lib support to remove even the REPL restart is definitely something 
I want to look at next.

I hope this tool saves people some time, please don't hesitate to provide any 
feedback, and of course PRs are also welcome!

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] depify - generate deps.edn from project.clj

[Luke Burton]

depify reads your Leiningen project.clj configuration file and produces a 
deps.edn file suitable for the Clojure CLI tools.

GitHub: https://github.com/hagmonk/depify/ 

It's not meant to be an exhaustive replacement, just something to get you 
up-and-running with the basics, and to help you keep a project.clj and deps.edn 
in sync. Over time I may add more aliases to give your deps.edn more parity 
with the project.clj from which it's derived. PRs and feedback welcome!

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to cache deps with deps.edn, to speed docker image builds?

[Luke Burton]

> On Jun 3, 2018, at 2:55 PM, Alex Miller  wrote:
> 
> On Sunday, June 3, 2018 at 2:18:05 PM UTC-5, Luke Burton wrote:
> I have an outstanding pull request to add explicit copying of deps to a 
> target folder (like lein libdir) to depstar:
> 
> https://github.com/healthfinch/depstar/pull/3 
> <https://github.com/healthfinch/depstar/pull/3>
> 
> (for esoteric reasons one of the places I deploy demands I use "thin jars" 
> which is essentially the app and dependencies side-by-side in individual jars)
> 
> But if all you want to do is make sure you're caching the deps in their own 
> step in the dockerfile, just run something like clj -Stree before your build 
> step.
> 
> If you want to control the exact place where they end up instead of ~/.m2, 
> there's probably an environment variable somewhere to customize that.
> 
> You can set :mvn/local-repo in your deps.edn to use something other than 
> ~/.m2/repository. 

Oh, by the way, scraping GitHub with some search filters can often yield some 
good boilerplate to get a Dockerfile started. You can search for Dockerfiles 
that seem to be based off the clojure:tools.deps "official" image:

filename:dockerfile "clojure:tools.deps"

Aside: this yielded only 11 Dockerfiles :/ I'm building all my new projects 
using tools.deps targeting Kubernetes via Docker containers – if you're not 
doing this you're missing out :) Perhaps a blog post is called for …

Anyway, here's someone doing exactly what the OP requested:

https://github.com/FundingCircle/fc4c/blob/master/Dockerfile 
<https://github.com/FundingCircle/fc4c/blob/master/Dockerfile>

Slightly different search yields a couple more Dockerfiles (only 7, my 
admonishment still stands)

filename:dockerfile clojure "deps.edn"

One of which is for the "dispatchframework.io" Clojure image:

https://github.com/dispatchframework/clojure-base-image/blob/master/image-template/Dockerfile
 
<https://github.com/dispatchframework/clojure-base-image/blob/master/image-template/Dockerfile>

Which looks pretty cool. Anyone using it? https://github.com/vmware/dispatch 
<https://github.com/vmware/dispatch>


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to cache deps with deps.edn, to speed docker image builds?

[Luke Burton]

I have an outstanding pull request to add explicit copying of deps to a target 
folder (like lein libdir) to depstar:

https://github.com/healthfinch/depstar/pull/3 


(for esoteric reasons one of the places I deploy demands I use "thin jars" 
which is essentially the app and dependencies side-by-side in individual jars)

But if all you want to do is make sure you're caching the deps in their own 
step in the dockerfile, just run something like clj -Stree before your build 
step.

If you want to control the exact place where they end up instead of ~/.m2, 
there's probably an environment variable somewhere to customize that.

Luke.

> On Jun 3, 2018, at 11:30 AM, Jason Whitlark  wrote:
> 
> When building docker images with lein, I frequently have a run lein.deps 
> before building, to cache the deps in the image.  I'm not clean on how to 
> accomplish the same thing with deps.edn.  Any suggestions?
> 
> Cheers,
> ~Jason
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Schism, a set of CRDTs for Clojure and ClojureScript

[Luke Burton]

This is cool! 

There seems to be some room for a recommendation on how to apply these data 
structures effectively in practice, especially around the distribution 
mechanism. For instance, I can imagine storing these in an atom and using 
watchers to push changes out over zero-mq. Or is that the Worst Idea Ever™? I 
feel like the transit layer probably has *some* impact on whether CRDTs are the 
right solution to the problem, and I'm interested to know what those boundaries 
are.

Have you used them in production over the network, and if so, what solution did 
you use?

Thanks again!



> On Apr 18, 2018, at 5:57 PM, Alex Redington  wrote:
> 
> Good evening!
> 
> I submit for your evaluation and reasoned feedback a library I've been 
> working on to provide a set of convergent replicated data types to Clojure 
> and ClojureScript:
> 
> https://github.com/aredington/schism
> [com.holychao/schism "0.1.0"]
> 
> Schism provides convergent collections for sets, vectors, lists, and maps, 
> along with edn readers and writers out of the gate. My intent is to provide 
> you with all the tools necessary to replicate a collection across two or more 
> computing nodes.
> 
> Replication is a hard problem, so there are a few caveats to these tools as I 
> provide them:
> 
> - You must identify nodes within your cluster. You may choose to identify 
> nodes with a random UUID, in which case schism will help you to do so. Node 
> identifiers must be clojure serializable data.
> - Schism purposefully avoids carrying around a monotonically increasing 
> historical collection of data about deleted entries. Consequently there are 
> some ambiguities during convergence that may not exactly mirror local 
> modification.
> - Schism is only solving the problem of synchronizing two in memory sets. 
> Maintaining identity of those two sets, tracking state changes, and long term 
> durability are responsibilities left to the schism user.
> 
> Schism collections are persistent collections, so you should feel free to 
> work with them as a drop in replacement for a function which would work 
> against a Clojure collection. The usual utilities such as conj, assoc, 
> dissoc, disj, and rest are pure functions which will return you derived 
> copies, implicitly soing the convergence bookkeeping necessary in the 
> background. As you work with it, schism will maintain node and timestamp 
> information, with the goal of convergence providing the same result as if all 
> previous invocations had occurred on one local collection in memory. Schism's 
> requirements are your expectations of a Clojure collection, so hash, =, and 
> support for meta are all included, as well as many other functions defined 
> against Clojure's own collections. Particularly with hash and =, you can 
> expect a schism collection to return the same hashcode as its Clojure 
> equivalent, and to determine equality in the same way as its Clojure 
> equivalent would.
> 
> I've built schism in the absence of a direct itch to scratch with it. Should 
> you find that it betrays your expectations of Clojure collections, I'd 
> greatly appreciate a bug report and will work to correct schism quickly.
> 
> I hope that it assists you in solving your own problems.
> 
> -Alex
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Tack scheduler

[Luke Burton]

There may be other games in town these days, but I've had some success with 
http://clojurequartz.info , although I think I 
ended up just using the Java Quartz library directly.

Folks will chime (heh) in and recommend libraries like at-at, chime, or tick. 
However, once your app starts causing side effects in the real world, you want 
a scheduling system that has the remaining 80% of features you need, like:

- persistent storage of jobs
- logging and monitoring
- pause and resume
- dependencies between jobs
- configurable retry policies

Based on your additional criteria, I'm aware of no other scheduling system that 
satisfies all that, other than Quartz. Would be happy to hear other's 
recommendations too though.

Luke.


> On Apr 17, 2018, at 2:34 PM, Brjánn Ljótsson  > wrote:
> 
> Hi,
> 
> I'm about to implementera a task scheduler in a Clojure web app. It should do 
> various tasks such as cleaning up temporary files, send e-mail reminders to 
> users, and flagging database entries based on rules. It is currently 
> implemented in a php script which runs as a Cron job and executes task 
> objects (one for each task) that inherit from a task superclass that has 
> methods such as initialize, execute, and finish. But now I want to implement 
> it using Clojure idioms. 
> 
> Are there good libraries/examples of
> - A scheduling function that executes a task runner every X minutes?
> - An abstract description of a task with methods and timeout rules?
> - An extensible task list that can prioritize tasks according to conditions 
> such as time of day, time since last run, or other arbitrary conditions?
> 
> Thanks,
> Brjánn Ljótsson
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com 
> 
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com 
> 
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] clj-async-profiler — embeddable profiler with flame graphs, based on Java's async-profiler

[Luke Burton]

> On Dec 11, 2017, at 6:42 AM, Alexander Yakushev  wrote:
> 
> I've just released a wrapper around 
> https://github.com/jvm-profiling-tools/async-profiler that allows controlling 
> the profiler directly from the REPL of the program you want to profile. The 
> JAR file ships the profiling agent and the flamegraph generation script from 
> https://github.com/brendangregg/FlameGraph, so that you can profile your code 
> and instantly generate flame graphs from it.
> 
> Introductory blog post: 
> http://clojure-goes-fast.com/blog/profiling-tool-async-profiler/
> Repository: https://github.com/clojure-goes-fast/clj-async-profiler
> 
> Hope you'll like it!


Like it?? I had to *physically prevent* myself from dropping what I was doing 
to hack on this, already having a full plate at work … I've wanted something 
like this for such a long time! 

My first reaction is to create a pedestal interceptor that allows you to 
remotely trigger a flamegraph. I first saw this in rack-mini-profiler 
 though I'm sure others 
have done it too.

My second reaction is to wonder how difficult it would be to aggregate samples 
across a cluster of machines running the same code, similar to 
riemann-jvm-profiler .

My third reaction is to wonder how close you can get to having an "always on" 
flamegraph. I want a scrubber so I can reintroduce the time dimension. Imagine 
having a report of some perf issues and being able to say "let's go see what 
the flamegraph looked like at that time" - that'd be some Star Trek level 
observability right there.

Great work!

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to try/catch Let bindings?

[Luke Burton]

> On Oct 1, 2017, at 9:21 PM, Didier  wrote:
> 
> I can't emphasize enough the utility of the interceptor chain pattern, as 
> employed heavily in pedestal.
> 
> Interesting... Its almost like a workflow framework, but for simpler in code 
> workflows. I'm reluctant to have a dependency on pedestal just for this 
> though.

The dependencies required are pretty minimal, you just pull in the interceptor 
stuff by itself. Those guys did a good job of breaking pedestal into reusable 
pieces.

 [io.pedestal/pedestal.interceptor "0.5.2"]
   [io.pedestal/pedestal.log "0.5.2"]  
 [io.dropwizard.metrics/metrics-core "3.1.2"]  
   [org.clojure/core.match "0.3.0-alpha4" :exclusions 
[[org.clojure/clojurescript] [org.clojure/tools.analyzer.jvm]]]   

Plus core.async.

I've rolled my own mini-interceptor too. If you don't need the "leave" 
interceptor concept and are happy with one-way execution, we're just talking 
about pushing functions onto a queue. Each function takes a context map that 
includes the queue itself. Then, implement an executor that dequeues each 
function and executes it. Capture any errors at this point and short-circuit 
execution.

Having gone down that route I just ended up using pedestal.interceptor, after 
reading the source code and realizing it's so simple. I like using other 
people's heavily debugged code :) 


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to try/catch Let bindings?

[Luke Burton]

> On Sep 30, 2017, at 3:14 PM, Didier  wrote:
> 
> Is there another way to execute a set of complex steps which does not rely on 
> let and can be try/catched in the manner I describe?

I can't emphasize enough the utility of the interceptor chain pattern, as 
employed heavily in pedestal.

I use this *everywhere* now. I have code that interleaves interceptors into a 
chain to augment them with timing information, audit logs, and more. I have 
written code that maps a subset of an interceptor chain over a cluster of 
machines, then reduces the results back down locally. Most importantly, they 
are fantastically useful when it comes time to debug a complex business process 
… you can swap interceptors in and out easily, isolate stateful interceptors to 
enable better mocking during tests, and more.

Here's a basic application of interceptors. It's really not much code 
considering what you get. This doesn't even scratch the surface of what they 
can do.

(ns interceptor.test
  (:require
[io.pedestal.interceptor.chain :as chain]))

(def step1
  {:name  :step1
   :enter (fn [ctx]
(let [x "sheep"]
  (assoc ctx :result x)))})

(def step2
  {:name  :step2
   :error (fn [ctx ex-info]
(println "so much for step2:" (:exception-type (ex-data ex-info
   :enter (fn [{:keys [result] :as ctx}]

(println "time to transform" result "into a number")
(update ctx :result #(Long/parseLong %)))})

(defn run []
  (chain/execute {} [step1 step2]))


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: JDBC Connection auto-commit?: [ANN] clojure.java.jdbc 0.7.0 Beta 1

[Luke Burton]

I've been meaning to chime in here …

My take is that clojure.java.jdbc should not be responsible for providing 
options to the driver to guarantee streaming lazy results, as much as I 
understand how convenient that would be for users of the framework. It should 
of course support passing all of these options and not accidentally eat them :) 

My understanding so far is that JDBC has nothing to say about declaring a 
result set as streamable or lazy in the sense we're looking for. The ResultSet 
is stepped through item-by-item and it's an implementation detail of the driver 
as to whether that is happening across an in-memory buffer or being 
incrementally fetched from the network.

The PostgreSQL docs give a flavor of how many criteria have to be met before a 
ResultSet can be treated this way in the specific case of PostgreSQL. If any of 
these are violated it will happily fall back to buffering in-memory and you'll 
be none the wiser until you OOM:

https://jdbc.postgresql.org/documentation/head/query.html 


* >= V3 protocol
* no autocommit
* "forward only" ResultSet type
* single statement, no semicolons
* fetch size > 0

To me, what we need is a Clojure specific PostgreSQL framework that wraps 
clojure.java.jdbc and takes care of settings like this. It could also utilize 
pgjdbc-ng to provide access to LISTEN and NOTIFY, provide type conversions for 
special PG only types, capitalize on core.async channels, and anything else 
that could live in a more opinionated framework.

Luke.

> On Jul 5, 2017, at 10:31 AM, Sean Corfield  wrote:
> 
> Rather than doing something that requires a dependency on specific JDBC 
> driver classes, it seems that if an :auto-commit option in the db-spec were 
> honored by get-connection you would get what you needed:
>  
> (into [] (take 2) (jdbc/reducible-query (assoc config/db 
> :auto-commit false) query {:fetch-size 500}))
>  
> You can already pass additional options into the DriverManager/getConnection 
> call as properties (from the db-spec) but autocommit does not appear to be 
> supported in that format.
>  
> Actually, it would be cleaner if get-connection had a 2-arity accepting 
> db-spec and opts, and then everything could pass opts into get-connection and 
> you could do:
>  
> (into [] (take 2) (jdbc/reducible-query config/db query 
> {:fetch-size 500 :auto-commit false }))
>  
> Are there other settings that folks would find worthwhile to support here?
>  
> Feedback / comments: https://dev.clojure.org/jira/browse/JDBC-153 
> 
>  
> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
> An Architect's View -- http://corfield.org/ 
> 
> "If you're not annoying somebody, you're not really alive."
> -- Margaret Atwood
>  
> From: Ralf Schmitt 
> Sent: Wednesday, July 5, 2017 5:42 AM
> To: Sean Corfield ; Clojure Mailing List 
> 
> Subject: Re: [ANN] clojure.java.jdbc 0.7.0 Beta 1
>  
> Sean Corfield > writes:
>  
> >> The required steps to setup streaming are different from database to
> >> database and I guess they may change with the driver version being used.
> > 
> > This is really the crux of the problem here – I’m not sure what java.jdbc 
> > can do generically to make this much easier.
>  
> Please take a look at the attachment or
>  
>   https://gist.github.com/schmir/6e03b3d649950d0108a06bf6fd653dec 
> 
>  
> This is using the robert.hooke library to hook into prepare-statement.
> That hook calls into a protocol function if streaming is required. This
> protocol function can be defined by users for the database they are
> using.
>  
> I think it would be really nice for clojure.java.jdbc to provide this
> functionality.
>  
> --
> Cheers
> Ralf
>  
>  
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com 
> 
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com 
> 
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> 

Re: Stubbornly eager results in clojure.java.jdbc

[Luke Burton]

> On Jun 23, 2017, at 2:26 PM, Sean Corfield  wrote:
> 
> This is excellent news as far as I’m concerned because it shows there’s no 
> specific bug in clojure.java.jdbc that is fundamentally causing the OOM 
> problem you’re seeing!

I'm relieved too, given that I use clojure.java.jdbc extensively :) 

> (that’s not to say there aren’t _other_ bugs in clojure.java.jdbc and the 
> idea of the reducible result set definitely has appeal, which is why there’s 
> a JIRA issue for it, but it doesn’t appear to be necessary in this instance).

I've been trying to find a use case for the reducible result set, other than 
avoiding the memory issues I was experiencing, and there aren't too many that 
spring to mind. Probably the main one is mentioned in the JIRA, which is 
cleaning up clojure.java.jdbc internals. You could drop the need for sprinkling 
 ^:once everywhere.

One idea: as we have the constraint that regular result sets cannot be passed 
around, and must be consumed entirely in the context of their JDBC connection, 
perhaps they could be wrapped somewhat like `eduction` and represent the 
'recipe' for a query. At the start of reduce/iterator it would attempt to 
execute the query it owns. That would be kind of handy. Such a thing could be 
passed around to reduce or transduce, and would implement IReduceInit and 
clojure.lang.Sequential (apropos post from Alex: 
http://insideclojure.org/2015/01/18/reducible-generators/)

Another option that achieves similar goals is simply feeding the results to a 
channel. I seem to code this up myself on every new project :) 
clojure.java.jdbc internals could handle the bookkeeping of keeping the 
ResultSet alive until fully consumed, at which point the channel is closed. I'm 
sure this isn't the first such request, and that there are probably good 
reasons for it not having happened as yet.

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Stubbornly eager results in clojure.java.jdbc

[Luke Burton]

Well, shoot. I went back and revisited this because it was bugging me … I 
looked at the code generated with and without usage of ^:once fn* and that led 
me down the right path. TL;DR – the problem appears to be locals clearing 
always disabled in Cursive REPL. This is just my current hypothesis based on 
testing and not 100% certainty. I guess I'll put this in a blog post or 
something once I validate this hypothesis and get some feedback from Colin 
(1.6.0-eap1-2017.2 if you're watching, Colin ;)

For mailing list posterity, here's the difference, using Lubomir's examples and 
the decompiler in IntelliJ, between ^:once fn* and regular fn (or fn*)

Note the inner-bad function:

(defn bad []
  (let [x (for [n (range)] (make-array Object 1)), f (fn* inner-bad [] (nth 
x 1e6))] (f)))

The inner-bad class generated is:

public final class core$bad$inner_bad__431 extends AFunction {
  Object x;
  public static final Object const__1 = 100.0D;

  public core$bad$inner_bad__431(Object var1) {
this.x = var1;
  }

  public Object invoke() {
return RT.nth(this.x, RT.intCast((Number)const__1));
  }
}

Note the inner-good function:

(defn good []
  (let [x (for [n (range)] (make-array Object 1)), f (^:once fn* inner-good 
[] (nth x 1e6))] (f)))

The inner-good class generated is:

public final class core$good$inner_good__414 extends AFunction {
  Object x;
  public static final Object const__1 = 100.0D;

  public core$good$inner_good__414(Object var1) {
this.x = var1;
  }

  public Object invoke() {
Object var1 = this.x;
this.x = null;
return RT.nth(var1, RT.intCast((Number)const__1));
  }
}

You can see pretty clearly the difference is that each time the function is 
invoked in the "good" case, the local reference to this.x is cleared, meaning 
this function really is only good for one invocation. No wonder it's 
undocumented!

=> (let [z 1000
  f (^:once fn* one-shot [x] (println x))
  p (f z)]
  (p)
  (p))

1000
CompilerException java.lang.NullPointerException

Anyway, I saw this and jumped to the thought: "locals!! I have fallen victim to 
uncleared locals!!"


So I started some tests, going back to my previous code 
<https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e> and trying 
the JDBC result set that wasn't being lazy before. I put the test1 invocation 
into -main and tried:
lein run – no OOM.
lein repl, then invoking (-main) – no OOM.
lein repl :headless, then connecting via lein repl :connect "localhost:54321" 
and invoke (-main) – no OOM.
Cursive nREPL via Leiningen (locals clearing, debug REPL) – OOM!!
Cursive nREPL via Leiningen (no locals clearing, debug REPL) – OOM!!
Cursive nREPL via Leiningen (no locals clearing, regular REPL) – OOM!!
Cursive nREPL via Leiningen (no locals clearing, regular REPL) – OOM!!
lein run, with -Dclojure.compiler.disable-locals-clearing=true – OOM!!
I feel slightly better for knowing I can point the finger of blame at disabled 
locals clearing. It would be great to have this fixed in Cursive, considering 
that I run quite memory intensive applications and I'm sure locals clearing 
being permanently disabled has caused me to engineer around some non-existent 
problems. I have not tested CIDER or any other REPL, YMMV.

Luke.


> On Jun 20, 2017, at 11:26 AM, Lubomir Konstantinov <lubo...@gmail.com> wrote:
> 
> ^{:once true}
> 
> should deal with this. Try the following simplified test case with and 
> without it:
> 
> (defn query [result-fn]
>   (let [x (for [n (range 1e6)] (make-array Object 10)) 
> f (^:once fn* [rs] (result-fn rs))] (f x)))
> 
> (defn testq []
>   (let [myfn (fn [rs] (doseq [r rs] nil))]
> (query myfn)))
> 
> 
> 
> On Tuesday, 20 June 2017 03:47:45 UTC+3, James Reeves wrote:
> This might be a bug in java.jdbc. The code that passes the result set seq to 
> the function is:
> 
> (^{:once true} fn* [rset]
>   ((^{:once true} fn* [rs]
>  (result-set-fn (if as-arrays?
>   (cons (first rs)
> (map row-fn (rest rs)))
>   (map row-fn rs
>(result-set-seq rset opts)))
> 
> I'm wondering if this function holds onto the head of the seq, since it's 
> bound to "rs".
> 
> On 20 June 2017 at 00:20, Luke Burton <luke_...@me.com <>> wrote:
> 
> Anyone have any insights here? Really the most important thing I'm trying to 
> learn is 2) how to identify when a lazy seq head is being retained, other 
> than waiting for it to become bad enough that your program OOMs.
> 
> 
>> On Jun 16, 2017, at 6:14 PM, Luke Burton <luke_...@me.com <>> wrote:
>> 
>> 
>> Riddle me this:
>> 
>> https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e 
>> <https://gist.github.com/hagmonk/a

Re: Stubbornly eager results in clojure.java.jdbc

[Luke Burton]

Postgres, as mentioned in the mail and the linked source code. The problem at 
this point doesn't appear to be options given to the driver, since I show two 
implementations using the same driver options. One processes the results 
lazily, one does not. 

Now, I'm calling two different methods in clojure.java.jdbc, so there's a 
chance the driver options aren't being propagated correctly to the 
implementation in one of the methods. It's already soaked up way too much of my 
time (and I like the elegance of the reducible-result-set alternative) so I 
haven't revisited the code to explore further.


> On Jun 23, 2017, at 7:38 AM, r0man <roman.sche...@burningswell.com 
> <mailto:roman.sche...@burningswell.com>> wrote:
> 
> Hi Luke,
> 
> which database are you using? I had the same issue with MySQL recently. At 
> the end I got it working with clojure.java.jdbc.
> I don't have the code at hand, but according to the MySQL docs you have to 
> set the fetch size to Integer.MIN_VALUE.
> 
> https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-implementation-notes.html
>  
> <https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-implementation-notes.html>
> 
> r0man
> 
> On Saturday, June 17, 2017 at 3:15:13 AM UTC+2, Luke Burton wrote:
> 
> Riddle me this:
> 
> https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e 
> <https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e>
> 
> Wherein I synthesize a large table in Postgres, then attempt to lazily load 
> the table, discarding each row as I receive it. I tried *many* permutations 
> and experiments, but settled on these two tests to illustrate my point. Which 
> is that I simply can't get it to work with clojure.java.jdbc.
> 
> test1, according to all my research and reading of the source code involved, 
> should consume the query results lazily. It does not, and I can't for the 
> life of me figure out why. Traffic starts to stream in, and the heap is 
> overwhelmed almost immediately. I've deliberately set the heap to 1 GB.
> 
> test2 uses a technique I borrowed wholesale from Ghadi Shayban in JDBC-99 
> <https://dev.clojure.org/jira/browse/JDBC-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs>,
>  which is to have ResultSet implement IReduceInit. It consumes a nominal 
> amount of memory. I've verified it's actually doing something by putting 
> counters in, and using YourKit to watch about 20 MB/s of traffic streaming 
> into the JVM. It's brilliant, it doesn't even break 200 MB total heap usage.
> 
> I used YourKit to track where the memory is being retained for test1. 
> Initially I made the mistake of not setting the fetchSize, so I saw an 
> ArrayList inside the driver holding the reference. The driver documentation 
> <https://jdbc.postgresql.org/documentation/head/query.html> confirms that 
> autoCommit must be disabled and the fetchSize set to some non-zero number.
> 
> After making that change, YourKit confirmed that the GC root holding all the 
> memory was the stack local variable "rs". At least I think it did, as a 
> non-expert in this domain. I tried disassembling the functions using 
> no.disassemble and the IntelliJ decompiler but I'm not really at the point 
> where I understand what to look for.
> 
> So my questions are:
> 
> 1) what am I doing wrong with clojure.java.jdbc?
> 
> Note some things I've already tried:
> 
> * using row-fn instead of result-set-fn
> * using prepared statements
> * explicitly setting auto-commit false on the connection
> * declaring my result-set-fn with (^{:once true} *fn […]) (I did not see a 
> change in the disassembly when using this)
> * probably other things I am forgetting
> 
> 2) in these situations where you suspect that the head of a lazy sequence is 
> being retained, how do you reason about it? I'm kind of lucky this one blew 
> the heap so quickly, who knows how much of my production code might burning 
> memory unnecessarily but not quite as fatally. Do you disassemble the 
> functions and observe some smoking gun? How do you peek under the covers to 
> see where the problem is? 
> 
> Luke.
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com 
> <mailto:clojure@googlegroups.com>
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com 
> <mailto:clojure+unsubscr...@googlegroups.com>
> For more options, visit this group at
> http://groups.google.com

Re: Stubbornly eager results in clojure.java.jdbc

[Luke Burton]

Anyone have any insights here? Really the most important thing I'm trying to 
learn is 2) how to identify when a lazy seq head is being retained, other than 
waiting for it to become bad enough that your program OOMs.


> On Jun 16, 2017, at 6:14 PM, Luke Burton <luke_bur...@me.com> wrote:
> 
> 
> Riddle me this:
> 
> https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e 
> <https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e>
> 
> Wherein I synthesize a large table in Postgres, then attempt to lazily load 
> the table, discarding each row as I receive it. I tried *many* permutations 
> and experiments, but settled on these two tests to illustrate my point. Which 
> is that I simply can't get it to work with clojure.java.jdbc.
> 
> test1, according to all my research and reading of the source code involved, 
> should consume the query results lazily. It does not, and I can't for the 
> life of me figure out why. Traffic starts to stream in, and the heap is 
> overwhelmed almost immediately. I've deliberately set the heap to 1 GB.
> 
> test2 uses a technique I borrowed wholesale from Ghadi Shayban in JDBC-99 
> <https://dev.clojure.org/jira/browse/JDBC-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs>,
>  which is to have ResultSet implement IReduceInit. It consumes a nominal 
> amount of memory. I've verified it's actually doing something by putting 
> counters in, and using YourKit to watch about 20 MB/s of traffic streaming 
> into the JVM. It's brilliant, it doesn't even break 200 MB total heap usage.
> 
> I used YourKit to track where the memory is being retained for test1. 
> Initially I made the mistake of not setting the fetchSize, so I saw an 
> ArrayList inside the driver holding the reference. The driver documentation 
> <https://jdbc.postgresql.org/documentation/head/query.html> confirms that 
> autoCommit must be disabled and the fetchSize set to some non-zero number.
> 
> After making that change, YourKit confirmed that the GC root holding all the 
> memory was the stack local variable "rs". At least I think it did, as a 
> non-expert in this domain. I tried disassembling the functions using 
> no.disassemble and the IntelliJ decompiler but I'm not really at the point 
> where I understand what to look for.
> 
> So my questions are:
> 
> 1) what am I doing wrong with clojure.java.jdbc?
> 
> Note some things I've already tried:
> 
> * using row-fn instead of result-set-fn
> * using prepared statements
> * explicitly setting auto-commit false on the connection
> * declaring my result-set-fn with (^{:once true} *fn […]) (I did not see a 
> change in the disassembly when using this)
> * probably other things I am forgetting
> 
> 2) in these situations where you suspect that the head of a lazy sequence is 
> being retained, how do you reason about it? I'm kind of lucky this one blew 
> the heap so quickly, who knows how much of my production code might burning 
> memory unnecessarily but not quite as fatally. Do you disassemble the 
> functions and observe some smoking gun? How do you peek under the covers to 
> see where the problem is? 
> 
> Luke.
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> <http://groups.google.com/group/clojure?hl=en>
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> <mailto:clojure+unsubscr...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout 
> <https://groups.google.com/d/optout>.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Stubbornly eager results in clojure.java.jdbc

[Luke Burton]

Riddle me this:

https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e 


Wherein I synthesize a large table in Postgres, then attempt to lazily load the 
table, discarding each row as I receive it. I tried *many* permutations and 
experiments, but settled on these two tests to illustrate my point. Which is 
that I simply can't get it to work with clojure.java.jdbc.

test1, according to all my research and reading of the source code involved, 
should consume the query results lazily. It does not, and I can't for the life 
of me figure out why. Traffic starts to stream in, and the heap is overwhelmed 
almost immediately. I've deliberately set the heap to 1 GB.

test2 uses a technique I borrowed wholesale from Ghadi Shayban in JDBC-99 
,
 which is to have ResultSet implement IReduceInit. It consumes a nominal amount 
of memory. I've verified it's actually doing something by putting counters in, 
and using YourKit to watch about 20 MB/s of traffic streaming into the JVM. 
It's brilliant, it doesn't even break 200 MB total heap usage.

I used YourKit to track where the memory is being retained for test1. Initially 
I made the mistake of not setting the fetchSize, so I saw an ArrayList inside 
the driver holding the reference. The driver documentation 
 confirms that 
autoCommit must be disabled and the fetchSize set to some non-zero number.

After making that change, YourKit confirmed that the GC root holding all the 
memory was the stack local variable "rs". At least I think it did, as a 
non-expert in this domain. I tried disassembling the functions using 
no.disassemble and the IntelliJ decompiler but I'm not really at the point 
where I understand what to look for.

So my questions are:

1) what am I doing wrong with clojure.java.jdbc?

Note some things I've already tried:

* using row-fn instead of result-set-fn
* using prepared statements
* explicitly setting auto-commit false on the connection
* declaring my result-set-fn with (^{:once true} *fn […]) (I did not see a 
change in the disassembly when using this)
* probably other things I am forgetting

2) in these situations where you suspect that the head of a lazy sequence is 
being retained, how do you reason about it? I'm kind of lucky this one blew the 
heap so quickly, who knows how much of my production code might burning memory 
unnecessarily but not quite as fatally. Do you disassemble the functions and 
observe some smoking gun? How do you peek under the covers to see where the 
problem is? 

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slackpocalypse? next steps?

[Luke Burton]

> On May 25, 2017, at 3:09 PM, Sean Corfield <s...@corfield.org> wrote:
> 
> On 5/25/17, 12:43 PM, "Luke Burton" <clojure@googlegroups.com on behalf of 
> luke_bur...@me.com> wrote:
>> I've added the Discord invite link in this document
> 
> I gather that requires an app installed, not just a web browser? I clicked 
> the link, clicked to accept the invite, and got an error about a discord:// 
> link not being supported.
> 
> (this is the sort of discoverability / affordance issue that people complain 
> about)

There are web, desktop, and mobile versions of Discord. 

That's a curious workflow, one I hadn't seen before because I didn't need to 
invite myself to the Discord I made :) On a Mac, here's what happened when I 
tried after clicking the link:

- I was bounced to a "you've been invited to join" page (with a scary look Orc 
as the background) 
- I accepted the invite
- Safari popped a "do you want to open the Discord app?" dialog
- I hit cancel
- The Discord page was still there offering me a "Continue to Discord" button 
that opened the web app

Not sure what platform you're on, but their attempt to pop a discord:// link is 
surely not going to work in all cases :/ 

If it helps, the final resting place of my browser's URL when I made it into 
the web chat is below. It's quite a fearsome URL, I'm not sure if that's a 
session ID or a globally unique link for our server.

https://discordapp.com/channels/313110246643990528/313110246643990528 
<https://discordapp.com/channels/313110246643990528/313110246643990528>

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: slackpocalypse? next steps?

[Luke Burton]

I put together this "survey" a few days ago in the hope of starting to collect 
some structured feedback on the pros and cons of each service. Very much along 
the lines of what you're saying – I expect the end result would be "people have 
diverse needs and each service satisfies a different need", but it's nice to go 
through an exercise like this anyway, to proactively think about what those 
needs are.

Unfortunately I have not had time to drive this further, but if anyone would 
like to jump in and help that would be cool.

https://gist.github.com/hagmonk/284b1f2a6e5162f5b5a1b54cfa821318

I've added the Discord invite link in this document in case people missed it 
the first time.

A minor Discord update:

- we have about 200 people who signed up, and around 20 people online right 
now. So, not too many folks have gone out and experimented as yet. I encourage 
people who haven't experimented to give Discord a try, it has distinct 
strengths and weaknesses compared to Slack.

- we currently feel that Discord won't elegantly scale from a UI perspective to 
many hundreds of channels, since there is no notion of "joining" a channel – 
you see all of them all the time (grouped channels are landing soon, however). 
This promotes a different channel structure compared to Slack.

- the REPL bot code is Clojail'd and the code is up on GitHub (see channel for 
link). Discord's bot capabilities are pretty slick and high speed, there's a 
ton of cool stuff we could do. I genuinely think the repl-bot is territory 
worth exploring. It would not be too hard to give it almost notebook-like 
capabilities by allowing it to render images, as bots can attach arbitrary 
documents.

- default notifications have been fixed - people were getting buzzed by default 
way too much.

Ping me in Discord at @hagus if you would like settings changed, have 
questions, want to set up your own bot, etc!


> On May 24, 2017, at 4:18 PM, Sean Corfield  wrote:
> 
> I agree. There’s been a lot of hand-wringing about the potential impending 
> “Slackpocalypse” and it seems like some people think this is a problem that 
> needs to be “solved” (by consensus) – but, despite a lot of conversations (in 
> the #community-development channel on the Clojurians Slack primarily), no 
> “perfect solution” has yet been agreed… and to be honest that’s very unlikely 
> to happen: different people have different criteria for what is acceptable.
>  
> Putting on my Clojurians Admin Hat, what I’d like to see happen here is for 
> each proposed “solution”:
>  
> Post a new thread on this mailing list with a subject “Slack alternative? 
> ” – with a brief overview of what the system is, 
> what platforms it runs on (including native mobile if available) and, most 
> importantly, how to sign up and try it out for Clojure-related chatter. 
> Please include links to the service/product and other stuff you think is 
> relevant ☺
>  
> Until folks actually go and sign up and try out each service, we’re not going 
> to make progress.
>  
> As folks try each service, they can provide feedback in the relevant thread – 
> both positive as well as negative (constructively) please!
>  
> We’re almost certainly not going to find a replacement for Slack (or any 
> other communications medium) but we may find several new, additional ways to 
> communicate as a community.
>  
> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
> An Architect's View -- http://corfield.org/ 
> 
> "If you're not annoying somebody, you're not really alive."
> -- Margaret Atwood
>  
> On 5/23/17, 6:22 PM, "Colin Yates"   on behalf of colin.ya...@gmail.com 
> > wrote:
>  
> I've only been skimming this but "analysis paralysis" comes to mind :-). What 
> is the harm in establishing a presence in matrix (bagsy the "neo" handle) and 
> letting people know? As has been said, people will vote with their feet so if 
> in a months time matrix is a Clojure ghost town then lesson learned. 
>  
> Or, if I have missed some pertinent fact then by all means, sigh, tut and 
> mutter "sheesh, these drive by commenters are annoying" :-).
> 
> On Wednesday, 24 May 2017, Herwig Hochleitner  > wrote:
>> 2017-05-23 23:04 GMT+02:00 Colin Fleming < <>colin.mailingl...@gmail.com 
>> >:
>>> On 24 May 2017 at 00:13, Herwig Hochleitner < <>hhochleit...@gmail.com 
>>> > wrote:
 I doubt the whole community would want to move anywhere from Slack.
>>>  
>>> Perhaps this will have to wait until Slack inevitably throws us off, then.
>>>  
>>  
>> What I'm saying is, that the whole community isn't in a single place anyway. 
>> Slack happens to be most popular, right now, but we are spread across IRC, 
>> gitter / github, slack, mailing lists, discord, stack overflow, 

Clojure Discord server

[Luke Burton]

Hi all,

After reading about some of the hassles people had with Slack, I posted earlier 
today in /r/clojure A Discord for Clojurians 
. 

We've had about 50 people pass through the Discord so far. Please drop by and 
see if you like it. To reiterate what I mentioned in the reddit post – this is 
very much an experiment, and a fun one at that. 

I spent a portion of today hacking together a repl-bot that lives in #repl-bot. 
Everything said in that channel will be evaluated in a very non-judgemental 
manner by the repl-bot. Even, as we discovered, System/exit. All part of the 
fun.

We also found that Clojure has full syntax highlighting support in Discord. So 
far, threaded conversations are the major missing feature as compared to Slack. 

Hopefully this little experiment will help the community figure out future 
directions for reliable community collaboration, whatever they may be.

The URL for joining the Discord server: https://discord.gg/v9QMy9D 


Luke.


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Functional Pattern to Replace Temp Var

[Luke Burton]

You can also memoize “expensive-calculation” …

https://clojuredocs.org/clojure.core/memoize

> On May 13, 2017, at 1:26 PM, Kevin Kleinfelter  
> wrote:
> 
> How would one convert the following procedural logic into functional/Clojure?
> x = expensive-calculation
> return (f1(x), f2(x), f3(x))
> 
> I'm sure I could force it by using a var to save the intermediate value, but 
> that's still procedural.  It seems like the pattern of reusing an expensive 
> result in multiple function calls without recalculating it must have a 
> functional approach, but I'm not finding it.
> 
> What I really want to do is to return:
> {:key1 f1(f0(x)), :key2 f2(f0(x)), :key3 f3(f0(x))}
> 
> without paying the price of executing f0 multiple times.
> 
> Tnx.
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What to read after 3 dozen "introduction to transducers" blog posts

[Luke Burton]

> On May 6, 2017, at 10:56 AM, Matching Socks  wrote:
> 
> This one.  https://tech.grammarly.com/blog/building-etl-pipelines-with-clojure
> 
> "To be honest, this is a somewhat advanced usage of the transducers 
> machinery," says the Grammarly Engineering Blog, right after shoehorning a 
> BufferedReader into the mold with "reify IReduceInit".  I already felt I'd 
> got my money's worth from reading up to this half-way point.  But I was 
> astonished at what came next.  

Though it pains me to speak of it, I tell you now I have seen this "blog post", 
and seen too what came next. Astonishing indeed! I admire your simple 
description of these events, as it suggests either a man of great fortitude in 
the face of horror, or a man who was able to flee and forget, and I know that I 
am neither.

I remember the reified specimen – still alive, god have mercy – placed upon an 
altar to be offered up to `eduction`. Yes! Placed squarely on its unsightly 
variadic first argument. Why? For what purpose? We are told: "it's a recipe for 
the values to come." The blood drained from my face.

It was soon apparent what eldritch function would feast upon the values 
begotten by this unholy recipe. I saw the invocation of `transduce` and it 
brought back sweeter memories, of burritos and airport conveyor belts. I could 
see the `eduction`, presumably still wrapping the reified IReduceInit, perched 
atop a ziggurat of lesser transducers. I looked, and at the very bottom lay a 
blasphemous side-effectful invocation of map. I did not belong in this place.

Beyond astonished, I tell you, but still at this point I clung to sanity. I 
cried out: Enough! The values have suffered enough! They were good values, 
persisted to disk, why are they now flung into dimensions far beyond our own? 
Why have you turned your back on open(2)? Why have you forsaken the file 
descriptor?

Alas, my sanity could not stand this final blow. The Grammarly necromancer 
spoke the words: "Here Comes The Parallel Implementation." And lo, he did throw 
`core.async` into the mix. I saw `pipeline` swallow `transduce` whole, like a 
multi-armed multi-beaked creature from a Boschian hellscape. The whine of the 
CPU fans drowned out my last cry: aren't you supposed to use 
`pipeline-blocking`?

You may think me sane now in having told you all this, but I assure you, I am 
no longer. I should have run when I had the chance, but now I dream of educing 
the pipeline, reifying the transducer, and shouting the misbegotten outcomes 
into every channel that will >!! my message. It is too late for me, but perhaps 
not too late for you.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What sorcery is this? (`reduced`, `reduced?`)

[Luke Burton]

To add to Ivh's answer, the source code isn't too hard to follow here. Cursive 
makes this really easy because you can follow the source right into 
clojure.lang.RT, but you don't need to go that far to see how `reduced?` is 
used.

You can see that all implementations of `reduce` in clojure.core.protocols call 
`reduced?` to see if they should stop reducing. So, this is a convention that a 
reducing or transducing process needs to follow. If you ever need to write a 
function that takes a transducer, and which applies that transducer without 
simply calling `transduce` or friends, you'll also need to uphold that 
contract. More here .

I thought I had found a nice application for `reduced?` outside this context, 
but alas it was not to be. I wanted to detect whether a `reduce` had terminated 
due to a condition being met or because it had exhausted the input collection, 
assuming that `reduced?` was checking for some metadata glued onto the value.

(reduced? (reduce #(if (= %2 5) (reduced %2) (+ %1 %2)) (range 1 10)))
=> false

The reason is that `reduced` wraps the value in a Reduced, which implements 
IDeref, and one must dereference it to obtain the wrapped value. The reducing / 
transducing processes not only must check for `reduced?` but they are also 
obliged to deref any such value that returns true for that check.

(reduced 5)
=> #object[clojure.lang.Reduced 0x471e4b70 {:status :ready, :val 5}]
(deref *1)
=> 5

So you really will never make use of `reduced?` unless you're *inside* an 
implementation of a reducing / transducing process. As Ivh already stated, 
extremely narrow-use :)


> On Apr 25, 2017, at 12:34 PM, John Gabriele  > wrote:
> 
> Just recently stumbled upon `reduced` and `reduced?`. It seems rather 
> magical... how does the outer `reduce` (or `reductions`) know to stop? That 
> is, how does the function which is being called (`reduced`) affect the 
> function that's calling it (below, `reductions`)? :
> 
> ~~~clojure
> (defn main
>   []
>   (let [res (reductions (fn [accum x]
>   (if (< accum 100)
> (+ accum x)
> (reduced [x accum])))
> (range))]
> (prn res)
> (prn (reduced? res)) ;=> false
> (prn (reduced? (last res) ;=> false --- why isn't this `true`?
> ~~~
> 
> Also, what's the use of `reduced?`, and on what in the above snippet would I 
> call it to return true?
> 
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com 
> 
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com 
> 
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hexing the Technical Interview

[Luke Burton]

YES.

https://aphyr.com/posts/341-hexing-the-technical-interview 



-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] New Clojure story pages on clojure.org

[Luke Burton]

> On Apr 6, 2017, at 12:12 PM, Gregg Reynolds <d...@mobileink.com> wrote:
> 
> 
> 
> On Apr 6, 2017 2:10 PM, "Gregg Reynolds" <d...@mobileink.com 
> <mailto:d...@mobileink.com>> wrote:
> 
> 
> On Apr 6, 2017 2:08 PM, "Gregg Reynolds" <d...@mobileink.com 
> <mailto:d...@mobileink.com>> wrote:
> 
> 
> On Apr 6, 2017 1:55 PM, "Luke Burton" <luke_bur...@me.com 
> <mailto:luke_bur...@me.com>> wrote:
> 
> Where's Azerbaijan? Seriously, this was a fantastic story:
> 
> https://www.youtube.com/watch?v=-zM5fDN_AHY 
> <https://www.youtube.com/watch?v=-zM5fDN_AHY>
> 
> looks great. i'd like to believe it, but i'm an amercn in 2017 , i buleeve 
> nuthink. when i poke around on that link i get nuttin.  maybe it's my 
> android, but when i tap the story i get a completely different video.  where 
> is this wondrous azaerbaijani thing?
> 
> maybe the site is screwed up? when i click play i get derek slager's video.
> 
> or it could be operator error. ;)  

Huh, I triple checked the URL and it's definitely showing up as 
https://youtu.be/-zM5fDN_AHY <https://youtu.be/-zM5fDN_AHY>.

Here's a page from the conference which links the talk (and the link is the 
same there).

http://www.lambdadays.org/lambdadays2017/emin-hasanov 
<http://www.lambdadays.org/lambdadays2017/emin-hasanov>

Yeah I blame Android :)


-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] New Clojure story pages on clojure.org

[Luke Burton]

Where's Azerbaijan? Seriously, this was a fantastic story:

https://www.youtube.com/watch?v=-zM5fDN_AHY 


 

> On Apr 6, 2017, at 11:46 AM, Gregg Reynolds  > wrote:
> 
> 
> 
> On Apr 6, 2017 7:12 AM, "Jon Pither" > 
> wrote:
> We have also maintained a set of complimentary success stories here: 
> https://juxt.pro/clojure-in.html . The more 
> Clojure success stories the merrier!
> 
> love the i18n-ism!
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com 
> 
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com 
> 
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: I can not find any function that might give rise to this StackOverflow error

[Luke Burton]

I'm away from a Clojure REPL to poke at this but I think trapping this in a 
debugger would trivially reveal the source of recursion. Set a breakpoint at 
the start of the loop and step through. If using Cursive / IntelliJ set a 
breakpoint to detect StackOverflow and inspect the call stack. 


> On Apr 1, 2017, at 6:52 PM, piastkra...@gmail.com wrote:
> 
> 
> Crazy! I re-wrote the (loop) to use (reduce) instead and now everything 
> works: 
> 
> 
> (defn loop-over-scores
>   [set-of-scores]
>   "2017-03-08 -- called from start.clj"
>   (reduce
>;; 2017-04-01 -- we assume vector-with-path-score looks like this:
>;;  [[:positive :true 0.88 19 60 10 12 3 1 3 1 2 1] 1]
>;; so we do a minor bit of destructuring
>(fn [vector-of-maps-of-vector-key-and-score [vector-with-path score]]
>  (conj vector-of-maps-of-vector-key-and-score
>(dire/supervise #'map-of-vector-path-and-true-false-scores 
> vector-with-path)))
>[]
>set-of-scores))
> 
> 
> The StackOverflow error is gone. 
> 
> Sad to say, I'm left with a feeling as if I know nothing about Clojure. I 
> don't know why (loop) gave me a StackOverflow error whereas (reduce) works 
> just fine. This is black magic to me, which, sad to say, is not a comfortable 
> feeling. 
> 
> 
> 
> 
> 
> 
>> On Saturday, April 1, 2017 at 7:51:08 PM UTC-4, piast...@gmail.com wrote:
>> Well, I am out of ideas. Let's assume I'll re-write this some other way. 
>> What would be better than using (loop)? What would be less likely to cause 
>> StackOverflow, or at least reveal why I'm seeing it. 
>> 
>> 
>>> On Saturday, April 1, 2017 at 6:23:29 PM UTC-4, piast...@gmail.com wrote:
>>> 
>>> I have a function that will run repeatedly, so I use the at-at library to 
>>> call it:
>>> 
>>> https://github.com/overtone/at-at
>>> 
>>> I don't think this is the problem. 
>>> 
>>> Sad to say, the Error is catching a StackOverflow, which I'm having trouble 
>>> finding. I don't see a place where I call a function recursively, so I 
>>> don't see where any of this becomes stackoverflow. 
>>> 
>>> I'm using Dire to catch errors, but for now I'm just logging them. I don't 
>>> see anything fancy or clever that would give me a stackoverflow, and yet 
>>> this anonymous function called by at-at is definitely giving rise to 
>>> StackOverflow.
>>> 
>>> (defn- calculate--scores []
>>>   (let [my-pool (at/mk-pool)]
>>> (at/every 18
>>>   (fn []
>>> (append-to-file "/var/log/example/-scores.log" "Will 
>>> attempt to write scores")
>>> (future 
>>>   (try 
>>> (let [
>>>   map-of-all-sums   (api/get-accuracy-and-precision)
>>>   set-of-all-scores (dire/supervise 
>>> #'api/path-entries map-of-all-sums)
>>>   path-and--scores (dire/supervise 
>>> #'api/loop-over-scores set-of-all-scores)
>>>   ]
>>>   (append-to-file "/var/log/example/-scores.log" 
>>> "\n\n\n\n\n")
>>>   (append-to-file "/var/log/example/-scores.log" " 
>>> path-and--scores: ")
>>>   (append-to-file "/var/log/example/-scores.log"  (str 
>>> "(count set-of-all-scores): " (count set-of-all-scores)))
>>>   (append-to-file "/var/log/example/-scores.log" 
>>> path-and--scores))
>>> (catch Exception e
>>>   (do
>>> (append-to-file "/var/log/example/-scores.log" (str 
>>> " EXCEPTION:: " e))
>>> (append-to-file "/var/log/example/-scores.log" 
>>> (stack/parse-exception e))
>>> (errors/error e)))
>>> (catch Error o
>>>   (println (str " a problem in the anonymous function 
>>> in calculate--scores: " o)))
>>> )))
>>>   my-pool)))
>>> 
>>> 
>>> The problem appears to be here, which will loop several thousand times and 
>>> then die. But I don't see why. (This function used to just be a loop, and I 
>>> added the (let) so I could see if I could read the end -- but I could not 
>>> reach the end).
>>> 
>>> 
>>> 
>>> (defn loop-over-scores
>>>   [set-of-scores]
>>>   "2017-03-08 -- called from start.clj"
>>>   (try
>>> (let [return-value  (loop [
>>>how-many-loops 0
>>>scores set-of-scores
>>>vector-of-maps-of-vector-key-and-score []
>>>]
>>> 
>>>   (println  (str  " in loop-over-scores again " 
>>> how-many-loops))
>>>   (if (next scores)
>>> (recur
>>>  (inc how-many-loops)
>>>  (remove #{(first scores)} scores)
>>>  (conj vector-of-maps-of-vector-key-and-score 
>>> 

Re: [ANN and RFC] Bifurcan: impure functional data strucures

[Luke Burton]

I'm not well versed enough in these data structures to know this without asking 
(apologies if it's really obvious to some people): is there opportunity to 
improve Clojure's built-in data structures with Bifurcan rather than trying to 
wrap Bifurcan's structures in Clojure? 

As an aside, I want to draw people's attention to the sweet little criterium + 
gnuplot setup you have there for generating benchmarking plots. Nice!


> On Mar 27, 2017, at 10:13 AM, Zach Tellman  > wrote:
> 
> Benchmarks are available here, and the Clojure benchmarks make use of 
> transients wherever possible: 
> https://github.com/lacuna/bifurcan/blob/master/doc/benchmarks.md 
> .  
> 
> More generally, while transients are often used in practice to quickly 
> construct a read-only data structure, the more formal definition is that they 
> provide an O(1) mechanism for transforming between immutable and mutable 
> forms.  This isn't possible with purely mutable data structures like Java's 
> HashMap or Bifurcan's LinearMap.  So while wrapping these data structures in 
> the Clojure API would provide better performance for construction and 
> lookups, it wouldn't be quite the same thing as a transient.
> 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Clojure resume tips?

[Luke Burton]

> On Mar 23, 2017, at 11:59 AM, Mark Engelberg <mark.engelb...@gmail.com> wrote:
> 
> On Thu, Mar 23, 2017 at 11:24 AM, Luke Burton <luke_bur...@me.com 
> <mailto:luke_bur...@me.com>> wrote:
> 
> * So … if I was in your position, knowing what I know now, if I couldn't find 
> companies that had very progressive hiring practices, I would make my resume 
> stand out by leading in with an offer to spend a few hours writing a small 
> implementation of anything the hiring manager would like me to write. Many 
> hiring mangers are scared by take home projects because they're afraid of 
> what the best candidate will think. "It's an insult to experienced 
> candidates!" or "how would a rockstar candidate possibly spare the time?" But 
> secretly I think all hiring mangers *really* want to know what it will be 
> like to have you write code on their behalf. It's just not the industry norm 
> to ask.
> 
> Insightful post about a lot of things related to hiring, but I have to take 
> exception with this very last point.  Recently, a friend of mine sought out a 
> data science position in the Seattle area.  Each prospective employer gave 
> him a take-home assignment that required 30-40 work hours to complete.  Some 
> of the assignments were real problems the company was facing, so he was 
> effectively being asked to do free consulting work for each company.  This is 
> a horrible, burdensome interview practice and it would be dreadful if it 
> became the norm in the software industry. Suggesting that someone offer to do 
> a take-home project may make sense in specific cases for an inexperienced 
> candidate, but I fear it starts our industry down the slippery slope.

I definitely share the concern. 30-40 hours is totally not what I'd consider a 
take home project. I also think "real problems facing the company" has a host 
of issues associated with it, including the one you pointed out which is that 
it blurs the line between free labor and hiring pipeline. While I believe that 
a hiring pipeline this onerous is very unlikely to catch on and be the industry 
norm, I would put forward the proposal that a take-home project must *not* be 
work that directly contributes to the hiring company's value stream. 

I'd like to give you a sense of how I approached this problem, just to give a 
counterexample that is maybe more hopeful:

* We encouraged the candidate to have fun with the problem! The goal was not to 
stress them out under a deadline, but to give them enough time to relax into 
their normal engineering mindset. They could use any resource they liked, any 
framework or language they felt would suit the solution.

* They got to pick their own starting date, and deliver us a self-contained 
artifact of some kind a week later. Having to write "hand off" instructions was 
surprisingly telling! It was also surprising to see a candidate's git repo, to 
see how their solution evolved over time.

* The project was very, very general in specification. Almost uncomfortably 
open ended. I value candidates that can work with loose specifications, but I 
also wanted to let the candidate decide what they felt was "done" enough to 
share. And I wanted to ensure it wasn't possible to find canned solutions on 
the internet (Facebook found that their iOS candidates became increasingly 
adept at writing exactly the same tic-tac-toe interfaces, hmm …) 

* Their project solution would be the primary focus of any in-person interviews 
we followed up with. This is no small point, it really gave those interviews a 
sense of structure and purpose that I had not seen before. The candidate's 
memory for their code is fresh and they feel confident, while the interviewers 
all had plenty of good questions having checked out the candidate's repo and 
deep dived into it.

There was other minutiae specific to the role and company that isn't 
interesting here, but hopefully that gives some sense of how I tried to balance 
the various concerns. I think it parallelizes nicely too, in the future I would 
not hesitate to send a project like this out to *everyone* who submitted a 
resume, just to see what comes back. 

I guess I've spilled this many words I may as well share what the project was. 
Aside from lots of preamble explaining the above, what I sent out was:

> The city of San Francisco has an open data repository located at 
> data.sfgov.org <http://data.sfgov.org/>. The amount of available data is 
> pretty impressive. Your task is to create a composite metric for city 
> restaurants, based on at least two variables from data in the repository. It 
> need not take itself too seriously, if you want to rate a location based on 
> its seismic activity and wind speed, go for it. It could be qualitative or 
> quantitative. 
> 
> The end result should be a full s

Re: Clojure resume tips?

[Luke Burton]

> On Mar 23, 2017, at 10:10 AM, Jason Basanese  wrote:
> 
> Attached is a fairly bad resume that I am using. Any tips on how I might 
> change it to appeal to more places that are looking for functional developers?

Couple of general tips … based on a few decades of looking at these things :) 
Sorry about the length but I care a lot about hiring!

* Redact your home address and phone number when you circulate it on the open 
internet. I wouldn't even include that in a resume I sent to an organization, 
personally.

* Resumes are the worst. There's almost no correlation between "things the 
candidate had on his/her resume" and "that candidate is performing really well 
for us 2 years later", but we persist in using them because everything else is 
a lot more work. Just keep that in mind before you think there's something 
"wrong" with your resume.

* Resumes should be considered a marketing document. A classic new grad mistake 
is to think that what hiring mangers and recruiters are looking for is a 
complete transcript and accurate information. What they're actually looking for 
is purely subjective. I could tell you stories - resumes skipped over for the 
dumbest of reasons. After looking through hundreds, people start cutting 
corners or just plain discriminating. So, think of it as serving roughly the 
same purpose as a blurb on the back of a paperback novel.

* To play along with the resume game, I recommend customizing your resume to 
the greatest extent possible based on what you know about the job. That's "the 
job" singular, as in you need to do this for each application. It's easy to 
carpet bomb the same resume across many companies, but you simply won't stand 
out. You are on the right track here already - what would make it appealing to 
places looking for functional developers? Take it a step further and find those 
places, then write something that specifically targets them. Remember it's not 
just about keywords, you also have to come across as someone who might fit the 
culture.

* To avoid (or supplement) playing the resume game, build relationships. Keep 
all your projects up on GitHub, find and cultivate 
 things you want to stick at and 
improve, and start getting to know people. Meetups, fixing issues on other 
people's projects, it doesn't have to be super involved, it just has to be an 
involvement. The absolute best jobs you will ever land will come from people 
who know the quality of your work and recommend you from the inside. 

* I am a huge fan of Thomas Ptacek's perspective on hiring 
. Very much worth your 
time to read. If you can find a company that asks candidates to write some code 
- actual code like you'll be writing on the job, not a graph traversal puzzle - 
then jump on that chance. I started applying these techniques and we 
immediately experienced an inversion between who we thought looked strong on 
paper and who ended up being a great engineering fit for our team. This 
candidate we hired would have flunked the keyword filter had we used one, they 
had not used a single technology in common use for our team. Eye opening. 

* So … if I was in your position, knowing what I know now, if I couldn't find 
companies that had very progressive hiring practices, I would make my resume 
stand out by leading in with an offer to spend a few hours writing a small 
implementation of anything the hiring manager would like me to write. Many 
hiring mangers are scared by take home projects because they're afraid of what 
the best candidate will think. "It's an insult to experienced candidates!" or 
"how would a rockstar candidate possibly spare the time?" But secretly I think 
all hiring mangers *really* want to know what it will be like to have you write 
code on their behalf. It's just not the industry norm to ask.

Hope this helps! Good luck …

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: off-topic: stackof developer survey

[Luke Burton]

> On Mar 22, 2017, at 2:26 PM, Gregg Reynolds  wrote:
> 
> very interesting stuff, esp. the sociological bits:
> 
> http://stackoverflow.com/insights/survey/2017 
> 
> 
> sadly, clojure does not even rank in popularity.  but it's number 1 in pay 
> worldwide.  o sweet vengeance!

Some fun reading in there, Clojure features a couple of times. It would be fun 
to watch for spikes in traffic to Clojure related resources, because I'm sure 
that landing "most highly paid" will cause a few people to sit up and take 
notice.

This did get me thinking though. If the community *did* want to score highly on 
some of these metrics, what would those be? Or do none of them adequately 
capture what is valued by the Clojure community?

I think I'd claim that popularity is a terrible metric, even though it can be 
gratifying to be popular. The fact that lots of people do a particular thing 
doesn't mean that thing is inherently good, or worth striving for. Some very 
popular things are bad lifestyle choices, like smoking, a diet high in sugary 
foods, and writing JavaScript.

Conversely some very, very good things can die from even the perception of 
being unpopular. We often get people asking on the subreddit why they find so 
many "abandoned" libraries in Clojure. The fact a piece of software might have 
been written years ago, and still be perfectly usable, is such an anomaly in 
more "popular" languages that people assume we've all curled up and died. I 
recently had a project steered away from Clojure (suffice to say it was a very 
good fit, I thought) due to concerns around the availability of Clojure 
programmers in the long term. In Silicon Valley. Where you can throw a rock in 
the air and be certain it will hit a programmer on the way down.

Anyway, my personal metric for Clojure success would be: "for projects where 
Clojure is an appropriate technical fit, how often are you able to choose 
Clojure?" It's a selfish metric but the higher it goes, the happier I am ;)

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Startup time of standalone .jar executable

[Luke Burton]

There isn't enough information here to really recommend a fix. But here are 
some general thoughts:

* If you're wondering whether there's a magic --fast-mode flag to make this 
problem go away, no, there is not. 

* Can you make this a whole lot faster? Probably, yes, but it will involve a 
lot of work. It would involve profiling where the startup time is being spent, 
picking appropriate GC and runtime parameters that prefer startup time over 
other considerations, etc.

* Is this an architectural smell? Absolutely. Executing any external binary "a 
great many times" is always going to be inefficient compared to letting the 
binary stay resident in memory. If performance is a concern you need to 
understand what the constraints are and design something that specifically 
addresses that.

Luke.


> On Mar 21, 2017, at 8:24 AM, Michael Lindon  
> wrote:
> 
> I wrote a collaborator some clojure code which I distributed to them as a 
> standalone jar file which they are executing with 
> 
> java -jar mystandalone.jar
> 
> The problem is that this executable is called a great many times in their 
> application and every time the 
> executable is called there is a java/clojure startup cost. In fact the 
> startup cost heavily domiantes the
> run time, whereas executing the code in a repl is almost instantaneous.
> 
> How can I get around this? Would it be better to make a clojurescript 
> executable?
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en 
> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: ANN: I wrote a beginner datomic tutorial I wanted to share with y'all, constructive feedback welcome

[Luke Burton]

> On Mar 19, 2017, at 2:34 PM, Fenton Travers  > wrote:
> 
> https://www.reddit.com/r/Clojure/comments/5zu1oc/my_datomic_tutorial_feedback_sought/
>  
> 

My feedback: this is really awesome for several reasons!

1) I really like the direct, unadorned descriptions you’ve used. While I’m a 
huge fan of Clojure for the Brave and True (and its spiritual ancestor: why’s 
poignant guide to Ruby), and refer a lot of people on to those resources, for 
me personally I really need to be taken straight to the point. "You can think 
about the data that is stored in datomic as just a bunch of maps” - perfect, 
that makes it feel like we’re standing at a whiteboard together and you’re 
trying to move us along quickly.

2) I feel that there aren’t many Clojure resources that directly explain a 
feature in terms of a similar feature in other contexts or languages (with the 
exception of Java). So I really appreciate the contrasts you’re drawing with 
SQL, because it helps me as a practicing programmer to jump straight to 
understanding how things are different. It’s like giving me just the patch I 
need for my brain, not the entire repo. 

3) Similarly, I like how you’ve made this contrast especially meaningful by 
highlighting two things. First, how Datomic solves a real pain point for me. I 
used to have to do SQL joins, which are ugly and tricky to get right, but now 
that pain is *gone*. Secondly, Datomic goes further by letting me do things 
that are basically impossible in regular SQL, namely, getting heterogeneous 
result sets based on “column” across the entire database. Boom, you just sold 
me on Datomic in a way that not much else has.

Why do I think this is important? I believe we have a lot of resources that 
explain how Clojure solves problems experienced by Java developers, by 
highlighting contrasts like this, but not so much for other technologies. Most 
Ruby or Python programmers don’t feel the pain of concurrent programming and so 
spending a lot of time on immutable data structures doesn’t feel very “real” to 
them. This is a crucial hook for beginner tutorials targeted at working 
programmers, you need that hook that makes them say “Holy crap, that’s 
something new, and it totally solves that pain. I really need to learn this"

4) Putting the tutorial and the code together in the same repo is nice. It 
removes yet another barrier to entry, I know when I read “and all the source 
code is available elsewhere at http://…” I almost never bother to chase it 
down. Being able to say “and the source code is right under your nose” is 
always great.

5) I’m not sure if you thought explicitly about your pacing, but you did a good 
job here too. Take big concept, break it up into smaller concepts, then bring 
each concept into focus. Acknowledge to the reader that certain concepts will 
remain fuzzy until we revisit them, and don’t take too long before you do that. 
Having to revisit a topic isn’t just an awkward thing you have to work around, 
if it’s well written it’s actually another “hook” that will keep people reading 
because it’s enticing to learn more.

6) The style of “let’s try this and see what we get? Hmm not quite right, but 
we can now do this …” is reminiscent of how we interact at the REPL, but I 
don’t think you encourage the reader to fire up the REPL and follow along. Some 
people might not even realize this is a thing you can do. Dare I say it, but 
perhaps the Klipse plugin would be an exciting addition to a tutorial like 
this. One could imagine making the training more “real” by actually asking the 
reader to demonstrate their understanding by producing some data that “unlocks” 
the next part of the tutorial. Just a thought …

All said and done, it looks like you put a ton of effort into writing this, and 
it was very worthwhile. I have already forwarded it to a bunch of colleagues 
who were Datomic-curious and had positive feedback. I hope you keep writing!

Luke.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.