Re: [ANN] cpython bindings for clojure

[Jonah Benton]

Wow, that is incredibly cool. Eager to check it out.


On Wed, Jun 5, 2019, 12:41 PM Chris Nuernberger 
wrote:

> Good morning Clojurians,
>
> About 2 months ago during a talk about the tech.ml systems the point was
> made that if we could load the python C libraries then that would provide
> us with a lot more libraries to draw from.  In addition, I seem to remember
> one of the ML talks a concern about lack of libraries being the problem.
>
> So, in the spirit of building bridges and providing clojure with the best
> possible libraries for any particular problem on the planet, I took the
> last month and built out raw cpython (libpython3.X) bindings based on JNA.
>
> So, today after cleaning things up just a little bit I decided it was time
> to announce libpython-clj.  My hope is that there are enough clojuian
> pythonistas that we can take this library and make it an absolute joy to
> work with; the best python hosting environment that exists.
>
> https://github.com/cnuernber/libpython-clj
>
> Some features include:
>
> * Python dicts are represented in java as java.util.Map's.
> * stdout,stderr are overridden to write to *out* and *err*, respectively.
> * transparent access to the python interpreter from any thread.
> * zero-copy pathways for numpy->tech.datatype tensors and back.  So you
> can intermix heavy processing between clojure and python without pay a
> large conversion cost.
> * Decent performance.  This is built on the raw cpython shared library so
> our cost of executing or interoperating with python should not be any
> higher than anyone else's.
> * two way bridging.  Expose your JVM objects to python and vice versa.
>
> I have a simple keras demo where I use Keras from clojure to train a
> neural network included in the examples.  So that proves out keras and
> tensorflow access from clojure.
>
> I think this soundly addresses our ability to load cutting edge math and
> machine learning libraries.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/clojure/CADbpEJvOWoxZu0Y_LHi%2B36bEbcOfxCV5n0jd7mhvq8TO9MaD_w%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/clojure/CANoO%3DwR%3Dwau9Ci%2BbCxaO_vRJXbbPoE%2B8bg_tYsUTEU9Wz3LrSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

core.async buffered channel behavior

[Jonah Benton]

Hi folks,

It's been a while since I've used core.async. Documentation suggests that

(chan n)

where n is a number creates a fixed size channel buffer supporting n
elements.

The below clj repl session seems to indicate that I can put 4 items into a
3-sized buffer:

user=> (def c (async/chan 3))
#'user/c
user=> (async/>!! c :a)
true
user=> (async/>!! c :b)
true
user=> (async/>!! c :c)
true
user=> (async/>!! c :d)
;; correctly blocks, ctrl-c to break
user=> (async/ (async/ (async/ (async/http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Similar lisps and emacs reimplementations?

[Jonah Benton]

I definitely don't want to get into a pointless legal discussion! Obviously
everyone's situation- and legal interpretation thereof- is different, yadda
yadda.

Will just say in passing having gone through the language of the EPL and
other licenses with IP lawyers on a few separate occasions over the years,
I have not come across a strong manifestation of this particular
interpretation.

The risk your lawyers seem to be concerned about has more to do with the
EPL's Commercial Distribution language- section 4- which specifies that
Contributors/Distributors of EPL code have to indemnify other contributors.
This means that a company distributing an application under the EPL could
be liable for claims against components that they didn't create and just
packaged with their app- e.g. Clojure- except that the EPL also says

"The obligations in this section do not apply to any claims or Losses
relating to any actual or alleged intellectual property infringement."

The intent is for the commercial distribution clause to work with the
patent clause- which just means that if you initiate a suit over a
component you lose the right to continue to use that component- such that
under the EPL

* you are not incurring obligations for IP violations in code you
distribute but didn't write
* if you sue or countersue, you need to keep the suit to 1st party IP, or
else you lose the rights to 3rd party IP

This protocol has been sufficient for the folks I've worked with but
obviously everyone's lawyers get to decide how they feel about it.

Obviously Ellison or Myhrvold or some other patent troll could decide that
there was IP trespass in Clojure itself. In that case, anyone with deep
enough pockets found to be using it would be presented with a bill, license
be damned.

Anyway- good luck!


On Sat, Mar 19, 2016 at 11:51 AM, Sam Halliday 
wrote:

> It's got nothing to do with contributing to Clojure (the Grant of Right
> is standard in all modern free software licences). The problem is the
> patent retaliation clause, which I quote from Section 7 of the
> [EPL](http://www.eclipse.org/legal/epl-v10.html)
>
>"If Recipient institutes patent litigation against any entity
>(including a cross-claim or counterclaim in a lawsuit) alleging that
>the Program itself (excluding combinations of the Program with other
>software or hardware) infringes such Recipient's patent(s), then such
>Recipient's rights granted under Section 2(b) shall terminate as of
>the date such litigation is filed."
>
> In other words, if you ever have a legal dispute with anybady about a
> patent violation in clojure (which somebody else could have contributed
> without your permission), then you lose your right to use clojure. As
> in, turn off your production systems, now.
>
> This could be persued by anybody (corporate or individual, including the
> person who you are suing for implementing your patents) who has ever
> contributed to Clojure. Rich Hickey is in a privileged position where he
> can grant ad hoc / tailored licences to corporate customers, granting
> immunity to the patent retaliation clause.
>
> This may not be a concern for small companies or hobbyists, who are
> unlikely to find themselves in such a situation, but it is a major
> concern for corporate entities that are often finding themselves in huge
> IPR counterclaims with their peers. I can only guess that the current
> use of clojure in corporate environments was sneaked in "under the
> radar" and a modern legal audit would be quite a gruelling ordeal for
> everybody involved in choosing clojure as a platform.
>
> So, I can enjoy the language, but only as a hobby.
>
> Best regards,
> Sam
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> "A. Levy"  writes:
>
> > [ text/plain ]
> > Can you elaborate a little more on those nightmare scenarios? From my
> (amateur) reading of the EPL, it looks like the patent clauses apply to
> contributors to the program. In this case, Clojure. Does developing
> something in Clojure force you to release it under the EPL?
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Clojure" group.
> > To post to this group, send email to clojure@googlegroups.com
> > Note that posts from new members are 

Re: [Help] core.async and jetty

[Jonah Benton]

Hi Miguel- pipelining is essentially http keep alive. A very old jetty
thread

http://jetty.4.x6.nabble.com/HTTP-1-1-Request-Pipelining-handling-td18682.html

indicates that for simplicity jetty will execute subsequent requests on the
kept-alive socket serially.

Jonah


On Mon, Feb 22, 2016 at 3:33 PM, Miguel Ping  wrote:

> Hi guys,
>
> I'm trying to replicate an experiment on nodejs and http pipelining:
> http://blog.yld.io/2016/02/08/squeeze-the-juice-out-of-node/
>
> This is what I got right now:
> https://gist.github.com/mping/98bb8eb9faf3c51f9889 (using
> *com.ninjudd/ring-async*)
>
> Problem is I can't get pipelining to work as in nodejs; by doing two reqs
> they are sequential, ie, the dates that I'm printing have two secs.
> Basically I want that the server prints two "identical" accpt dates, and
> the  response should show that (now not working):
>
> $ tail -f reqs.txt | nc 127.0.0.1 
>
>
> HTTP/1.1 200 OK
> Date: Mon, 22 Feb 2016 20:29:10 GMT
> Transfer-Encoding: chunked
> Server: Jetty(7.6.8.v20121106)
>
> 48
> id: 1, uri: /a
> Mon Feb 22 20:29:10 WET 2016
> Mon Feb 22 20:29:12 WET 2016
> 0
>
> HTTP/1.1 200 OK
> Date: Mon, 22 Feb 2016 20:29:12 GMT
> Transfer-Encoding: chunked
> Server: Jetty(7.6.8.v20121106)
>
> 48
> id: 2, uri: /b
> Mon Feb 22 20:29:12 WET 2016
> Mon Feb 22 20:29:14 WET 2016
>
>
> I'm guessing that this is either my mistake with core.async or the adapter.
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Datascript return values

[Jonah Benton]

Hey Derek-

Your datascript query is actually returning a set- syntax is #{}- of array
elements, also called "tuples". You can treat the set as a sequence, use
doseq to visit each item, and destructuring on the items to assign
individual array/tuple members to symbols.

For instance, say that each array item has 2 elements, which semantically
you think of as name and value. You can do something like this:

(let [results (d/q q-list @conn)]
  (doseq [[name value] results]
(println (str "name: " name " has value: " value

Does that help?

Jonah


On Tue, Feb 16, 2016 at 11:38 AM, Derek Frost  wrote:

> I need some advice about return values please.  I'm using Datascript on
> the console with clojure and I'm doing this to get values from the d/b
> (println(d/q q-list @conn)) which gives me a hash-map of arrays.  I'm
> trying to put values in a text file to convert to pdf - this is a bit mad
> but it's just to see if I can move some stuff away from xslt.  Well I can
> get at individual values doing something like this (def
> -main[](println(first(d/q etc... but it seems really messy.  When I try
> to do this in a let block I get arity problems and I also tried passing
> this to a str then to the third argument of spit to create a text file.
> Ideally I'm trying to format the output with newlines which I know how to
> do but I can't figure out how to nest the d/b results into a variable.
>
> So my question is, is there a nicer way of putting my bulk return values
> into a variable?  If I could do that, I think I could figure the rest out.
> Many thanks.  I'm relatively new to clojure but I have some experience in
> scheme and lisp...
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Command Interpreter in Clojure?

[Jonah Benton]

Hey Nick, tools.cli is great for interpretation of command line arguments
to a program; to actually provide a user with your own
limited-functionality shell, take a look at wrapping JLine:

https://github.com/jline/jline2

along the lines of what Reply did:

https://github.com/trptcolin/reply/tree/master/src/clj/reply/reader

Good luck,

Jonah


On Tue, Feb 16, 2016 at 9:42 AM, Alan Thompson  wrote:

> Have you seen this:
>
> https://github.com/clojure/tools.cli
>
> https://github.com/clojure-cookbook/clojure-cookbook/blob/master/03_general-computing/3-07_parse-command-line-arguments.asciidoc
>
> Alan
>
> On Tue, Feb 16, 2016 at 5:06 AM, Nick Vargish 
> wrote:
>
>> My current project requires a command-line interpreter with limited
>> functionality... For various non-technical reasons we can't expose bash to
>> our users.
>>
>> Our plans call for writing a console shell that will accept some limited
>> command set, and Clojure would be the natural implementation language,
>> since much of our functionality has been implemented in a Clojure library.
>> We even have a command-line tool that exposes a lot of the functionality to
>> shell scripts, but the parsing/function routing is quite primitive...
>>
>> Any suggestions? Has anyone done work on a CLI implemented in Clojure?
>>
>> Thanks,
>>
>> Nick
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Core functions for searching lists of values

[Jonah Benton]

Hey cycl...@speakeasy.net,

Jay's excellent main point is really about the use of anonymous functions-
that in many cases it's an anti-pattern.

Having a name for every function means that those functions become testable
and that code consuming those functions stays readable.

Another point is that core has a lot of functionality and especially when
one is new to the functional mindset, it's easy to miss something that
makes your job easy.

There will be many ways of expressing a particular operation, and it's
worth trying a few different ways to get to the semantics that make most
sense.

Here's one suggestion- turn the anonymous function into a named function
and use partial to compose the rule you need:

(defn quad-piece?
  [row col piece]
  (and (= row (:quad-row piece))
 (= col (:quad-col piece

(defn get-quad-pieces
  [galaxy [row col]]
  (filter (partial quad-piece? row col) galaxy))

Then maybe it looks like get-quad-pieces isn't doing very much and maybe
there are some filter cases to capture instead:

(def zeros (partial quad-piece? 0 0))

(filter zeros test-galaxy)

Anyway, hope that's helpful.

Jonah



On Wed, Jan 27, 2016 at 8:19 PM,  wrote:

> (def test-galaxy [{:quad-col 0, :quad-row 0, :sec-row 4, :sec-col 4, :type
> :E}
>{:quad-col 0, :quad-row 0, :sec-row 4, :sec-col 3,
> :type :base}
>{:quad-col 3, :quad-row 5, :sec-row 7, :sec-col 5,
> :type :star}
>{:quad-col 1, :quad-row 3, :sec-row 0, :sec-col 5,
> :type :klingon}])
>
> (defn get-quad-pieces
>   [galaxy [quad-row quad-col]]
> (let [quad-pieces (filter (fn [elem] (and (= (:quad-row elem) quad-row)
>   (= (:quad-col elem)
> quad-col))) galaxy)]
> quad-pieces))
>
> (get-quad-pieces test-galaxy [0 0])
> ; ({:quad-col 0, :quad-row 0, :sec-row 4, :sec-col 4, :type :E} {:quad-col
> 0, :quad-row 0, :sec-row 4, :sec-col 3, :type :base})
>
>
> Jay Fields makes an excellent argument for using core functions, instead
> of anonyous functions, wherever possible
> http://blog.jayfields.com/2012/10/clojure-avoiding-anonymous-functions.html
> In his example, he was grouping elements, not filtering by multiple
> values, which is what I'm trying to do.
>
> I'm trying to refactor one particular piece of code, and have no idea
> what, if any, core functions, would work.
>
> I want to filter a list of hashes, based on some of the values - (code
> topside)
>
> I have come up with a few variations of the code, with reduce, etc - but
> they all use an anonyous function.  The only options I can figure, using
> filter or reduce, require a variable to represent each list element, which
> means an anon-function with a parameter.
>
> Thanks.
>
> Okay, I now have two questions - why does the code-highlighter insist on
> putting the code at the top, instead of in the middle, where I had the
> cursor? :)
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Debugging idea for a constrained situation

[Jonah Benton]

If you're in a place where you're hunting for major architectural patterns
to consider in what could amount to a rewrite, well, certainly there is no
free lunch. But the pattern you mention- capturing external event history-
is in the vein of Event Source systems

http://martinfowler.com/eaaDev/EventSourcing.html

Those sometimes go along with a Command pattern

https://en.wikipedia.org/wiki/Command_pattern

for applying changes to state.

The full context of your system isn't clear, but if the event source
abstraction applies, and events can be persisted in some way that you can
have access to, then it could help in recreating problematic conditions.
The one suggestion I'd make if possible is to capture the events in a
facility that is outside your program. If you do it inside your program, as
you were describing- whatever the potential benefit, now you're giving your
program yet another thing to do, the complexity (and bugs) in which is not
zero.

Along the same lines are the concepts of decomposition and decomplecting.
Prefer more simple things, loosely coupled, than a smaller number of more
complex tightly coupled things. If possible, rather than adding things to
one program, split the program into multiple smaller programs that have
precisely defined and limited interfaces. Some motivation in this vein are
Dan Bernstein's 7 fundamental rules:

https://cr.yp.to/qmail/guarantee.html

and paper about the security of qmail:

http://cr.yp.to/qmail/qmailsec-20071101.pdf

Hope that's helpful. Good luck.




On Thu, Jan 21, 2016 at 1:45 PM,  wrote:

> While not strictly Clojure-related, I thought I'd share this idea with you
> here because (1) I came up with it while thinking about design from a
> Clojure / functional point of view and (2) I respect your opinion. It's
> very likely you'll have better ideas...
>
> *I'm in an highly constrained situation*
>
>- When an Incident occurs (possible bug, bad behavior), I'm told days
>later and I never have access to the machine where the code ran
>- I only have the following
>   - a log file with size constraints (no more than n KB)
>   - the version of the code that was running
>   - I don't have the following
>   - core dumps (they'd be bigger than the n KB anyway, plus no one
>   knows a priori when to persist one for Incidents where the code fully
>   believed it was doing fine)
>   - complete info on how to re-create the environment (only partial
>   info)
>   - since the code can be running on any kind of machine with any
>  kind of configuration
>  - and since there are a lot of other applications of various
>  versions running as well
>  - besides, even if I had complete info, actually re-creating
>  such an environment would be very time consuming and error prone
>
> Figuring out what went wrong has been *painful*.
>
>
> But if I had access to all the values that a program *obtained/received*
> from its environment leading up to the Incident then I could just have my
> program use these values while running in a debugger.
>
>
> *The basic idea is  *
>
>1. Log *external* values used by the program over time in production.
>Don't worry about internal / local values since they are all derived
>functionally from these external values.
>2. When an Incident occurs, load this log and a final time-stamp into
>the program's "state map"
>3. Any time the program needs a value from the outside, it uses a
>value from the state map instead
>4. Set breakpoints and debug away (I'm stuck using C++ (sadness!))
>
> I like this because minimal time is spent re-creating the crime scene. I
> just have to tweak the program to start the task / thread in question after
> it's done loading the state. I won't have to ask QA "do you have a test
> environment where this problem is reproducible?" And I won't be making any
> mistakes in reproducing the Incident because all the values used will be
> loaded in an automated fashion.
>
>
> *Considerations*
>
>- Since values may be large, I may have to tweak the logging to enable
>re-using a value from earlier if it hasn't changed instead of logging it
>all over again. (current / expired / re-use)
>   - The program would be checking if the value has changed for those
>   values which have "expired" (that is, values expire if the task they're
>   related to has finished -- when that task starts up again the program 
> would
>   check the map for a value that it needs, find that it has expired, and 
> go
>   fetch the current one from the environment. Then it can decide how to 
> log
>   it in the state log.)
>   - I have to make sure every value I need would be within the most
>recent n KB of log. I may have a separate thread that logs a snapshot of
>the entire state every n KB.
>- I'm forced to change "every" external access into a conditional that
>

Re: Composing Stuart Sierra's components

[Jonah Benton]

Hey Colin, it sounds like:

* if the 2 systems really can't function without each other, and their
start/stop lifecycles are tightly bound, then somehow they have to be
merged into a single system

or

* if the 2 systems can't be merged into a single system because of true
functional or lifecycle independence and non-overlapping dependencies, then
semantics for what it means for each system to not have the other available
at lifecycle event times have to be established

It smells like the latter may be the way to go. So in pseudocode, the
reusable-component-system that owns the reusable-component would not
receive an instance of the external-collaborator at creation time. Instead,
the reusable-component would have a somewhat weaker definition of its start
semantics, and perhaps would have a way to refer to the
external-collaborator only by name.

That is, perhaps the external-collaborator provided to the
reusable-component is really an external-collaborator-client that is
configured to know the name of the external-collaborator- which is owned by
larger-system- and shields the reusable-component in cases where the larger
system has not been started or the collaborator isn't available.

And perhaps similarly the larger-system has a reusable-component-client or
reusable-client-proxy that is able to bridge appropriately from
larger-system to the reusable-component.

And maybe both client components are configured with the same :bus for
cross-system communication (and perhaps the cross-system bus is owned by a
bridge system). Just riffing, as it's not precisely clear what the
semantics of the systems are.

Stuart's comment that he hasn't run into a need for systems of systems is
coming to mind. Perhaps it makes sense to break these apart more explicitly
in accordance with guidelines around managing micro-services. Easy to say,
of course. :)

Not sure if that's helpful

Jonah



On Wed, Mar 11, 2015 at 3:01 PM, Colin Yates colin.ya...@gmail.com wrote:

 merge won't help as there will be name space clashes.

 I wonder if a more elegant approach would be to construct the 'inner'
 system and then assoc onto it the external dependencies it needs
 before calling start.

 On 11 March 2015 at 18:49,  adrian.med...@mail.yu.edu wrote:
  I believe I misunderstood your question; I didn't realize it was system
 (as
  opposed to any general component) specific. I think systems can be merged
  together (via 'merge'). Would that help?
 
  On Wednesday, March 11, 2015 at 2:40:14 PM UTC-4, Colin Yates wrote:
 
  Hi Adrian - I don't follow how that helps integrate two different
  systems - I wonder if my question was unclear or I am missing
  something in your answer. Would you mind posting some pseudo code to
  clarify please?
 
  On 11 March 2015 at 18:32,  adrian...@mail.yu.edu wrote:
   You can specify component dependencies using the 'using' function as
 you
   know. As long as you know the key of the component in the system you
 can
   specify this dependency wherever you construct the component. If you
   want to
   parameterize dependencies, write a constructor function which takes
 the
   external dependency as a value.
  
   On Wednesday, March 11, 2015 at 2:17:12 PM UTC-4, Colin Yates wrote:
  
   I have a non-trivial component which requires a bunch of internal and
   external collaborators to work. This component is itself re-usable.
  
   What I really want to do is have ReusableComponent be a component in
 a
   system so it can pull its external collaborators. However,
   ReusableComponent
   constructs its own services etc. so it really want to be (or at least
   have
   access to) a system.
  
   For example, let's say I have the following:
  
   (defrecord InternalToReusableComponent [bus]
 (component/Lifecycle
   (start [this]...))
  
   (defrecord ReusableComponent [bus logger]
 (component/Lifecycle
   (start [this]
  
 this)
   ))
  
   (defn reusable-component-system [external-collaborator]
 (component/system-map
   :bus ()
   :logger ()
   :reusable-component (component/using (map-ReusableComponent {})
   [:bus
   :logger external-collaborator]))
  
   Fine - I now have a system from which I can pull the reusable
   component.
   However, where does 'external-collaborator' come from? Obviously
 there
   is a
   larger system which I want this component to be part of so I can do:
  
   (defn larger-system []
 (component/system-map
:external-collaborator (...)
:reusable-component (component/using (some-magic-glue)
   [:external-collaborator])))
  
   I am struggling to see what (some-magic-glue) should be. I imagine it
   needs to be something like:
  
   (defrecord SystemAdaptor [external-collaborator internal-system]
 component/Lifecycle
 (start [this]
   (let [internal-system (or internal-system
   (reusable-component-system
   external-collaborator))
  internal-system (component/start 

Re: Idiomatic way to co-ordinate 'disconnected' services into a single transaction (ala Spring's @Transactional functionality)?

[Jonah Benton]

Hi Colin, no worries, responses below-

On Thu, Mar 5, 2015 at 2:31 PM, Colin Yates colin.ya...@gmail.com wrote:

 Hi Jonah,

 Coffee consumed, post read - thoughts emerging:

  - so the 'context' is essentially a service registry as well as
 run-time state?


More the latter- about transient, intermediate state rather than global
state like a registry.

A service registry could properly be a Component that depended on other
components, and then the registry could be delivered in the context map. I
usually include Components in the context map, though not the whole System
map. Flows are like functions, they should be called by Components, not
the other way around.



  - if 'renew-session' and 'return-renewed-token' both needed to
 execute in a single transaction that was separate to
 'some-thing-else', how would that be co-ordinated - would each tx
 boundary mandate a distinct flow for example


Right- generally since transaction boundaries are an edge semantic,
transaction scope should ideally be confined to a single top level edge
function, and data needed for the transaction should be prepared
beforehand- perhaps including generating primary key values separately. If
the transaction is complex and requires in-scope processing, perhaps it
should live inside the database. In theory an isolated flow could be used
*inside* transaction scope, but any complex logic occurring inside
transaction scope is a smell that perhaps transaction scope could be
rethought.


  - the whole thing feels very 'state-machiney' - is that the point?


I would say data-driven.


 Thanks again for sharing. I have to say that the 'flattening' of the
 control, or rather conditional logic seems quite expensive. My mind
 thinks in graphs not flattened lists, but maybe my resistance is just
 the pain of a new paradigm...

 At the moment I am slightly nervous as it seems to be conflating a
 whole bunch for the win of a consistent way of moving state around.


In a way- it's just another view of an execution chain.

We get the stack-based model for free, where functions call into
functions, into other functions, etc, then are popped back out when they're
done, and conditional logic is typically embedded inside functions. When
those functions need data, it's supposed to be available in the class, or
less often, arrive on the stack, or they have to reach out onto the heap-
into global space shared by all threads. Conditional decisions aren't first
class entities- an if doesn't give you a new stack frame. etc. There are
good things about that model for certain kinds of solutions. Clojure begins
to decouple it by, e.g. by not hiding data inside classes.

A data flow model has different priorities and different costs and
benefits, and though it is less native to the JVM one can imagine a data
flow compiler that makes it a little friendlier.



 Thanks for the time to respond and these questions are those of
 clarity not criticism :).


Of course!



 On 5 March 2015 at 16:45, Jonah Benton jo...@jonah.com wrote:
  Hi Colin- a simple example that uses a flow operator would be
 
  (- context
do-something
do-some-io
do-a-last-thing )
 
  with
 
  (defn do-something [context]
(let [some-data (get-in context [:path :to :data :element])
   result (pure-edge-function some-data)]
  (assoc-in context [:path :to :revised :data] result)))
 
  and
 
  (defn do-some-io [context]
(let [some-other-data (get-in context [:path :to :data :to :save])
   database (get-in context [:path :to :database :resource])]
  (save! database some-other-data)
   context)))
 
  Then save! can be a plain function, or
 
  (defprotocol Saver
(save! [this data]))
 
  (defrecord DatabaseSaver [conn ... ]
Saver
(save!  ))
 
  (defrecord AtomSaver [a ... ]
Saver
(save! ))
 
  And context is a map of maps that contains both application state and
 edge
  resources that should provide or receive data.
 
  Encoding conditional steps in a data flow presents a challenge. Several
  variants for doing this are in flow operators in Clojure and there are
 also
  a number of third party approaches. The below is a concrete example from
 an
  internal library that is not open source but doesn't differ significantly
  from approaches that are.
 
  A flow looks like this:
 
  (defflow renew
  { :startvalid-request?
   valid-request?{ true valid-token? false invalid-request }
   valid-token?  { true ingest-token false not-authorized }
   ingest-token  renew-session
   renew-session renewed?
   renewed?  { true return-renewed-token false
 not-authorized
  }
   return-renewed-token   :end
   not-authorized :end
   invalid-request:end })
 
  It's a regular map with at least a :start key and an :end value. Symbols
 in
  a flow map are plain functions that take a context map.
 
  Semantically, those functions represent either

Re: Idiomatic way to co-ordinate 'disconnected' services into a single transaction (ala Spring's @Transactional functionality)?

[Jonah Benton]

 and then executing the data flow, rather than
relying solely on the call stack.



On Thu, Mar 5, 2015 at 9:39 AM, Colin Yates colin.ya...@gmail.com wrote:

 Sounds interesting - are there are instances that I can look at?

 On 5 March 2015 at 14:15, Jonah Benton jo...@jonah.com wrote:
  Yes, exactly.
 
  That's fine, a node in the middle of the data flow can be responsible
 for
  reaching out to the edge to synchronously get some data to feed into a
  decision performed by the data flow. It would just do that through the
 edge
  functions.
 
  A request for data could also be asynchronously initiated once enough
  context had been determined by the data flow, and a promise added to the
  context map for later derefing (likely with a timeout).
 
  If the request for the data is context-free, it could be pulled in at the
  start of the data flow.
 
  By edge, it's is not that IO can't be done in the middle of a dataflow,
 it's
  just that functions that deal with dataflow follow one pattern (taking a
  context map) so they can be easily composed and rearranged, while
 functions
  that deal with edge resources take those resources along with other
  parameters more explicitly, so the data flow can run in an insulated
  fashion.
 
 
 
  On Thu, Mar 5, 2015 at 8:55 AM, Colin Yates colin.ya...@gmail.com
 wrote:
 
  Hi Jonah,
 
  This sounds very much like the model layer in DDD, which ironically is
  exactly what I am building.
 
  However, in the middle of a data flow a function needs to reach out to
  a repository to make a decision - how does that fit in with the data
  flow approach?
 
 
  On 5 March 2015 at 13:39, Jonah Benton jo...@jonah.com wrote:
   Hi Colin,
  
   Another option, other than HOFs or dynamic binding, is what might be
   called
   a data flow approach.
  
   At the edge of the application are the functions that explicitly take
   parameterized resources to perform edge state IO. These might be bare
   functions, or they might be protocol implementations that await the
   delivery
   of a type to perform their operation, or multimethods.
  
   At the center are functions that take a context map, which contains
 all
   relevant application state- IO resources and transient state data.
 Those
   functions are arranged in a data flow and capture the logic or state
   transitions of the application independent of any specific IO
   commitments.
   They can use schema or type annotation or pre/post conditions to
 enforce
   invariants.
  
   When data flow processing arrives at a place where some edge IO should
   occur, these data flow functions act as adapters to get-in the
   appropriate
   resource or Record from the context map and call the edge functions to
   perform IO.
  
   The result is a wider, flatter system, propagating state explicitly
   rather
   than implicitly through the call stack.
  
   Jonah
  
  
  
   On Wed, Mar 4, 2015 at 12:58 PM, Colin Yates colin.ya...@gmail.com
   wrote:
  
   Hi,
  
   I am looking for the Clojure equivalent of:
  
   class Whatever {
   @Transactional
   void doSomething(IDoSomething one, IDoSomethingElse two) {
 one.doSomething()
 two.doSomething()
   }
   }
  
   where both one and two are dependency injected with a proxy which
   resolves
   to a thread local database connection. In addition, one might itself
   have a
   collaborator which itself has a collaborator which needs a
 datasource.
  
   So far I have two protocols:
  
   (defprotocol IDoSomething
(do-something [this ...])
  
   (defprotocol IDoSomethingElse
(do-something [this ...])
  
   Each protocol may have a number of implementations, one of which is a
   JDBC
   implementation:
  
   (defrecord JdbcIDoSomething [db]
 (do-something [this ...] ...))
  
   The problem is that the calling code only gets provided an
 IDoSomething
   and an IDoSomethingElse and it wants to do something like:
  
   (let [one (-JdbcDoSomething db) two (-JdbcDoSomethingElse db)]
 (with-transaction [tx db]
   (do-something one)
   (do-something-else two)))
  
   The problem here is that the implementations of do-something and
   do-something-else won't have access to the local bound 'tx', they
 will
   have
   their own 'db'.
  
   I realise the general argument is to be explicit and pass a db as the
   first argument to the protocol but this isn't appropriate in this
 case
   as
   there are validly multiple implementations. I could abstract a
   'unit-of-work' and pass that as the first argument to the protocols
 but
   that
   seems a bit painful.
  
   Also, these protocols may be used quite far away from where the
   database
   code lives and passing a parameter all the way through the call stack
   is
   painful.
  
   I am using Stuart Sierra's components if that makes any difference.
  
   I can't be the first person to run into this but google is
 surprisingly
   unhelpful which makes me think I have missed something fundamental

Re: Idiomatic way to co-ordinate 'disconnected' services into a single transaction (ala Spring's @Transactional functionality)?

[Jonah Benton]

Yes, exactly.

That's fine, a node in the middle of the data flow can be responsible for
reaching out to the edge to synchronously get some data to feed into a
decision performed by the data flow. It would just do that through the edge
functions.

A request for data could also be asynchronously initiated once enough
context had been determined by the data flow, and a promise added to the
context map for later derefing (likely with a timeout).

If the request for the data is context-free, it could be pulled in at the
start of the data flow.

By edge, it's is not that IO can't be done in the middle of a dataflow,
it's just that functions that deal with dataflow follow one pattern (taking
a context map) so they can be easily composed and rearranged, while
functions that deal with edge resources take those resources along with
other parameters more explicitly, so the data flow can run in an insulated
fashion.



On Thu, Mar 5, 2015 at 8:55 AM, Colin Yates colin.ya...@gmail.com wrote:

 Hi Jonah,

 This sounds very much like the model layer in DDD, which ironically is
 exactly what I am building.

 However, in the middle of a data flow a function needs to reach out to
 a repository to make a decision - how does that fit in with the data
 flow approach?


 On 5 March 2015 at 13:39, Jonah Benton jo...@jonah.com wrote:
  Hi Colin,
 
  Another option, other than HOFs or dynamic binding, is what might be
 called
  a data flow approach.
 
  At the edge of the application are the functions that explicitly take
  parameterized resources to perform edge state IO. These might be bare
  functions, or they might be protocol implementations that await the
 delivery
  of a type to perform their operation, or multimethods.
 
  At the center are functions that take a context map, which contains all
  relevant application state- IO resources and transient state data. Those
  functions are arranged in a data flow and capture the logic or state
  transitions of the application independent of any specific IO
 commitments.
  They can use schema or type annotation or pre/post conditions to enforce
  invariants.
 
  When data flow processing arrives at a place where some edge IO should
  occur, these data flow functions act as adapters to get-in the
 appropriate
  resource or Record from the context map and call the edge functions to
  perform IO.
 
  The result is a wider, flatter system, propagating state explicitly
 rather
  than implicitly through the call stack.
 
  Jonah
 
 
 
  On Wed, Mar 4, 2015 at 12:58 PM, Colin Yates colin.ya...@gmail.com
 wrote:
 
  Hi,
 
  I am looking for the Clojure equivalent of:
 
  class Whatever {
  @Transactional
  void doSomething(IDoSomething one, IDoSomethingElse two) {
one.doSomething()
two.doSomething()
  }
  }
 
  where both one and two are dependency injected with a proxy which
 resolves
  to a thread local database connection. In addition, one might itself
 have a
  collaborator which itself has a collaborator which needs a datasource.
 
  So far I have two protocols:
 
  (defprotocol IDoSomething
   (do-something [this ...])
 
  (defprotocol IDoSomethingElse
   (do-something [this ...])
 
  Each protocol may have a number of implementations, one of which is a
 JDBC
  implementation:
 
  (defrecord JdbcIDoSomething [db]
(do-something [this ...] ...))
 
  The problem is that the calling code only gets provided an IDoSomething
  and an IDoSomethingElse and it wants to do something like:
 
  (let [one (-JdbcDoSomething db) two (-JdbcDoSomethingElse db)]
(with-transaction [tx db]
  (do-something one)
  (do-something-else two)))
 
  The problem here is that the implementations of do-something and
  do-something-else won't have access to the local bound 'tx', they will
 have
  their own 'db'.
 
  I realise the general argument is to be explicit and pass a db as the
  first argument to the protocol but this isn't appropriate in this case
 as
  there are validly multiple implementations. I could abstract a
  'unit-of-work' and pass that as the first argument to the protocols but
 that
  seems a bit painful.
 
  Also, these protocols may be used quite far away from where the database
  code lives and passing a parameter all the way through the call stack is
  painful.
 
  I am using Stuart Sierra's components if that makes any difference.
 
  I can't be the first person to run into this but google is surprisingly
  unhelpful which makes me think I have missed something fundamental, and
 that
  I have something upside down.
 
  What do you all do?
 
 
  --
  You received this message because you are subscribed to the Google
  Groups Clojure group.
  To post to this group, send email to clojure@googlegroups.com
  Note that posts from new members are moderated - please be patient with
  your first post.
  To unsubscribe from this group, send email to
  clojure+unsubscr...@googlegroups.com
  For more options, visit this group at
  http://groups.google.com

Re: Idiomatic way to co-ordinate 'disconnected' services into a single transaction (ala Spring's @Transactional functionality)?

[Jonah Benton]

Hi Colin,

Another option, other than HOFs or dynamic binding, is what might be called
a data flow approach.

At the edge of the application are the functions that explicitly take
parameterized resources to perform edge state IO. These might be bare
functions, or they might be protocol implementations that await the
delivery of a type to perform their operation, or multimethods.

At the center are functions that take a context map, which contains all
relevant application state- IO resources and transient state data. Those
functions are arranged in a data flow and capture the logic or state
transitions of the application independent of any specific IO commitments.
They can use schema or type annotation or pre/post conditions to enforce
invariants.

When data flow processing arrives at a place where some edge IO should
occur, these data flow functions act as adapters to get-in the appropriate
resource or Record from the context map and call the edge functions to
perform IO.

The result is a wider, flatter system, propagating state explicitly rather
than implicitly through the call stack.

Jonah



On Wed, Mar 4, 2015 at 12:58 PM, Colin Yates colin.ya...@gmail.com wrote:

 Hi,

 I am looking for the Clojure equivalent of:

 class Whatever {
 @Transactional
 void doSomething(IDoSomething one, IDoSomethingElse two) {
   one.doSomething()
   two.doSomething()
 }
 }

 where both one and two are dependency injected with a proxy which resolves
 to a thread local database connection. In addition, one might itself have a
 collaborator which itself has a collaborator which needs a datasource.

 So far I have two protocols:

 (defprotocol IDoSomething
  (do-something [this ...])

 (defprotocol IDoSomethingElse
  (do-something [this ...])

 Each protocol may have a number of implementations, one of which is a JDBC
 implementation:

 (defrecord JdbcIDoSomething [db]
   (do-something [this ...] ...))

 The problem is that the calling code only gets provided an IDoSomething
 and an IDoSomethingElse and it wants to do something like:

 (let [one (-JdbcDoSomething db) two (-JdbcDoSomethingElse db)]
   (with-transaction [tx db]
 (do-something one)
 (do-something-else two)))

 The problem here is that the implementations of do-something and
 do-something-else won't have access to the local bound 'tx', they will have
 their own 'db'.

 I realise the general argument is to be explicit and pass a db as the
 first argument to the protocol but this isn't appropriate in this case as
 there are validly multiple implementations. I could abstract a
 'unit-of-work' and pass that as the first argument to the protocols but
 that seems a bit painful.

 Also, these protocols may be used quite far away from where the database
 code lives and passing a parameter all the way through the call stack is
 painful.

 I am using Stuart Sierra's components if that makes any difference.

 I can't be the first person to run into this but google is surprisingly
 unhelpful which makes me think I have missed something fundamental, and
 that I have something upside down.

 What do you all do?


  --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: idiomatic use of Stuart Sierra's component

[Jonah Benton]

Hi Colin, for me this is usually an Invented Here/Not Invented Here
question.

When I'm inventing a thing with state and a lifecycle I'll define it as a
Record and slap the Lifecycle protocol implementation onto it. I'd do that
with your Registry.

When I'm using a thing someone else made, usually that thing will have its
own state and implementation encapsulation and its own lifecycle, so I'll
make a simple Component only to manage it. I won't have that Component play
any other privileged role in regards to accessing the NIH thing or getting
to its internal state. Database connection is a canonical example- as often
discussed functions that do something with a database should take a
connection parameter, not a Component parameter from which they have to
extract a connection.

That all said, sometimes I'll have multiple implementations of Invented
Here things that the app needs just one of at runtime- say, Registry
implementations backed by different underlying technology that are
appropriate for different application use cases. In that case it starts to
smell like an NIH thing and I'll see if I can treat it that way and make a
single Component that can be told which Registry to manage. Sometimes that
works, but sometimes it makes sense to extend the protocol to the
individual implementations.

Jonah



On Mon, Mar 2, 2015 at 6:32 PM, Dylan Butman dbut...@gmail.com wrote:

 I would do the latter.

 I like the extend types to component/Lifecycle wherever possible. The goal
 is to have all components in your system only interact via protocols. This
 way, you have established interfaces between components, and if you want to
 swap an implementation, you just satisfy the protocol. So adapting your
 example

 (defprotocol IRegistry
   (register-with [registry cb spec]))

 I also like to provide all reference as arguments instead of assoc'ing
 them inside component/start. This allows them to be more easily used in
 other parts of the system, and gives you the opportunity to validate the
 references, continuing the idea of protocol interface

 (defn validate-system
   validate the portion of the system contained within the component
   [schema cmp]
   (s/validate {s/Keyword s/Any} schema)
   (- (select-keys cmp (keys schema))
(s/validate schema)))

 (defrecord Registry [state]
   Lifecycle
   (start [this]
 (validate-system {:state clojure.lang.Atom} this) ;; this could be
 more general, let's be specific though
 ;; start the registry
 this)
   (stop [this]
 ;; stop the registry
 this)
   IRegistry
   (register-with [registry cb spec]
 (swap! state :assoc cb spec)))

 (defrecrod UsesRegistry [registry]
   Lifecycle
   (start [this]
  (validate-system {:registry (s/pred (partial satisfies?
 IRegistry))} this)

  this)
   (stop [this]
 this))


 On Sunday, March 1, 2015 at 7:50:58 AM UTC-5, Colin Yates wrote:

 If I have a stateful thing with a lifecycle then is the system component
 the instance of the thing or a wrapper that contains the thing.

 For example, let's say I have a registry of clients that want to be
 polled then I might have the following:

 (defrecord Registry [state])
 (defn register-with [registry cb spec] (swap! (:state registry) :assoc cb
 spec))
 (defn start [registry] ...)
 (defn stop [registry] ...)
 (den some-other-thing [..] )

 Great.

 Now I want to expose that and use Stuart's excellent component library
 (which I really wish a global 'everything has been defined, now the actual
 system is starting' lifecycle event, but anyway). I seem to find most
 examples do:

 (defrecord RegistryComponent []
   component/Lifecycle
   (start [this]
 (let [registry (-Registry (atom {}))]
   (start registry)
   (assoc this :registry registry))
   (stop [this]
  (stop (:registry this))
  (assoc this :registry nil)))

 Other components get the RegistryComponent but then have to unravel the
 actual :registry.

 Another approach would be to adjust the actual Registry and that that
 implement the lifecycle:

 (defn register-with [registry cb spec] (swap! (:state registry) :assoc cb
 spec))
 (defn start [registry] ...)
 (defn stop [registry] ...)
 (den some-other-thing [..] )
 (defrecord Registry [state]
   component/Lifecycle
   (start [this] (start this) this)
   (stop [this] (stop this) this))

 On the one hand, the Lifecycle component can be viewed as separate from
 the thing it is managing, and this wrapping and unwrapping is only a
 concern of Components. Nothing outside of a Lifecycle instance worries
 about it and nothing ever receives a Lifecycle instance.

 On the other hand, it all felt a little bit OO-ish - and actually I don't
 quite see the 'vanilla' Registry implementing the Lifecycle component as
 complecting anything inappropriate. The Lifecycle instance is only
 describing behaviour - there is no separate 'thing'.

 Is that correct or have I missed something? What do you all do?

  --
 You received this 

Re: Is Clojure a language for growth?

[Jonah Benton]

To add a data point to this, while the technology is great, it is not
necessarily right for all companies at all lifecycle stages.

My experience has been that C++ skills and interests don't necessarily
translate directly to Clojure. The kinds of microdecisions one makes in
modeling, algorithm design and so forth are quite different, and the costs
one optimizes for in C++ land don't make sense in Clojure. What I have seen
more often is motivated C++ers transition to Scala- getting up to speed on
the JVM and the APIs, get a little bit of a REPL, while retaining types and
compilation and a lot of the mental model- and then go from Scala to
Clojure. But since Clojure is a different toolchain and level of
abstraction, changing both at the same time even for the motivated can be a
bridge too far.

I would also not look at Clojure as a commodity tool, in the enterprise IT
staffing model where one wants to just be able to throw bodies at problems,
shift them around between teams, scale them up and down quickly with
fluidity between staff and consultant roles. It's currently significantly
more specialized and elite than that.

On the flip side, because it is a different level of abstraction, the right
team + Clojure can be incredibly productive in a wide variety of domains.

If you have a budget and can afford to frame a Proof-of-Concept in your
domain for a consulting team to demonstrate this, I would highly recommend
getting in touch with Cognitect themselves. I don't think it is widely
known that aside from being the creators of Clojure and Datomic, they have
an extremely proficient consulting practice. Watching them work can be
mind-blowing, from both technical and management perspectives.




On Wed, Aug 20, 2014 at 11:06 AM, Nando Breiter na...@aria-media.com
wrote:

 Perhaps the question is more Is your boss (or company) suitable for
 Clojure?

 On Wed, Aug 20, 2014 at 2:56 PM, Joshua Ballanco jball...@gmail.com
 wrote:

 My advice on convincing your boss to use Clojure for a new project: don’t.

 Projects succeed or fail for any number of different reasons, but I can
 guarantee you that if you *start* a new project with Clojure, and it does
 happen to fail, then the choice of Clojure will bear the brunt of the blame
 whether it deserves it or not.


  --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Compilation failure to reject arguments matching formal parameters

[Jonah Benton]

Hi Dave,

Somewhat related: I've been looking at Prismatic's Schema [1] along with
ordinary pre and post conditions to apply stricter runtime controls on
suites of functions that take and return maps:

user= (require '[schema.core :as sc])
nil

user= (def FooIn FooIn must have a string at :bar, a number at :baz, and
no other keys { :bar sc/Str :baz sc/Num })
#'user/FooIn

user= (def FooOut  FooOut must have :baz and :qux, but can have any other
keys/values { :baz sc/Num :qux sc/Str sc/Any sc/Any })
#'user/FooOut

user= (defn foo
  #_=   Input map must match FooIn, returns FooOut
  #_=   [ {:keys [bar baz] :as m}]
  #_=   {:pre [(sc/validate FooIn m)]
  #_=:post [(sc/validate FooOut %)]}
  #_=   (assoc m :qux zoop))
#'user/foo

user= (doc foo)
-
user/foo
([{:keys [bar baz], :as m}])
  Input map must match FooIn, returns FooOut
nil

user= (doc FooIn)
-
user/FooIn
  FooIn must have a string at :bar, a number at :baz, and no other keys
nil

user= (doc FooOut)
-
user/FooOut
  FooOut must have :baz and :qux, but can have any other keys/values
nil

user= (foo { :bar 1 :baz 2 })
{:qux zoop, :baz 2, :bar 1}

user= (foo { :bar 1 :baz 2 })
ExceptionInfo Value does not match schema: {:bar (not (instance?
java.lang.String 1))}  schema.core/validate (core.clj:165)

user= (foo { :bar 1 :baz 2 :beez 3 })
ExceptionInfo Value does not match schema: {:beez disallowed-key}
schema.core/validate (core.clj:165)

It's not compile time, requires multiple declarations, and imposes some
runtime cost, but in a repl workflow can provide guidance and immediate
feedback.

Prismatic has gone significantly further in the keyword functions space
with Plumbing and Graph. [2]

HTH,

Jonah

1. https://github.com/Prismatic/schema
2. https://github.com/Prismatic/plumbing



On Fri, Aug 15, 2014 at 7:47 AM, Dave Tenny dave.te...@gmail.com wrote:

 This is getting old.   Any suggestions or plans to fix it?  Do I need to
 use some other common declaration style for functions?

 ; CIDER 0.5.0 (Clojure 1.6.0, nREPL 0.2.3)
 user (defn foo [ {:keys [bar]}] bar)
 #'user/foo
 user (foo :baz 1)
 nil

 In my opinion the compilation of the call to foo should have complained
 about :baz not matching a known parameter.

 Failure to do this leads to lots of extra debugging because someone had a
 typo in a keyword name,
 e.g. :Name instead of :name or :product instead of :project, or
 whatever.

 This is probably the leading cause of unexpected program operation in
 clojure for me.


  --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: timbre logging, java libs

[Jonah Benton]

Sean Corfield has a great example of writing a log4j logging backend in
clojure:

http://corfield.org/blog/post.cfm/real-world-clojure-logging





On Tue, Jul 22, 2014 at 6:17 PM, Brian Craft craft.br...@gmail.com wrote:

 Is there any good way to use timbre in a project with java libs, e.g.
 c3p0, that use java logging APIs? Java logging is such a hopeless muddle I
 don't pretend to understand it, but my impression is that libs like c3p0
 use assorted backends (log4j, etc.), which in turn use assorted appenders
 to write to files, dbs, etc. clojure.tools.logging uses the same backends
 (log4j, etc.), to the same purpose. Timbre, however, just implements
 appenders, so it can't forward to log4j, and log4j can't forward to timbre.
 Timbre can operate as a backend for clojure.tools.logging, but that only
 helps with clojure libs.

 Is any of this correct?

 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: 56 second GC on Heroku - seems a bit long

[Jonah Benton]

Some good tactics at bottom:

https://blogs.oracle.com/poonam/entry/troubleshooting_long_gc_pauses




On Tue, Jun 17, 2014 at 9:38 AM, Brian Marick mar...@exampler.com wrote:


 On Jun 16, 2014, at 2:57 PM, Daniel Compton d...@danielcompton.net
 wrote:

  You didn't mention whether this was on the free tier or a paid dyno. I
 would expect the paid dynos to not be as heavily provisioned and could
 possibly alleviate this.

 Paid. (And using 2X dynos didn't seem to make a difference.)

 
 Latest book: /Functional Programming for the Object-Oriented Programmer/
 https://leanpub.com/fp-oo

 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: oob schemas, re: The Language of the System

[Jonah Benton]

I read these self-describing, extensible points in the context of EDN,
which has a syntax/wire format for some types- maps, strings, etc- and also
has an extensibility syntax:

#myapp/Person {:first Fred :last Mertz}

These tagged elements are extensions because they allow values of types
not known to EDN to be included in the stream, and are self-describing in
two senses:

* if a wire format reader does know how to create a myapp/Person{}, that
blob of data contains all the information needed to do so
* if a wire format reader doesn't known how to create a myapp/Person, it
can still read past this particular element in the stream, because tags
have a defined envelope, so a reader can figure out where data comprising
this element ends

The JSON example is mostly about the extensibility attribute. JSON's
format natively supports some types (like strings) but not others (like
dates), and for those others, JSON's format does not include a way to
bucket or envelope data comprising those unknown types. So JSON is not
extensible.

The google example is mostly about the self-describing attribute, and to
my mind is more accurately framed as a statement about the Internet as a
whole. Hypothetically, if all data exchange occurred using data formats
whose details were private arrangements between writers and readers- for
instance, all servers only spoke ProtocolBuffers and used a different
schema for each client- there would be no Internet at all, much less a
google who as a third party is able to broadly read and understand data
made available by servers. (Or, to your point, any ability to parse
anything useful from a server data stream by clients lacking knowledge of
the schema would be at best be inferential and heuristic- possible, but
infeasible on a large scale.)

With all that said- my read is that Rich bundled those two points together
in the JSON date example- JSON doesn't have an extensibility syntax to
support dates, but people still have to transmit dates over JSON, so how do
they do that? One way is by adopting a  convention, which in some ways is
better than an out of band schema, because, as you say, a convention gives
a reader additional information to heuristically interpret the stream, but
in other ways is worse because it isn't consistent- some people will want
date fields to look like dateModified, others will want modifiedDate,
and others use modificationDatetime.

So in a broad sense, it is not desirable to use a data format that does not
include an extensibility capability which itself is self-describing,
because a format that lacks extensibility creates a combinatorial explosion
in conventions to convey values not known to the format, and extensions
that are not self-describing require out of band agreements between readers
and writers that can preclude the scalable third-party interoperability
that is so important to the Internet.

Hope that helps.


On Sat, Jan 18, 2014 at 6:08 PM, Brian Craft craft.br...@gmail.com wrote:

 Ok, so consider a different system (besides google) that handles the JSON
 example. If it has no prior knowledge of the date field, of what use is it
 to know that it's a date? What is a situation where a system reading the
 JSON needs to know a field is a date, but has no idea what the field is for?


 On Saturday, January 18, 2014 1:27:31 PM UTC-8, Jonas wrote:

 IIRC in that particular part of the talk he was specifically talking
 about (non-self describing) protocol buffers and not JSON.

 On Saturday, January 18, 2014 10:00:09 PM UTC+2, Brian Craft wrote:

 Regarding Rich's talk (http://www.youtube.com/watch?v=ROor6_NGIWU), can
 anyone explain the points he's trying to make about self-describing and
 extensible data formats, with the JSON and google examples?

 He argues that google couldn't exist if the web depended on out-of-band
 schemas. He gives as an example of such a schema a JSON encoding where an
 out-of-band agreement is made that field names with substring date refer
 to string-encoded dates.

 However, this is exactly the sort of thing google does. It finds dates,
 and other data types, heuristically, and not through the formats of the web
 being self-describing or extensible.


 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
-- 
You received this message because you are subscribed to the Google

Re: OOP question re: The Language of the System

[Jonah Benton]

I'm curious for the response re: #3 as well, but one pain point I've
experienced with protocols vs multimethods is that as a system evolves,
retaining the semantics that led me to bundle functions together into a
protocol in the first place has sometimes been difficult, a source of
incidental complexity. It winds up being easier and more coherent to
distribute multimethods between namespaces- even though initially going all
multimethods at the start feels less clean.



On Sun, Jan 19, 2014 at 7:37 PM, Sam Ritchie sritchi...@gmail.com wrote:

 can you elaborate on #3? What constraints did you run into with type-based
 dispatch that a multimethod that dispatched on :op allowed you to get
 around? It seems like, if that's your pattern, lifting the :op field up
 into an actual record type would only give you MORE freedom. You rely on
 the convention of a type, and the typeclass-like protocol dispatch, rather
 than re-inventing the identifier with an :op field.

 Just curious as to what problems that records have caused (especially
 since you could write multimethods that dispatched on type and change that
 later if you actually do need the flexibility).

   Timothy Baldridge tbaldri...@gmail.com
  January 19, 2014 5:10 PM
 A couple of things on this subject:

 1) protocols were invented primarily for their speed. If one was going to
 write Clojure in Clojure it would be a little hard to implement
 PersistentHashMaps using pure data. You have to start with something. Thus
 deftype and defprotocol were born.

 2) Unlike in most OOP, the protocol methods are not tied to the object.
 Instead they are tied to the function. You can see that with the function
 clojure.core/extend. Extend says when you call function A with a type T as
 the first argument, dispatch to function B. This is quite a bit different
 from normal polymorphism (especially in languages like Python/Ruby, etc).

 3) Thirdly I'd say that using protocols/types for data is often considered
 a bit un-idiomatic. For example, see the ClojureScript compiler:
 https://github.com/clojure/clojurescript/blob/master/src/clj/cljs/compiler.clj#L377

 The AST of ClojureScript is stored in nested hashmaps, and dispatching is
 done on the :op value in each map. No records, no protocols, no types. Just
 hashmaps and multimethods. I would argue that this is the Clojure way.
 Use data, except when performance requirements dictate otherwise. Almost
 everytime I've built a large codebase using protocols instead of
 multimethods, I've regretted it. Data all the things...as the saying
 goes.

 Timothy Baldridge





 --
 “One of the main causes of the fall of the Roman Empire was that–lacking
 zero–they had no way to indicate successful termination of their C
 programs.”
 (Robert Firth)
 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.
   James Reeves ja...@booleanknot.com
  January 19, 2014 2:55 PM
 On 19 January 2014 19:55, Brian Craft craft.br...@gmail.com wrote:

 http://www.youtube.com/watch?v=ROor6_NGIWU

 Around 56:28 Rich is talking about whether components pass a data
 structure, or an object that has all these verbs and knows how to do stuff
 

 Clojure data types also have verbs, in protocols, and if one component
 passes a record to another component, the receiver will need to use those
 verbs to make use of the data. How is that different than the OOP example?
 There's the syntactic difference that in OOP you write it object, verb,
 arguments, and in clojure you write it verb, object, arguments. But that's
 trivial. How is it architecturally different to pass an object with methods
 vs passing a record with protocol?


 Protocols should be considered a tool for polymorphism, rather than a tool
 for encapsulation. You shouldn't consider protocols to be tightly coupled
 to a record or type.

 To give a more concrete example, consider a edn data structure like:

 #inst 2014-01-19T21:44:46.471-00:00

 By default Clojure reads this as a java.util.Date object, but we could
 equally read this as a org.joda.time.DateTime object, or
 a hirondelle.date4j.DateTime object.

 All these implementations describe the same data, but will have subtly
 different methods and interfaces. The data structure serialised in edn
 doesn't mandate a particular class, or a particular set of methods.

 Similarly, if we imagine a 

Re: OOP question re: The Language of the System

[Jonah Benton]

Will just add, in re: an earlier thread on this talk, one way to view this
is in re: the lifespan of the data.

Building and evolving the components that deal with live data passed around
between them is easier and more fluid when working with maps than with
working with types. One can add a field to a map without changing anything
else, often not so when adding a field to a class. Exposing data directly
in the data structure, rather than hiding it behind methods, forces better
data flow and management decisions and simplifies changes. The classic
example is of first and last name fields in a class that get combined in a
getFullName() method. Then something in the business rules changes and you
need to take a full name and decompose it into a first and last. Performing
changes like this is less work in a data flow design, more work in OO with
data encapsulation.

There are all sorts of decisions that one has to make with types when
working solely within an application that arguably create incidental
complexity. Types can bring more value at the edges- when exchanging data
between applications, and when persisting data in a durable form.

And as with the other responses- don't think of protocols as tied to the
data. They are not wrappers around data. Instead, think of them as their
name implies- protocols can be used to crisply describe how 2 components,
the caller and the callee, *participate* in an interaction (a protocol)
around different concrete bundles of data. (Though as in the other
responses, it's probably often the case that the interactions are not
really that crisp, so multimethods are a better choice for polymorphism.)





On Sun, Jan 19, 2014 at 2:55 PM, Brian Craft craft.br...@gmail.com wrote:

 http://www.youtube.com/watch?v=ROor6_NGIWU

 Around 56:28 Rich is talking about whether components pass a data
 structure, or an object that has all these verbs and knows how to do stuff
 

 Clojure data types also have verbs, in protocols, and if one component
 passes a record to another component, the receiver will need to use those
 verbs to make use of the data. How is that different than the OOP example?
 There's the syntactic difference that in OOP you write it object, verb,
 arguments, and in clojure you write it verb, object, arguments. But that's
 trivial. How is it architecturally different to pass an object with methods
 vs passing a record with protocol?

 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: How to handle configuration in Clojure?

[Jonah Benton]

A middle ground between dynamic vars and passing state through functions
that themselves don't need it, but which is needed by functions they call,
is to build on top of defprotocol and defrecord, to refactor so that the
functions that need configuration state in their operations are defined
by protocols, and the state they need is captured in a record, which can
then be prepared at initialization time with the necessary information.

In this solution, there is still noise in the sense that now protocol
functions have to take their record configuration as a first parameter,
but the semantic noise in having unused configuration data passing through
that first parameter should be relieved.

There's a lot more to say here, but the upshot/goal of the refactoring
should be that explicit dependencies between the function call tree-
organized into protocols- and initialization/configuration state- organized
into records- should be nicely revealed.
There are reasons that they might not, and then that's a time to look at
other state/configuration management approaches like atoms/refs and dynamic
vars.

Someone else mentioned Stuart Sierra's Component library:

https://github.com/stuartsierra/component

Stuart seems to have gone furthest in encapsulating this pattern, so his
examples are good to think about, even if you don't adopt the library.

HTH.



On Mon, Jan 13, 2014 at 8:50 AM, James Trunk james.tr...@gmail.com wrote:

 I've been investigating how to handle configuration in a Clojure
 application/library, and have discovered two main candidates: dynamics vars
 and argument passing.

 The downsides to dynamic vars seem to be: hiddenness, thread safety, and
 more complex tests (binding before each test).

 The downside to argument passing is noise in the code (especially when
 you're just passing the config through). Longer and more descriptive names
 for the config (i.e. not ctx or cfg) make this noise even more painful.

 Are there any alternatives that I've missed? What is the current best
 practice for handling configuration?

 Cheers,
 James

 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: structuring a db load, back pressure

[Jonah Benton]

I tend to think of pull operations in these situations. So perhaps wrap a
lazy sequence around a chunked form of the read + transform operation,
perhaps parallelizing the transform, and do the insert in a doseq on the
lazy sequence?




On Tue, Oct 22, 2013 at 8:03 PM, Brian Craft craft.br...@gmail.com wrote:

 I'm doing a load to db, which looks roughly like read, transform, insert,
 repeat.

 Blocking on the inserts leaves the cores sitting cold while they could be
 doing the next read/transform. It's tempting to try an agent for the
 inserts. If I understand them correctly, that would queue the inserts as
 they are ready, allowing the read/transform to continue. I'm concerned that
 this would just fill memory, as the inserts can't keep up with the reads.

 Is there some other obvious way to structure this? The only relevant hits
 I've found on this list are about the async lib, which I gather can handle
 this by keeping a bounded queue that can block the producer. I was hoping
 to put off learning the async stuff until later. ;)

 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Lifecycle Composition

[Jonah Benton]

Hi Murtaza,

The short answer is that  it's not necessarily that there are pros or cons
between having components know about each other in their constructed vs
started forms- it's that immutability imposes limitations in the way that
components can maintain local references to other components in the context
of the composite system global state management pattern Stuart is
describing.

If you want to use this pattern and have components that have circular
dependencies on each other, those components better have mutable
initialization operations. If they don't, the circular dependencies will
need to be factored out.

Jonah

==
Longer answer with an example-

Stuart's original insight, documented here [1] and referenced in that post
has to do with keeping all global system entities- like a database
connection, a routing table, a cache, a configuration object, etc, anything
that would be a singleton in a generic java application- in a single
composite data structure (map, record, vector, etc) that itself can be a
local variable, rather than storing each component individually as a global
var in a namespace, as has been the informal convention in much open source
Clojure application code. This single composite data structure pattern
allows for a clean code reloads and application state reconstruction at the
repl, easy testability, and a host of other benefits.

However, the pattern is not without its complications. Once you make a
commitment to working in accordance with this pattern, you run into the
situation where some of those individual singleton components have
lifecycle requirements, where after being created/constructed, they need
further setup to be made ready for use. A database has to be connected to.
A cache has to be started and given a backing store. A configuration object
has to be loaded with data.

In that situation you then run into the problem of managing the sequencing
of and dependencies between those singleton lifecycle operations. Perhaps
the configuration object has to be loaded with data and then given to the
database which needs to be opened, and then the opened database connection
needs to be given to the cache, which fronts the database. And whatever is
done at initialization time to get all those components ready probably has
to be undone in the reverse order when the system is shutting down.

Finally, the last piece that adds complexity to the mix is immutability.
From an immutability perspective, the result of a lifecycle initialization
operation on a component should be a new version of that component- even if
in practice the initialization of the particular component under the covers
is some mutable operation. But immutability leading to the expected
creation of a new version creates issues when there are dependencies
between the components.

For instance, say that the configuration object has to know about the
cache, because other objects expect to get the cache through configuration.
Then say the cache has to know about the database, because it has to know
about its backing store. But then the database may have to know about the
configuration object, because that's where it gets credentials and
connection URIs from.

With immutability, this cyclical knowledge dependency is impossible to
implement, as the new instance of each individual component as it is
updated with the knowledge of a predecessor dependent and is initialized,
or started, obviates the reference that a successor dependent was
initialized or started with. When individual component initialization is
a mutable operation, the cyclical update problem goes away- but there is
still a bit of code smell because the pattern of preserving the results of
what looks to be an immutable operation isn't maintained.

In any event, this is what Stuart is talking about when he makes the
distinction between components knowing about each other in the
constructed forms- as an unconnected database, a non-started cache, etc-
and their started forms- their post-initialization state of being
connected to database or as a cache that is ready for work.

Hope that helps.

1. http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded



On Fri, Sep 20, 2013 at 1:21 PM, Murtaza Husain 
murtaza.hus...@sevenolives.com wrote:

 Hi,

 I am going through the blog of Stuart Sierra regarding lifecycle
 composition. http://stuartsierra.com/2013/09/15/lifecycle-composition

 In the end there is this para -

 This technique is a form of dependency injection through constructors.
 The choice of whether to make the individual components mutable,
 stateful objects has an impact on how I can use them later on. In the
 original version of this pattern using mutable objects, each component
 gets stable references to other components it depends on. In the later
 version using immutable data structures, each component gets
 references to the *constructed* versions of other components it
 depends on, but not the *started* versions of those 

Re: Building Trees

[Jonah Benton]

I find your description interesting but I'm confused about what the actual
underlying problem is- is indexing and searching for data in documents
involved, or is that just an example? If the real problem is about
augmenting large document data set searches with potentially relevant
taxonomic or semantic terms...ok, but I'm not sure what that necessarily
has to do with frequent patterns in sequences/event streams? e.g. because
taxonomic or semantic associations have richer relationships than just
their location in a sequence...

On the other hand, if this is just about extracting sequences from event
streams- if it has to be done live, where you want to incrementally record
distinct sequences of events for a series of window lengths- that sounds
like you're about talking maintaining what are essentially lots of tries,
one for each distinct root? But if you can do it statically, then you're
basically just generating n-grams from the event sequence data and indexing
those?




On Mon, Sep 16, 2013 at 12:08 PM, Peter Mancini peter.manc...@gmail.comwrote:

 That is along the lines of my thinking. I am starting to look at zippers
 again. However much of what I need it to do either isn't documented at all
 or it was never intended to construct trees over time.

 The purpose of this is to look at sequenced events and detect frequent
 patterns. These patterns can have variations. It is built from the top down
 with the most frequent events added first, the 2nd most added second, etc.
 What ends up happening is that I see A-B-C-D a lot, for example, but I also
 see A-B-D-F sometimes. I see B-D-E-F-G rarely. So what I want to do is keep
 track of each of these. The first two are variations of each other stemming
 after A-B. What I want to do is to be able to look at D and see what events
 often precede it. I see that in all of my data A, B and C precede it.
 However just by examining the counts I can determine that C does not occur
 more than random chance so I can drop that and return B, A as my set, in
 that order because I see B preceeding more than A. Big Whoop you say. But
 now lets apply this to some interesting situations. Let's say that instead
 of letters we have topics. D is part of some greater topic concept. I want
 to know what other topics predict the existence of D such that if I see
 them I think that its possible they imply D. B and A do this, according to
 this analysis. So when I tell my search engine I'm interested in D it will
 also search for B and A knowing that while D isn't mentioned by name, there
 is a possibility it is implied when B or A are present.

 Concrete example. I want documents that talk about automobiles.
 Automobile associates with things like drivers seat steering wheel
 rear view mirror but also with car car wash Prius Ford and other
 terms. Why is this important? Let's say I don't want a general return of
 information but I am looking for a very specific document. If I don't have
 the right keyword I have to guess. A lot. I may never guess right. So I
 need to be sure my search is augmented properly. That is what this will do.
 However the number of events I will be looking at will be in the hundreds
 of millions and for some data sets will be in the billions or greater. The
 problem will be distributed. However, for today, I just need to get this to
 work on my laptop, with an eye to the future as to how I will get this to
 work on a large network of machines. So far I have loved Functional
 Programming... right up to this problem. This one is killing me. I am going
 to examine the other suggestions here for how to do it. I am certain that
 Clojure is extremely excellent at solving this kind of problem and I am
 becoming more certain I am missing some FP concept that is preventing me
 from tapping into that power!

 Solve this problem and Clojure becomes the best platform for large scale
 machine learning out there. My company has 3 complex applications up
 already making money and the bugs are 99% design issues instead of
 something not working in an expected manner. This is really fantastic and
 has me sold that Clojure was a good choice.

 --Pete


 On Monday, September 9, 2013 4:06:41 PM UTC-5, Laurent PETIT wrote:

 This is quite an interesting post, with the underlying question in my
 mind : where to put the line between pure datastructure manipulation
 with clojure.core only, and a datalog (datomic)/sql engine ?

  --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop 

Re: Lazy group-by for sorted maps?

[Jonah Benton]

Sounds like a job for partition-by:

http://clojuredocs.org/clojure_core/clojure.core/partition-by



On Mon, Aug 12, 2013 at 1:55 PM, Colin Yates colin.ya...@gmail.com wrote:

 Is there such a thing as a lazy group-by for a sequence of elements when
 the elements are sorted on the criteria used to group them?  I can imagine
 how one would look in a few lines of Clojure code but I am surprised there
 isn't one already.

 My actual criteria is that I am pulling things from a database that live
 on the many side of a one-many and I want to sort on the one FK.

 This is all based on the assumption that
 http://clojuredocs.org/clojure_core/clojure.core/group-by is eager!

 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: (newbie-ish) Modelling question - multi methods?

[Jonah Benton]

It sounds like it depends on whether the collaborators are properly known
to the command, e.g. at command generation/creation time, or to the
handler, e.g. at handler creation/registration time.

If the former, then it would make sense for each command to be a record
that was created with all required collaborators and participated in a
Handle protocol, called by the gateway.

if the latter, then handlers are simple functions that take collaborators
as initial parameters, followed by a command parameter. Then the bindings
for collaborators are wrapped in a partial prior to registration, so the
gateway calls the registered function with the command map as the single
parameter.

A factor by which to decide- do the collaborators remain the same per
command for the lifetime of the app? If the same, then assigning them every
time a command is created is wasteful, but if different, then assigning per
command is required.

That said, an issue I've run into when using types like this is that the
type distinction required for dispatch is complected with field membership
of the records. When the same fields need to be handled in potentially
different ways, then different command records need to have the same
fields; or when different fields need to be handled the same way, either
different command records dispatch to the same handler, or a single record
contains the union of fields needed for the handler. All of those are
unpleasant, so unless the handler functions neatly map to distinct
collections of fields, just using maps is more appealing.



On Mon, Aug 12, 2013 at 12:05 PM, Colin Yates colin.ya...@gmail.com wrote:

 Hi Russell,

 Maybe the concrete case will help.  I have a single entry point into which
 Commands can be posted, let's call it a CommandGateway.  This gateway will
 do many things, the least of which will be to delegate the handling of the
 command to the command handler registered with the gateway.  The strategy
 is 'how to handle a command'.

 Now, the handler for :command-a needs a database for example.  The handler
 for :command-b needs another collaborator.

 CommandGateway has no idea about this and defines a defmulti called
 handle-command which dispatches on the :type of the command (which is
 really a trivial map).

 The command handler for :command-a registers a defmethod for :command-a
 but how does the body of that defmethod access the database connection for
 example?

 I can see a few options:
  - give up on avoiding global state and have a bound 'system' register
 (i.e. a global/static service locator) - please no.
  - don't use multi-methods for this and have CommandGateway have a map of
 predicate:handler.  The handler for command-a would then register itself
 and use lexical scoping to access the required collaborators
  - use multi-methods, but have a CommandAHandler which is a stateful
 record and retains its collaborators (i.e. the database) and extends itself
 to implement command-handler)

 My question is really 'when does using types stop being idiomatic'.  So
 far I have gotten surprisingly far using maps/sequences and passing
 collaborators around.  Now it seems to use multi methods to dispatch the
 command I need the target of the dispatch to be stateful to remember its
 collaborators.

 Might all be a storm in a teacup :), hence the clarity request.



 On 12 August 2013 16:50, Russell Mull russell.m...@gmail.com wrote:

 Generally, I'd go for a simple strategy (ahem) like this:

 (defn make-handler [wibblie wooblie]
   (fn [woosy]
 eloquent code here))

 But perhaps there's something about your case that I don't understand;
 I'm not entirely sure where multimethods need to come into it, unless you
 need to change which handler you're using based on the thing it's handling.

 Russell

 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to a topic in the
 Google Groups Clojure group.
 To unsubscribe from this topic, visit
 https://groups.google.com/d/topic/clojure/hRKZTclcllQ/unsubscribe.
 To unsubscribe from this group and all its topics, send an email to
 clojure+unsubscr...@googlegroups.com.

 For more options, visit https://groups.google.com/groups/opt_out.




  --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 

Re: core.async: async java.jdbc

[Jonah Benton]

These are interesting code listings and questions, but they conflate the
syntactic detail that core.async's go is implemented as a macro, with the
semantic intention to support designs around lightweight data flow between
independent code blocks.

It might be of value to check out Go, the language, with its channels and
go-routines, which were part of the inspiration for core.async. In Go,
channels and go-routines are language rather than library features. Go also
lacks macros. In that context, answers to these questions are clear: e.g.
if you want to run an insert asynchronously and then run a select following
the insert in the context of the same transaction, the only way to do it
looks like Timothy's example, where the code implements a data flow- the
particular data to insert and select is sent over a channel, handled by a
function running in a go routine whose semantic purpose is to wait on an
input channel, do the database operation, and stick the result on an output
channel.

From on the other thread, Go doesn't have exceptions, but error handling
between go routines is usually handled with a separate error channel that
again serves to support data flow, rather than control flow.  Similarly,
the reason that a special channel-aware throw wasn't included in core.async
is that the idea isn't to use go blocks in a lexical, control-flow sense-
even though that's what the implementation expands to- it's to support
completely different data-flow oriented designs.

Hope that helps...



On Thu, Aug 1, 2013 at 5:57 AM, Alice dofflt...@gmail.com wrote:

 What if I want to run a select query in the middle of a transaction?


 On Thursday, August 1, 2013 3:33:29 AM UTC+9, tbc++ wrote:

 DB work is a IO operation, and IO shouldn't be done inside a go block (go
 blocks use limited thread pools).

 So what about something like this?

 (defn handle-db [chan]
  (thread
 (while true
   (let [[data result] (! c)]
  (run-db-transaction-as-normal data)
  (! result :done)

 Now spin up a few of these:

 (def db-chan (let [c (chan 4)]
  (dotimes [x 4]
(handle-db c))
  c))

 Now it's trivial to use this from a go block:

 (defn db-async [data]
   (go
 (let [c (chan)]
   (! db-chan [data c])
   (! c

 So this is the pattern: create processors using threads, then send data
 to those processors via a shared channel, passing in a response channel.

 Anyways, that's the way I'm thinking these days.

 Timothy




 On Wed, Jul 31, 2013 at 12:24 PM, Alice doff...@gmail.com wrote:

 I have an async http handler and I don't want it to consume a thread.

 On Thursday, August 1, 2013 3:16:52 AM UTC+9, tbc++ wrote:

 Why not use !! ?

 Timothy


 On Wed, Jul 31, 2013 at 11:58 AM, Alice doff...@gmail.com wrote:

 It doesn't produce a compile time error but I think it's not the
 correct code because the transaction can be committed while insert-async!
 is still executing.

 On Thursday, August 1, 2013 2:46:29 AM UTC+9, Sean Corfield wrote:

 On Wed, Jul 31, 2013 at 10:29 AM, Alice doff...@gmail.com wrote:
  (go
(jdbc/db-transaction [t-con db-spec]
  (! (insert-async! t-con :fruit {:name apple}

 Does this work:

 (jdbc/db-transaction [t-con db-spec]
   (go
  (! (insert-async! t-con :fruit {:name apple}

 --
 Sean A Corfield -- (904) 302-SEAN
 An Architect's View -- http://corfield.org/
 World Singles, LLC. -- http://worldsingles.com/

 Perfection is the enemy of the good.
 -- Gustave Flaubert, French realist novelist (1821-1880)

  --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clo...@googlegroups.com

 Note that posts from new members are moderated - please be patient
 with your first post.
 To unsubscribe from this group, send email to
 clojure+u...@**googlegroups.com

 For more options, visit this group at
 http://groups.google.com/**group**/clojure?hl=enhttp://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to clojure+u...@**googlegroups.com.

 For more options, visit 
 https://groups.google.com/**grou**ps/opt_outhttps://groups.google.com/groups/opt_out
 .






 --
 “One of the main causes of the fall of the Roman Empire was
 that–lacking zero–they had no way to indicate successful termination of
 their C programs.”
 (Robert Firth)

  --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clo...@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+u...@**googlegroups.com
 For more options, visit this group at
 

Re: lazy seq termination

[Jonah Benton]

Try:

(defn fetch-atom-feed
  Returns a list of ATOM chunks from the feed-url going back to the
from-date
  [feed-url]
  (if-let [chunk (fetch-atom-chunk feed-url)]
  (cons chunk (lazy-seq (fetch-atom-feed (:HREF (:PREVIOUS_LINK
chunk)))



On Tue, May 28, 2013 at 6:30 AM, mond r...@mcdermott.be wrote:

 I have managed to create a function that will read an ATOM feed
 indefinitely and now need a way to stop it ;-)

 (defn fetch-atom-chunk
   Returns an ATOM chunk from the feed-url
   [feed-url]
   (:ATOM_FEED (:body (client/get feed-url {:as :json}

 (defn fetch-atom-feed
   Returns a list of ATOM chunks from the feed-url going back to the
 from-date
   [feed-url]
   (let [chunk (fetch-atom-chunk feed-url)]
 (lazy-seq
   (cons chunk (fetch-atom-feed (:HREF (:PREVIOUS_LINK chunk)))

 (defn after?
   Returns true if first-date is after second-date
   [first-date second-date]
   ( (.getTime first-date) (.getTime second-date)))

 (defn feed-date-to-clj-date
   Patch for ATOM feed date formats like '2013-05-27T14:35:00.982+02:00'
we have to remove the final ':' from the string to enable its
 conversion
   [feed-date]
   (.parse (java.text.SimpleDateFormat. -MM-dd'T'HH:mm:ss.SSSZ)
 (str/replace feed-date #(.*)(\d\d):(\d\d) $1$2$3)))

 (defn more-feed-chunks-needed?
   Look at the atom chunk and decide if we have enough
   [from-date atom-chunk]
   (after? from-date (feed-date-to-clj-date (:UPDATED atom-chunk

 (def date-from (.parse (java.text.SimpleDateFormat. -MM-dd)
 2013-05-27))

 (take-while #(more-feed-chunks-needed? date-from %) (fetch-atom-feed
 some-url))

 ; example of the feed
 ; {:FEED_ID 5650f3ad-b793-42c6-9fef-80ba105f847d, :TITLE Delta Feed,
 :UPDATED 2013-05-28T11:15:34.860+02:00, :GENERATOR Products, :LINK
 {:REL Self, :HREF http://some-url}, :PREVIOUS_LINK {:REL
 prev-archive, :HREF http://another-url}, .

 If I run without the take-while it runs forever (so I am happy that its
 generating data) but when I add the take-while I get an empty list.

 I guess I'm missing something simple and would welcome suggestions.

 Thanks

 Ray

 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: I don't feel the absence of a debugger, because I've learnt enough that I don't ever need a debugger.

[Jonah Benton]

I just watched that video and interpreted it to mean that of the many
strategies available to understand problematic behavior in a live app:

* reading the code offline + sufficient hammock time
* reproducing the problem offline using test cases in an appropriate
simulation environment
* appropriate logging and tracing at runtime
* repl into the live app
* tool-assisted stepwise execution and runtime examination (debugging)

He's able to be sufficiently productive using the first 4 strategies and
doesn't miss the 5th.

In that video he also clarified that his recent experience in the field
with clojure has been with datomic, of which he and Rich are probably the
principal authors, so it's familiar code, with test cases and  the logging
and tracing he wants. He usually responds very quickly to bug reports on
the datomic list, often within hours, so in that case it's fair to say that
he's not being hampered by the absence of a debugger.

I've found debuggers are most useful in cases where:

* the code is unfamiliar/source is unavailable
* there isn't an appropriate staging environment or test harness
* there aren't good logs
* there's no repl

However, in a language like clojure, a live repl can fill the role (and
then some) of a debugger.

I personally can't speak to vim integration, don't use it with clojure.










On Mon, May 27, 2013 at 2:34 PM, Oskar Kvist oskar.kv...@gmail.com wrote:

 Stuart Halloway said in his video Clojure in the Field (
 http://www.infoq.com/presentations/Clojure-tips) from March 1, 2013 (I
 think): I don't feel the absence of a debugger because I've learnt enough
 that I don't ever need a debugger. I am very intrigued by that statement.
 What does he (or you, if you are reading, Stuart) mean? For me, debugging
 is the biggest thing that I don't know how to do well currently in Clojure
 (I use Vim, and have not programmed in Clojure for a while), so I am really
 interested in what he meant.

 And by the way: As I said, I have not been using Clojure for a few months,
 but: What's the state of the art of debugging for Vim users? I've been
 meaning to try out vim-fireplace and ritz and see if they work together
 well, but have not gotten around to it yet. I would very much appreciate a
 nudge in the right direction.

 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Opinion on testing strategies?

[Jonah Benton]

The feedback that comes immediately to mind:

* it sounds like property #3 complects black box vs white box and
specification vs implementation concerns
* it sounds like the feature functions potentially combine the state and
calculation layers in ways that should be internally reused, but also are
externally exposed through a public API. These two needs should probably be
decoupled
* it should be an implementation detail of X that X calls Y. This is just a
convenience. From a blackbox perspective, all that X's callers care about
is that X's contract is satisfied. They don't know and shouldn't care that
this is done by calling Y. Satisfying this behavior could be done by
calling Y, or calling a Y', or some other internal magic. The knowledge
that it currently is done by calling Y is irrelevant to the perspective of
satisfying a contract and shouldn't be reflected in testing that contract
* that said, it sounds like Y, contractually, is actually a subset of X, a
shortcut, to be used when the extra conditions that require X don't exist.
So testing Y should be able to be done through X's tests.

Some suggestions:

* it sounds like there should be another internal layer for reusable
combinations of state and calculation functionality, and Y's implementation
should get moved in there. Then Y, as a public API, is mostly just a
wrapper of that fn. X, as a public API, should also call that library fn,
not Y.
* that library fn can be tested in a whitebox fashion, with-redefs to mock
out the underlying internal functions so the unique behaviors are exercised
* for blackbox API testing, since it sounds like X and Y have
specification/contractual details in common, and Y may satisfy callers of X
who have simpler conditions- the 25 tests for X are needed, but the 5 of
those that apply when the extra X requirements are null can simply be
repurposed against the Y endpoint- as long as that's what the contracts say.

Hope that helps/makes sense, apologies if it misinterprets the question.



On Sat, Apr 6, 2013 at 8:42 PM, Steven Degutis sbdegu...@gmail.com wrote:

 Disclaimer: this isn't strictly about Clojure, more about semi-functional
 programming techniques, which Clojure excels at. (Plus I'm using Clojure
 for it.)

 Lately I've been experimenting with ditching the conventional MVC approach
 to writing a web app.

 Now controllers are just dead-simple functions that translate HTTP to/from
 our feature functions (or business logic functions). Each of these feature
 functions do some or all of these things:

 1) change some persisted state
 2) return result of calculations on persisted and/or given data
 3) call another feature function

 Functions often build upon other functions. This leads to a sort of
 pyramid, where the entry-point of the feature (usually called by a
 controller) is the tip, and the most primitive functions are at the bottom.

 Testing each individual function for behaviors #1 and #2 is
 straightforward. But I'm running into a problem testing behavior #3.

 One way is to use `with-redefs`. This technique would be used at every
 level along the pyramid (except the bottom level which doesn't call
 anything else).

 This feels very fragile. For one thing, if any of the function signatures
 change, any tests that reference that function have to change too, or
 they'll become false positives, no longer representing the real world, just
 the imaginary world that was setup in the test.

 Another way is to test state changes and return values at every level in
 the pyramid. This represents the real world more, because if any feature
 function changes, all the tests above it will fail. But there's two
 problems with this. First, there's going to be a lot of redundancy in test
 data in the vertical slice of the pyramid for a single feature. I'm on the
 fence of whether that's really a problem. Secondly and more importantly,
 each feature function's tests will also have to cover all the behaviors of
 the functions it calls, besides testing its own behaviors.

 For example, X calls Y. X has 5 behaviors and Y has 5 behaviors. So Y will
 have 5 tests, but X will have 25. This is because, from a high level, we
 only care about the feature as a whole (a.k.a. X) and Y is only an
 auxiliary function to assist X. We, the stakeholders of feature X, don't
 know or care that Y exists. So Y's tests are irrelevant to us, who just
 want to know that X does its job. So we want to see all of X and Y's
 behaviors tested, without knowing about Y, to give us confidence that X
 does all 25 things. But we still need to be responsible and test Y,
 especially because X delegates half his work to Y in the name of healthy
 abstraction.

 A third way is to sprinkle small amounts of Y's behavior into some or all
 of X's tests. This relies heavily on probability, suggesting that X is
 probably using Y because he shares at least one of Y's behaviors. If it's
 true, then we know that X shares all of Y's behaviors. It becomes a sort of
 

Re: How does clj-http work regarding https?

[Jonah Benton]

Try adding

:insecure? true

to the map. Charles dynamically generates a cert pretending to be the
target host when acting as an ssl proxy, and clj-http probably has to be
told to accept it.




On Sat, Feb 23, 2013 at 4:18 PM, larry google groups 
lawrencecloj...@gmail.com wrote:


 This might be a dumb How does the Internet work kind of question.

 I have been asked to pull data from Omniture, using the Omniture API.
 I thought this would take me an hour, but I've been working on this
 for 3 days now. I keep getting authentication errors.

 I became curious about exactly what code was sending to Omniture, so I
 downloaded Charles, the network debugging tool:

 http://www.charlesproxy.com/

 I am using clj-http to make the POST. The post is suppose to be
 https.

 This is the thing that surprises me: If I use http then I can see all
 the headers in Charles, and they all look correct. But if I use https
 (which is what I need to use) then there are no headers that I can see
 in Charles.

 Is that because Charles does not want to show me a bunch of encrypted
 garbage? Or is clj-http not adding in the headers with https? Maybe I
 need a special setting to get clj-http to correctly send to https? (I
 have not been able to find any such setting.)

 This is the code where I use clj-http (here I call http-client):

 (defn omniture-call-api [url-with-queue-method api-payload headers]
   (timbre/spy :debug  return value of omniture-call-api 
   (try+
 (http-client/post url-with-queue-method
   {:body api-payload
:headers {X-Api-Version 2
  X-WSSE headers}
:content-type :json
:socket-timeout 4000
:conn-timeout 4000
:accept :json
:client-params
 {http.protocol.allow-circular-redirects false
http.useragent
 clj-http}})
 (catch Object o (println (pp/pprint o))

 The url is:

 https://api2.omniture.com/admin/1.3/rest/?method=Report.QueueRanked

 At first I assumed this was a problem with Omniture's code. I have
 asked several question on the Developer forum at Omniture, but no one
 there could help me (Adobe has apparently cut most of the developer
 support since Adobe bought Omniture in 2009). However, I am now
 wondering if maybe I am doing this POST incorrectly.

 I am puzzled by something else as well: I call this function once, yet
 Charles shows 4 calls being made to Omniture, and my own code, when it
 prints data to the terminal, seems to show many requests being made.
 Why would that be?

 Again, if I change the URL so it uses http then in Charles I can see
 all the headers that suppose to be in this line of code:

 X-WSSE headers

 and the headers look correct (I posted them to the developer forums at
 Omniture and the one guy from Omniture who gave it a look felt there
 was nothing terribly amiss -- but he couldn't rule anything out.)

 But If I change the URL to use https, then I see no headers in
 Charles. Why is that?

  lawrence







 --
 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google Groups
 Clojure group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: How does clj-http work regarding https?

[Jonah Benton]

Ok- so in Charles, you'll need to do that, tell it to ssl proxy the domain

api2.omniture.com

Described in a little more detail here:

http://www.charlesproxy.com/documentation/proxying/ssl-proxying/





On Sat, Feb 23, 2013 at 5:47 PM, larry google groups 
lawrencecloj...@gmail.com wrote:

  Try adding
 
  :insecure? true
 
  to the map. Charles dynamically generates a cert pretending to be the
  target host when acting as an ssl proxy, and clj-http probably has to be
  told to accept it.

 Okay, I've done so, but I don't think I understand what you are
 telling me. Are you saying that clj-http won't send its POST if it
 feels the cert is false?

 Charles has consistently said, in each report, SSL Proxying not
 enabled for this host: enable in Proxy Settings, SSL locations

 And I've made the change you suggested, but I still see that message.



 On Feb 23, 5:24 pm, Jonah Benton jo...@jonah.com wrote:
  Try adding
 
  :insecure? true
 
  to the map. Charles dynamically generates a cert pretending to be the
  target host when acting as an ssl proxy, and clj-http probably has to be
  told to accept it.
 
  On Sat, Feb 23, 2013 at 4:18 PM, larry google groups 
 
 
 
 
 
 
 
  lawrencecloj...@gmail.com wrote:
 
   This might be a dumb How does the Internet work kind of question.
 
   I have been asked to pull data from Omniture, using the Omniture API.
   I thought this would take me an hour, but I've been working on this
   for 3 days now. I keep getting authentication errors.
 
   I became curious about exactly what code was sending to Omniture, so I
   downloaded Charles, the network debugging tool:
 
  http://www.charlesproxy.com/
 
   I am using clj-http to make the POST. The post is suppose to be
   https.
 
   This is the thing that surprises me: If I use http then I can see all
   the headers in Charles, and they all look correct. But if I use https
   (which is what I need to use) then there are no headers that I can see
   in Charles.
 
   Is that because Charles does not want to show me a bunch of encrypted
   garbage? Or is clj-http not adding in the headers with https? Maybe I
   need a special setting to get clj-http to correctly send to https? (I
   have not been able to find any such setting.)
 
   This is the code where I use clj-http (here I call http-client):
 
   (defn omniture-call-api [url-with-queue-method api-payload headers]
 (timbre/spy :debug  return value of omniture-call-api 
 (try+
   (http-client/post url-with-queue-method
 {:body api-payload
  :headers {X-Api-Version 2
X-WSSE headers}
  :content-type :json
  :socket-timeout 4000
  :conn-timeout 4000
  :accept :json
  :client-params
   {http.protocol.allow-circular-redirects false
  http.useragent
   clj-http}})
   (catch Object o (println (pp/pprint o))
 
   The url is:
 
  https://api2.omniture.com/admin/1.3/rest/?method=Report.QueueRanked
 
   At first I assumed this was a problem with Omniture's code. I have
   asked several question on the Developer forum at Omniture, but no one
   there could help me (Adobe has apparently cut most of the developer
   support since Adobe bought Omniture in 2009). However, I am now
   wondering if maybe I am doing this POST incorrectly.
 
   I am puzzled by something else as well: I call this function once, yet
   Charles shows 4 calls being made to Omniture, and my own code, when it
   prints data to the terminal, seems to show many requests being made.
   Why would that be?
 
   Again, if I change the URL so it uses http then in Charles I can see
   all the headers that suppose to be in this line of code:
 
   X-WSSE headers
 
   and the headers look correct (I posted them to the developer forums at
   Omniture and the one guy from Omniture who gave it a look felt there
   was nothing terribly amiss -- but he couldn't rule anything out.)
 
   But If I change the URL to use https, then I see no headers in
   Charles. Why is that?
 
    lawrence
 
   --
   --
   You received this message because you are subscribed to the Google
   Groups Clojure group.
   To post to this group, send email to clojure@googlegroups.com
   Note that posts from new members are moderated - please be patient with
   your first post.
   To unsubscribe from this group, send email to
   clojure+unsubscr...@googlegroups.com
   For more options, visit this group at
  http://groups.google.com/group/clojure?hl=en
   ---
   You received this message because you are subscribed to the Google
 Groups
   Clojure group.
   To unsubscribe from this group and stop receiving emails from it, send
 an
   email to clojure

Re: How does clj-http work regarding https?

[Jonah Benton]

Very likely it's the automatic retry logic:

;; Apache's http client automatically retries on IOExceptions, if
you;; would like to handle these retries yourself, you can specify a;;
:retry-handler. Return true to retry, false to stop
trying:(client/post http://example.org; {:multipart [[title Foo]
   [Content/type text/plain]
   [file
(clojure.java.io/file /tmp/missing-file)]]
   :retry-handler (fn [ex try-count
http-context]
(println Got: ex)
(if ( try-count
4) false true))})

from

https://github.com/dakrone/clj-http




On Sat, Feb 23, 2013 at 5:57 PM, larry google groups 
lawrencecloj...@gmail.com wrote:


 Any idea why a single call to clj-http/post causes 4 transactions to
 appear in Charles?



 On Feb 23, 5:47 pm, larry google groups lawrencecloj...@gmail.com
 wrote:
   Try adding
 
   :insecure? true
 
   to the map. Charles dynamically generates a cert pretending to be the
   target host when acting as an ssl proxy, and clj-http probably has to
 be
   told to accept it.
 
  Okay, I've done so, but I don't think I understand what you are
  telling me. Are you saying that clj-http won't send its POST if it
  feels the cert is false?
 
  Charles has consistently said, in each report, SSL Proxying not
  enabled for this host: enable in Proxy Settings, SSL locations
 
  And I've made the change you suggested, but I still see that message.
 
  On Feb 23, 5:24 pm, Jonah Benton jo...@jonah.com wrote:
 
 
 
 
 
 
 
   Try adding
 
   :insecure? true
 
   to the map. Charles dynamically generates a cert pretending to be the
   target host when acting as an ssl proxy, and clj-http probably has to
 be
   told to accept it.
 
   On Sat, Feb 23, 2013 at 4:18 PM, larry google groups 
 
   lawrencecloj...@gmail.com wrote:
 
This might be a dumb How does the Internet work kind of question.
 
I have been asked to pull data from Omniture, using the Omniture API.
I thought this would take me an hour, but I've been working on this
for 3 days now. I keep getting authentication errors.
 
I became curious about exactly what code was sending to Omniture, so
 I
downloaded Charles, the network debugging tool:
 
   http://www.charlesproxy.com/
 
I am using clj-http to make the POST. The post is suppose to be
https.
 
This is the thing that surprises me: If I use http then I can see all
the headers in Charles, and they all look correct. But if I use https
(which is what I need to use) then there are no headers that I can
 see
in Charles.
 
Is that because Charles does not want to show me a bunch of encrypted
garbage? Or is clj-http not adding in the headers with https? Maybe I
need a special setting to get clj-http to correctly send to https? (I
have not been able to find any such setting.)
 
This is the code where I use clj-http (here I call http-client):
 
(defn omniture-call-api [url-with-queue-method api-payload headers]
  (timbre/spy :debug  return value of omniture-call-api 
  (try+
(http-client/post url-with-queue-method
  {:body api-payload
   :headers {X-Api-Version 2
 X-WSSE headers}
   :content-type :json
   :socket-timeout 4000
   :conn-timeout 4000
   :accept :json
   :client-params
{http.protocol.allow-circular-redirects false
   http.useragent
clj-http}})
(catch Object o (println (pp/pprint o))
 
The url is:
 
   https://api2.omniture.com/admin/1.3/rest/?method=Report.QueueRanked
 
At first I assumed this was a problem with Omniture's code. I have
asked several question on the Developer forum at Omniture, but no one
there could help me (Adobe has apparently cut most of the developer
support since Adobe bought Omniture in 2009). However, I am now
wondering if maybe I am doing this POST incorrectly.
 
I am puzzled by something else as well: I call this function once,
 yet
Charles shows 4 calls being made to Omniture, and my own code, when
 it
prints data to the terminal, seems to show many requests being made.
Why would that be?
 
Again, if I change the URL so it uses http then in Charles I can
 see
all the headers that suppose to be in this line of code:
 
X-WSSE headers
 
and the headers look correct (I posted them to the developer forums
 at
Omniture and the one guy from Omniture who gave it a look felt there
was nothing terribly amiss

Re: How does clj-http work regarding https?

[Jonah Benton]

It won't retry on http-level errors; those semantics are up to the
application. It should only retry connection-layer problems. So there must
have been something funny happening either between the client and Charles,
or between Charles and Omniture.



On Sat, Feb 23, 2013 at 6:50 PM, larry google groups 
lawrencecloj...@gmail.com wrote:

  Very likely it's the automatic retry logic:


 Thank you for that. One mystery solved. I wonder why it retries? The
 ping reaches Omniture, I get back a 401 error. Maybe it retries on any
 4xx error?




 On Feb 23, 6:30 pm, Jonah Benton jo...@jonah.com wrote:
  Very likely it's the automatic retry logic:
 
  ;; Apache's http client automatically retries on IOExceptions, if
  you;; would like to handle these retries yourself, you can specify a;;
  :retry-handler. Return true to retry, false to stop
  trying:(client/post http://example.org; {:multipart [[title Foo]
 [Content/type
 text/plain]
 [file
  (clojure.java.io/file /tmp/missing-file)]]
 :retry-handler (fn [ex try-count
  http-context]
  (println Got: ex)
  (if ( try-count
  4) false true))})
 
  from
 
  https://github.com/dakrone/clj-http
 
  On Sat, Feb 23, 2013 at 5:57 PM, larry google groups 
 
 
 
 
 
 
 
  lawrencecloj...@gmail.com wrote:
 
   Any idea why a single call to clj-http/post causes 4 transactions to
   appear in Charles?
 
   On Feb 23, 5:47 pm, larry google groups lawrencecloj...@gmail.com
   wrote:
 Try adding
 
 :insecure? true
 
 to the map. Charles dynamically generates a cert pretending to be
 the
 target host when acting as an ssl proxy, and clj-http probably has
 to
   be
 told to accept it.
 
Okay, I've done so, but I don't think I understand what you are
telling me. Are you saying that clj-http won't send its POST if it
feels the cert is false?
 
Charles has consistently said, in each report, SSL Proxying not
enabled for this host: enable in Proxy Settings, SSL locations
 
And I've made the change you suggested, but I still see that message.
 
On Feb 23, 5:24 pm, Jonah Benton jo...@jonah.com wrote:
 
 Try adding
 
 :insecure? true
 
 to the map. Charles dynamically generates a cert pretending to be
 the
 target host when acting as an ssl proxy, and clj-http probably has
 to
   be
 told to accept it.
 
 On Sat, Feb 23, 2013 at 4:18 PM, larry google groups 
 
 lawrencecloj...@gmail.com wrote:
 
  This might be a dumb How does the Internet work kind of
 question.
 
  I have been asked to pull data from Omniture, using the Omniture
 API.
  I thought this would take me an hour, but I've been working on
 this
  for 3 days now. I keep getting authentication errors.
 
  I became curious about exactly what code was sending to
 Omniture, so
   I
  downloaded Charles, the network debugging tool:
 
 http://www.charlesproxy.com/
 
  I am using clj-http to make the POST. The post is suppose to be
  https.
 
  This is the thing that surprises me: If I use http then I can
 see all
  the headers in Charles, and they all look correct. But if I use
 https
  (which is what I need to use) then there are no headers that I
 can
   see
  in Charles.
 
  Is that because Charles does not want to show me a bunch of
 encrypted
  garbage? Or is clj-http not adding in the headers with https?
 Maybe I
  need a special setting to get clj-http to correctly send to
 https? (I
  have not been able to find any such setting.)
 
  This is the code where I use clj-http (here I call http-client):
 
  (defn omniture-call-api [url-with-queue-method api-payload
 headers]
(timbre/spy :debug  return value of omniture-call-api 
(try+
  (http-client/post url-with-queue-method
{:body api-payload
 :headers {X-Api-Version 2
   X-WSSE headers}
 :content-type :json
 :socket-timeout 4000
 :conn-timeout 4000
 :accept :json
 :client-params
  {http.protocol.allow-circular-redirects false
 
  http.useragent
  clj-http}})
  (catch Object o (println (pp/pprint o))
 
  The url is:
 
 
 https://api2.omniture.com/admin/1.3/rest/?method=Report.QueueRanked
 
  At first I assumed this was a problem with Omniture's code. I
 have
  asked several question on the Developer forum at Omniture, but
 no one
  there could help me (Adobe has apparently

Re: How does clj-http work regarding https?

[Jonah Benton]

If you cut Charles out of the picture and just send your payload directly
to Omniture over https, how does Omniture respond?


On Sat, Feb 23, 2013 at 6:52 PM, larry google groups 
lawrencecloj...@gmail.com wrote:

 
  Described in a little more detail here:
 
  http://www.charlesproxy.com/documentation/proxying/ssl-proxying/
 


 Thank you, that is a huge help.

 I am finding it is a real headache to use several new technologies,
 all at once.




 On Feb 23, 6:28 pm, Jonah Benton jo...@jonah.com wrote:
  Ok- so in Charles, you'll need to do that, tell it to ssl proxy the
 domain
 
  api2.omniture.com
 
  Described in a little more detail here:
 
  http://www.charlesproxy.com/documentation/proxying/ssl-proxying/
 
  On Sat, Feb 23, 2013 at 5:47 PM, larry google groups 
 
 
 
 
 
 
 
  lawrencecloj...@gmail.com wrote:
Try adding
 
:insecure? true
 
to the map. Charles dynamically generates a cert pretending to be the
target host when acting as an ssl proxy, and clj-http probably has
 to be
told to accept it.
 
   Okay, I've done so, but I don't think I understand what you are
   telling me. Are you saying that clj-http won't send its POST if it
   feels the cert is false?
 
   Charles has consistently said, in each report, SSL Proxying not
   enabled for this host: enable in Proxy Settings, SSL locations
 
   And I've made the change you suggested, but I still see that message.
 
   On Feb 23, 5:24 pm, Jonah Benton jo...@jonah.com wrote:
Try adding
 
:insecure? true
 
to the map. Charles dynamically generates a cert pretending to be the
target host when acting as an ssl proxy, and clj-http probably has
 to be
told to accept it.
 
On Sat, Feb 23, 2013 at 4:18 PM, larry google groups 
 
lawrencecloj...@gmail.com wrote:
 
 This might be a dumb How does the Internet work kind of question.
 
 I have been asked to pull data from Omniture, using the Omniture
 API.
 I thought this would take me an hour, but I've been working on this
 for 3 days now. I keep getting authentication errors.
 
 I became curious about exactly what code was sending to Omniture,
 so I
 downloaded Charles, the network debugging tool:
 
http://www.charlesproxy.com/
 
 I am using clj-http to make the POST. The post is suppose to be
 https.
 
 This is the thing that surprises me: If I use http then I can see
 all
 the headers in Charles, and they all look correct. But if I use
 https
 (which is what I need to use) then there are no headers that I can
 see
 in Charles.
 
 Is that because Charles does not want to show me a bunch of
 encrypted
 garbage? Or is clj-http not adding in the headers with https?
 Maybe I
 need a special setting to get clj-http to correctly send to https?
 (I
 have not been able to find any such setting.)
 
 This is the code where I use clj-http (here I call http-client):
 
 (defn omniture-call-api [url-with-queue-method api-payload headers]
   (timbre/spy :debug  return value of omniture-call-api 
   (try+
 (http-client/post url-with-queue-method
   {:body api-payload
:headers {X-Api-Version 2
  X-WSSE headers}
:content-type :json
:socket-timeout 4000
:conn-timeout 4000
:accept :json
:client-params
 {http.protocol.allow-circular-redirects false
http.useragent
 clj-http}})
 (catch Object o (println (pp/pprint o))
 
 The url is:
 
https://api2.omniture.com/admin/1.3/rest/?method=Report.QueueRanked
 
 At first I assumed this was a problem with Omniture's code. I have
 asked several question on the Developer forum at Omniture, but no
 one
 there could help me (Adobe has apparently cut most of the developer
 support since Adobe bought Omniture in 2009). However, I am now
 wondering if maybe I am doing this POST incorrectly.
 
 I am puzzled by something else as well: I call this function once,
 yet
 Charles shows 4 calls being made to Omniture, and my own code,
 when it
 prints data to the terminal, seems to show many requests being
 made.
 Why would that be?
 
 Again, if I change the URL so it uses http then in Charles I can
 see
 all the headers that suppose to be in this line of code:
 
 X-WSSE headers
 
 and the headers look correct (I posted them to the developer
 forums at
 Omniture and the one guy from Omniture who gave it a look felt
 there
 was nothing terribly amiss -- but he couldn't rule anything out.)
 
 But If I change the URL to use https, then I see no headers

Re: Attractive examples of function-generating functions

[Jonah Benton]

You've probably seen these, but if not, Doug Crockford's video series
on javascript walks through a number of interesting information
sharing examples like the ones you're looking for using
fn-generating-fns-

http://yuiblog.com/crockford/

They're all great but act 3 - function the ultimate is especially juicy.

The motivation for his examples is a little different than it would be
for clojure, because that pattern is basically javascript's only
abstraction trick. And certainly the semantics are different too.

But if for whatever reason you haven't seen these videos, they're
terrific and will probably spur some ideas.


On Wed, Aug 8, 2012 at 12:48 PM, Brian Marick mar...@exampler.com wrote:
 I'm looking for medium-scale examples of using function-generating functions. 
 I'm doing it because examples like this:

 (def make-incrementer
  (fn [increment]
(fn [x] (+ increment x

 ... or this:

 (def incish (partial map + [100 200 300]))

 ... show the mechanics, but I'm looking for examples that would resonate more 
 with an object-oriented programmer. Such examples might be ones that close 
 over a number of values (which looks more like an object), or generate 
 multiple functions that all close over a shared value (which looks more like 
 an object), or use closures to avoid the need to have some particular 
 argument passed from function to function (which looks like the `this` in an 
 instance method).

 Note: please put the flamethrower down. I'm not saying that looking like 
 objects is the point of higher-order functions.

 I'll give full credit.

 -
 Brian Marick, Artisanal Labrador
 Contract programming in Ruby and Clojure
 Occasional consulting on Agile


 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with your 
 first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Libraries and build management hell

[Jonah Benton]

Re:


 Hopefully some bright spark will come along and revolutionise dependency
 management, like how Rich revolutionised the notion of state in
 programming.  As far as I'm aware nobody has so far, for any programming
 language (or OS distro), I'm not completely happy with any of them.

 It needs more hammock time.

What golang has done is interesting, insofar that a more complete
lifecycle is supported with the out of the box tools.

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Idea for personal Clojure project

[Jonah Benton]

As others have said, there isn't an algorithm that does this. Useful
results depend on precise definitions of context and similarity.
The waters get deep quickly.

As a clojure exercise, though, there are lots of good starting points.
For instance: get a set of words, create all pairs from the set, run a
google search on each pair, extract the count of documents from the
results, use the counts as a distance between the words as nodes, and
throw that in a graph.

Something like this would make an interesting topology, and could be
enhanced by using a different corpus, and/or swapping in different
distance measurements. Though as described it would not tell you
anything interesting semantically.

For a slightly more sophisticated framing of the problem, look at NLP
programming assignments, like wordnet distance, e.g.

http://www.cs.princeton.edu/courses/archive/spr07/cos226/assignments/wordnet.html

Hope that helps.

On Wed, Jul 28, 2010 at 4:58 PM, Daniel doubleagen...@gmail.com wrote:
 I want to write a clojure program that searches for similarities of
 words in the english language and places them in a graph, where the
 distance between nodes indicates their similarity.  I don't mean
 syntactical similarity.  Related contextual meaning is closer to the
 mark.

 For instance: fish and reel don't have much similarity, but in the
 context of fishing they do, so the distance in such a graph wouldn't
 be very large.

 I'm sure research has been done in this area (I suspect with no small
 portion belonging to google), so can anybody point me in the right
 direction?

 Thanks.

 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with your 
 first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Interest in creating a NYC Clojure user group?

[Jonah Benton]

Not sure if I could make it on a regular basis, but on a related note,
those in NYC may be interested in:

http://bits.blogs.nytimes.com/2009/06/29/new-york-city-starts-contest-for-big-apple-apps/?ref=technology

Could be an interesting use case for clj + cloud resources

On Thu, Jun 4, 2009 at 12:09 PM, Eric Thorsenethor...@enclojure.org wrote:

 I went to the Bay Area Clojure group meeting last night which was
 great.  People demoed some very interesting stuff and it was great
 having face time with more people using Clojure.   I wanted to see
 what kind of interest there might be in creating a Clojure user group
 in the NY metro area to meet up in Manhattan once a month to discuss
 all things Clojure. My company is in Westchester but I can _probably_
 get some space (depending on how many of us there are) to meet.

 Eric

 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: [OT] Convincing others about Clojure

[Jonah Benton]

FWIW, two points- Paul Graham, among others, has talked about issues
like this. See for instance (about Python):

http://www.paulgraham.com/pypar.html

The argument about using new technologies in the startup context is
generally that smarter people want to work with better tools at higher
levels of abstraction. Those people are naturally harder to find, but
a way to attract them is to use tools that can give very skilled
people more leverage.

Relatedly, being concerned about supporting the technology with
commodity skills is something for the future owners of your startup to
care about, not you.

That said, delivery risk with new technologies is *always* a concern.
So the other point is- sometimes you can address concerns about
risky choices obliquely by raising *your own* concerns about those
choices, to redirect the conversation to areas that make sense to you.
This indicates that *you're* thinking about what they're thinking
about, and maybe you have a different take on what the risks are- but
that you're thinking about it means you're going to takes steps to
mitigate the risks and keep the channels of communication open.

My main concern about using Clojure wouldn't have to do with people or
performance or academic/business focus, it would just have to do with
maturity, not so much of the language, more at the level of
application patterns. To that end, were I starting a project, I'd look
for skilled Java people with some Lisp, and approach Clojure not as a
Lisp, but as a better Spring, and use the patterns from Spring to
minimize delivery risk. Focus mostly on creating performance sensitive
stateless components + interfaces in Java, using non-bean generic data
structures, and do the configuration wiring/lifecycle/dataflow stuff
in Clojure. More work, but safer ramp-up. Then I'd go deeper into
Lispifying a subsystem.

$0.02.

On Thu, Jun 25, 2009 at 1:59 AM, Baishampayan Ghoseb.gh...@ocricket.com wrote:
 Hello,

 So I have been a Common Lisp user for quite sometime and at my earlier
 work we managed to build a state-of-the-art Travel portal in CL with a
 very small team of CL programmers.

 This was all fine except one thing. The management never really believed
 in Lisp and they eventually replaced the whole Lisp team with 3x Java
 programmers and are now rewriting the perfectly fine system in Java.

 That was my earlier job.

 Now I am at the moment doing a startup and I was thinking of using
 Clojure because it has the best of both the worlds. It can use Java
 libs, is a Lisp and is heavily geared towards concurrent programming;
 making it one of the most modern and pragmatic programming languages at
 the moment.

 This decision was based purely on the merit of Clojure and not because I
 just wanted to do some Lisp programming. I seriously believe that
 Clojure can really help us in building the kind of concurrent
 application we want to build.

 But then, there is another problem. The advisors of the current startup
 (who were techies in their time, but now are highly successful people in
 the Silicon Valley) reacted strongly to the word Lisp (it apparently
 brought back old memories of their AI class in college) and are not
 convinced enough about Clojure.

 I tried explaining that Clojure runs on the JVM and thus won't have any
 problem with libs or integrating with existing Java apps but they are
 not happy.

 Their concerns are thus:

 1. How do you get Clojure programmers? Lisp is not for the faint hearted.

 2. What about the performance of Clojure? Is it fast?

 3. People who want to use this are more academically inclined and are
 not practical. This will make the whole project fail.

 I need some pointers on this. This is a really crucial thing for me and
 any help will be appreciated.

 Regards,
 BG

 --
 Baishampayan Ghose b.gh...@ocricket.com
 oCricket.com



--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: Mnesia like?

[Jonah Benton]

Another schemaless db is mongo:

http://www.mongodb.org

It's written in c++, so it's out of process, but using the java driver
is pretty natural:

http://www.mongodb.org/display/DOCS/Java+Tutorial



On Tue, Jun 16, 2009 at 3:24 AM, Wilson MacGyverwmacgy...@gmail.com wrote:

 Sorry I wasn't very clear. What I meant was I didn't know if there
 was something that's tightly integrated and feels very
 native language-ish like Mnesia and erlang, but for clojure.

 It doesn't sound like such thing exist yet...

 On Tue, Jun 16, 2009 at 3:13 AM, rb raphi...@gmail.com wrote:

 On Jun 15, 6:02 pm, Wilson MacGyver wmacgy...@gmail.com wrote:
 Does clojure have anything like erlang's Mnesia? or is anyone working on 
 such
 project? I know I can fall back to using JDBC+ various RDBMS, but I
 was curious if there is something that works like Mnesia.

 Depending on what you mean by works like Mnesia, you might be
 interested to look at
 http://hadoop.apache.org/hbase/

 Cheers

 Raph


 Thanks,
 Mac

 --
 Omnem crede diem tibi diluxisse supremum.
 




 --
 Omnem crede diem tibi diluxisse supremum.

 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: Mnesia like?

[Jonah Benton]

Ah, that looks very nice...

On Tue, Jun 16, 2009 at 1:19 PM, Jim Menardjim.men...@gmail.com wrote:

 On Tue, Jun 16, 2009 at 9:26 AM, Jonah Bentonjo...@jonah.com wrote:

 Another schemaless db is mongo:

 http://www.mongodb.org

 It's written in c++, so it's out of process, but using the java driver
 is pretty natural:

 http://www.mongodb.org/display/DOCS/Java+Tutorial

 See also Geir Magnusson Jr.'s MongoDB Java driver, which comes with a
 Clojure example that I wrote for him last year:

 http://github.com/geir/mongo-java-driver/tree/master

 http://github.com/geir/mongo-java-driver/blob/ca5b3ab3c2ab1caf8918cc84902abb7b476ba52b/src/examples/clojure/mongo.clj

 Jim
 --
 Jim Menard, j...@io.com, jim.men...@gmail.com
 http://www.io.com/~jimm/

 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: performance concerns (again)

[Jonah Benton]

Snippets responded to below...

On Mon, Jun 8, 2009 at 5:43 PM, Robert Lehrrobert.l...@gmail.com wrote:

 I executed a simpler test in Java only, one that prints a simple
 Sleeping 30sec message then sleeps for 30sec.  The 30sec delay is so
 that I can record the memory allocated by JVM.

    * on a Windows XP laptop a 2.2GHz dual-core CPU and 2GB RAM
    * on several Linux v2.6 systems w/ 8 cores of ~2.6GHz and 16GB RAM
    * on several Linux v2.4 systems w/ the same configs as those w/
 the v2.6 kernels

 All tests were executed w/ a v6 JVM.

 NOTE - Linux version numbers refer to the kernel's version.

 In short, only the linux v2.6 kernels showed the exorbitant
 memory-utilization that I reported previously.  On both Windows XP and
 Linux v2.4, the memory allocated was on the order of MBs, not
 HUNDREDS of MBs.

 NOTE - in practice, I consider the 10, 11 and 12MBs that I observed to
 be on the order of MBs, not TENs of MBs.


Can't reproduce:

[jo...@x ~]$ uname -a
Linux x 2.6.18-128.1.10.el5 #1 SMP Wed Apr 29 13:53:08 EDT 2009
x86_64 x86_64 x86_64 GNU/Linux
[jo...@x ~]$ java -version
java version 1.6.0_13
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
[jo...@x ~]$ cat Test.java
public class Test
{
public static void main(String[] inargs)
{
  while(true)
  {
System.out.println(Sleeping);
try {
  Thread.sleep(3);
}
catch(Exception e) {
  System.out.println(Interrupted...);
  break;
}
  }
}
}

[jo...@x ~]$ javac Test.java
[jo...@x ~]$ java Test 
[1] 28527
[jo...@x ~]$ Sleeping
[jo...@x ~]$ ps -o comm,rss,vsz -p 28527
COMMAND   RSSVSZ
java15308 8533300
[jo...@x ~]$ Sleeping
[jo...@x ~]$ ps -o comm,rss,vsz -p 28527
COMMAND   RSSVSZ
java15308 8533300

Care to post yours?


 One would guess then that the overcommit feature of the Linux v2.6
 kernel is causing this impression of exorbitant memory-utilization.
 The question remains then whether the overcommit affects the
 scalability of the JVM on Linux v2.6 systems.  IOW, for a simple
 program that causes the JVM to allocate hundreds of megs of RAM on a
 multi-gigabyte system, will the number of JVM instances be severely
 limited?  For example, if the JVM allocates 400MB on a 16GB system, is
 the total number of JVM instances limited to ~40 (16000/400 or to the
 ~1300 (16000/12) that I would otherwise expect.  I can explore that
 indepedently and share the results here later if anyone is interested.


Nah. Overcommit doesn't mean that, and it doesn't have anything to do
with the behavior you're seeing. I don't mean this as a criticism of
your analysis, but you're being fooled, something in your setup is
broken.


 Errrm - so Java's value is limited to large systems ?  That statement
 does not seem to take into account all of the small systems, including
 mobile devices, that use Java.  What am I missing here?


They're completely different platforms. Same language, a few
commonalities in platform API, but completely different runtimes. The
regular JDK that ships on desktops and servers is not close to
suitable for constrained memory systems, and the ME platform isn't
close to suitable for enterprise work.


 When a comparable C++ progie allocates 40k, I do not consider an
 initial allocation of 400MB to be acceptable for any system.  It's
 wasteful enough to begin reconsidering whether the benefits of the
 system in question are worth the cost.

 IOW, when I see a increase of 4 orders of magnitude of attribute A in
 exchange for 1 order of magnitude of improvement in attribute B, my
 alarms are triggered.  I do my value calculations, start asking
 questions and looking for something to kill.  In this case, the JVM
 allocating 400MB instead of 40k; I expected something on the order of
 MBs, not HUNDREDs of MBs.

 NOTE that the above comment is a general statement and does not
 reflect my observations of the v6 JVM on Windows XP and Linux v2.4.x.


Well, there is no way an SE JVM can take less than 16M for hello
world, but that said-, trust me, if you dig further you'll discover
some configuration issue that explains the discrepancy between 16M and
400M. :) No one would use Java if a minimal working set of 400M was
even a corner case possibility.


 True.  Except that I can compare these metrics to values obtained for
 corresponding programs in other languages and on similar OSes, i.e.,
 linux vs linux, Windows vs Windows.  A metric may not be accurate.
 But if it is applied consistently then it can be used for comparisons.


Well...the userspace visible metrics are inaccurate in *different
ways* across platforms, even (especially) between linux 2.4 + 2.6.
Nevertheless, agreed, for a trivial app the numbers should be
basically identical.

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google

Re: performance concerns (again)

[Jonah Benton]

RSS is resident set size- as I recall from the days when I compiled
my own kernels, it's based on a lazily-maintained
not-guaranteed-to-be-accurate count of physical memory pages in use
by the process. On linux, this number may overstate memory use by 50%
or more for non-JVM processes. For JVM processes the overcount may
be much greater.

On Sat, Jun 6, 2009 at 8:46 AM, Jarkko Oranenchous...@gmail.com wrote:


 The problem was that it was not as fast as I expected it should be
 given that it was using no less than 100% of the CPU on my system.
 (two 3GHz Xeon CPUs [1 core]; 3GB RAM; a beefy workstation).  That
 this was visible in the GUI shows how slow it appeared to me.  Also,
 it was using 700MB RAM (VIRT in top).  Sure - it was swapped (I'm
 familiar w/ some of the interpretations of these memory issues) except
 that my system has ZERO swap space.  PMAP showed that 400MB of it was
 heap, too, not libraries or binaries or anything else that we can
 safely ignore.  This was (apparently) real, allocated heap, private to
 the process's address space.

 I doubt the VIRT size matters at all. From what I know, it represents
 the address space that is available to the process; there's no
 guarantee that it's actually allocated or in use. Hence, it's not even
 swapped. On my system, I have several processes with hundreds of
 megabytes in their VSIZE column shown in top. About 12GB in total;
 yet, I have barely any swap space... only a single 64MB file is
 allocated. (OS X allocates swapspace dynamically)
 According to top, I have 0 pageouts since last boot, so the swap isn't
 even being used. I have 3GB of RAM.

 Additionally, as the simulation ran, the initial RSS of 60MB rose to
 130MB then stopped.  The VIRT remained constant.  I had expected that
 - however I remained concerned.

 I'm not sure what kind of memory RSS is, but if it's shared, then
 it's java's own overhead, and not the application itself.
 the private memory areas are what you're interested in. Though of
 course the overhead is meaningful if it's the only java app running,
 but it's very difficult to tell what amount of the shared memory is
 actually used only by your application.

 You can try tuning the java VM and decrease its heap size and other
 things; see if it makes a difference.

 --
 Jarkko

 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: performance concerns (again)

[Jonah Benton]

Hi Robert,

I haven't been able to dig into Clojure/JVM/GC details, so I can't
speak in particular about problems e.g. with the ants application, but
there are few useful conclusions that one can draw from a JVM's RSS
and VIRT. It's unfortunate, but in general, Java app memory behavior
can't be reliably reported on by standard unix tools. Further, for
non-trivial applications, GC parameters often need to be custom-tuned
to avoid pathological behavior.

In its defense, the knobs the JVM makes available are much more
granular than knobs one typically has available at configuration or
run time in C++. And when GC and the memory model is properly tuned,
in many cases HotSpot code can run at comparable to or better
efficiency than a well written C/C++ app.

That said, in my experience, GC configuration is a big problem for
typical Java users, and it contributes to ongoing concerns about Java.
Every few weeks it seems another colleague reports service outages or
frequent restarts with an app, usually a web app. These problems are
usually fixable with a few changes to GC configuration. But few Java
developers have familiarized themselves with the JVM's behavior at
that level, and few operations folks, even those who have skills in
troubleshooting application problems, are familiar in particular with
JVM tuning.

There are certainly still cases where Java can't compete with C++, but
they're fewer and further between- quick startup and short-run apps
excepted. For long running apps, HotSpot code gen is so good, even
Fortran people are starting to migrate to the JVM. But as you point
out, proper memory management is a key part of the solution, and in
general it's still a bit of a black art.

In any event, try running your JVM with

-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
-XX:+PrintHeapAtGC -Xloggc:/tmp/gc.out

and watch or post details from the file at

/tmp/gc.out

for commentary.

Jonah


On Fri, Jun 5, 2009 at 5:18 AM, Robert Lehrrobert.l...@gmail.com wrote:

 First, Clojure has renewed my interest in Java, specifically the JVM
 and the plethora of available libraries.  Clojure and Scala are the
 kind of projects that inspire peoples' imaginations in a world that
 they might otherwise overlook.  Thank you, Rich Hickey.

 However, I have some concerns that fundamentally stem from the JVM's
 performance characteristics.  I hope to obtain some answers for them
 that were as thoughtful as the other discussion that I read in the
 group.

 I am dealing w/ some performance constraints in the current
 implementation of my application.  So, although Clojure offers some
 features that would solve some of my problems, I am concerned about
 Clojure's and the JVM's performance issues.  I read the when
 performance matters thread and, as usual, was struck by the usual
 omission of memory-utilization from the discussion of the JVM's
 performance characteristics.  This has always puzzled me about the
 Java world and the C++ vs Java debate in general.

 Simply stated, given that it stated clearly in this group that Clojure
 runs, on average, slower than the JVM, is the JVM's high memory-
 utilization a non-issue for the Clojure community as well?  I will
 explain the most recent source of my concern below.

 (I can hear people asking themselves now, What memory-utilization
 issues?)

 As part of my investigation, I spent a lot of time w/ Rich Hickey's
 ant simulation, the one that he demonstrates in his video
 presentations.  I ran it on Linux and Windows.  I tweaked the
 parameters.  I scrutinized the code.  I observed it, both the GUI and
 in top (on Linux) and pmap and strace, etc.

 The problem was that it was not as fast as I expected it should be
 given that it was using no less than 100% of the CPU on my system.
 (two 3GHz Xeon CPUs [1 core]; 3GB RAM; a beefy workstation).  That
 this was visible in the GUI shows how slow it appeared to me.  Also,
 it was using 700MB RAM (VIRT in top).  Sure - it was swapped (I'm
 familiar w/ some of the interpretations of these memory issues) except
 that my system has ZERO swap space.  PMAP showed that 400MB of it was
 heap, too, not libraries or binaries or anything else that we can
 safely ignore.  This was (apparently) real, allocated heap, private to
 the process's address space.

 Additionally, as the simulation ran, the initial RSS of 60MB rose to
 130MB then stopped.  The VIRT remained constant.  I had expected that
 - however I remained concerned.

 That was w/ the OpenJDK v6.  So, on a whim, I installed a v5 JVM and
 reran all everything and played w/ it all over again - just to see.

 Observations:  it was slower - visibly.  The JVM allocated less memory
 - 50% less.  However, at 700MB, 50% reduction is very disappointing
 for a simple simulation.  top showed VIRT to be 270MB which rose to
 300MB then remained constant; that differed from the v6 JVM.  RSS
 started at 45MB and rose to 70MB then remained constant.

 We all know the trade-off of time (CPU) for