date:20121210


On Dec 8, 2012, at 6:37 PM, Charles Comstock wrote:

 I still encounter some sort of issue where it appears that the documentation 
 querying functions, find-doc, and doc and the like are not being properly 
 brought into the repl namespace, which breaks ctrl-d d until I manually bring 
 that into the namespace. I'm not quite certain what was causing that problem 
 and have yet to find a permanent fix.

Are you using a recent Clojure 1.5.0 beta build?  A change went in recently to 
fix some behaviour where nREPL would inadvertently refer all of the REPL 
utilities (doc, find-doc, pp, etc.) into *every* namespace.  When used with 
Clojure 1.5.0, nREPL now only refers those vars into the user ns, matching the 
default Clojure REPL defaults.

(I actually got used to the old behaviour myself, but it can cause serious 
problems, e.g.: http://code.google.com/p/counterclockwise/issues/detail?id=443)

This all said, it's definitely the case that some piece of the toolchain 
(probably clients like reply and nrepl.el, based on project.clj / profile 
config) should instigate global namespace refers so that these vars, or some 
subset, in addition to whatever vars you use most often in your 
workflow/project/application are always available.  Meditation on this topic 
continues. :-)

Cheers,

- Chas

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Clojure raytracing scene description language

2012-12-10 Thread Karsten Schmidt

Hi Mike, you could maybe take some inspiration from Structure Synth, which
is based on CFDG (http://www.contextfreeart.org/)...

http://structuresynth.sourceforge.net/learn.php

Apart from that, wouldn't it be more worthwhile to just focus on the actual
scenegraph generation and scene export in different flavours instead of
writing a complete new renderer? There are many great open source renderers
apart from the slightly aged PovRay which could be (more) interesting to
work with, for example:

http://luxrender.net (OpenCL, multiple light models/samplers/lenses,
physically correct procedural textures, network rendering etc)

http://sunflow.sf.net (java, custom shades, could be fully wrapped)

Just thinking out loud...

Best, K.
On 10 Dec 2012 09:36, Thomas th.vanderv...@gmail.com wrote:

 A very cool idea, something I was thinking about myself as well (but never
 had the time/knowledge for).

 Are you planning to make use of multithreading? (there is a another
 Clojure Raytraces on githubm and that one doesn't unfortunately).

 And sticking close to the pov syntax would make sense I think, as it would
 make porting code easier.

 Thomas

 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: STM - a request for war stories

The very fact that there has been no reply to this for five days may mean 
something. I can personally attest to STM being very difficult to put to 
real-life use because there is always that one thing you absolutely need 
for your problem, that is mutable and not transactional. Most of the time 
it will have to do with an existing Java library, JDK not excluded. The 
property of STM that it is an all-or-nothing commitment has been a 
show-stopper for me every time I tried to use it.

My guess is, if your task is something purely computational and amenable to 
massive parallelization, you may have a go with STM; if it's just about 
business logic accessible concurrently by many clients, you won't find it 
workable.

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: abysmal multicore performance, especially on AMD processors

The main GC feature here are the Thread-Local Allocation Buffers. They are 
on by default and are automatically sized according to allocation 
patterns. The size can also be fine-tuned with the -XX:TLABSize=nconfiguration 
option. You may consider tweaking this setting to optimize 
runtime. Basically, everything that one call to your function allocates 
should fit into a TLAB because it is all garbage upon exit. Allocation 
inside TLAB is ultra-fast and completely concurrent.

Configure 
TLABhttp://docs.oracle.com/javase/7/docs/technotes/tools/windows/java.html

On Sunday, December 9, 2012 7:37:09 PM UTC+1, Andy Fingerhut wrote:


 On Dec 9, 2012, at 6:25 AM, Softaddicts wrote: 

  If the number of object allocation mentioned earlier in this thread are 
 real, 
  yes vm heap management can be a bottleneck. There has to be some 
  locking done somewhere otherwise the heap would corrupt :) 
  
  The other bottleneck can come from garbage collection which has to 
 freeze 
  object allocation completely or partially. 
  
  This internal process has to reclaim unreferenced objects otherwise you 
 may end up 
  exhausting the heap. That can even susoend your app while gc is running 
 depending 
  on the strategy used. 

 Agreed that memory allocation and garbage collection will in some cases 
 need to coordinate between threads to work in the general case of arbitrary 
 allocations and GCs. 

 However, one could have a central list of large pages of free memory 
 (e.g. a few MBytes, or maybe even larger), and pass these out to concurrent 
 memory allocators in these large chunks, and let them do small object 
 allocations and GC within each thread completely concurrently. 

 The only times locking of any kind might be needed with such a strategy 
 would be when one of the parallel threads requests a new big page from the 
 central free list, or returned a completely empty free page back to the 
 central list that it didn't need any more.  All other memory allocation and 
 GC could be completely concurrent.  The idea of making those pages large 
 is that such passing pages around would be infrequent, and thus could be 
 made to have no measurable synchronization overhead. 

 That is pretty much what is happening when you run Lee's benchmark 
 programs as 1 thread per JVM, but 4 or 8 different JVMs processes running 
 in parallel.  In that situation the OS has the central free list of pages, 
 and the JVMs manage their small object allocations and GCs completely 
 concurrently without interfering with each other. 

 If HotSpot's JVM could be configured to work like that, he would be seeing 
 big speedups in a single JVM. 

 Andy 



-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Clojure raytracing scene description language

I'd like to keep it close to POVRay if possible. Although I think there is 
a good opportunity to clojurize the POVRay syntax and also iron out a few 
rough edges. 

Also, I'm hoping to the automatic generation of scenes much more powerful. 
The POVRay macro language is OK for what it is designed for, but it's a 
long way from Lisp.

On Monday, 10 December 2012 13:54:23 UTC+8, Grant Rettke wrote:

 On Sun, Dec 9, 2012 at 6:03 PM, Mikera mike.r.an...@gmail.comjavascript: 
 wrote: 
  Any thoughts / ideas / feedback? 

 What if you try keeping it pretty close to the current POV syntax/format? 


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: abysmal multicore performance, especially on AMD processors

2012-12-10 Thread Marshall Bockrath-Vandegrift

cameron cdor...@gmail.com writes:

 There does seem to be something unusual about conj and
 clojure.lang.PersistentList in this parallel test case and I don't
 think it's related to the JVMs memory allocation.

I’ve got a few more data-points, but still no handle on what exactly is
going on.

My last benchmark showing the `conj*` speedup for `Cons` objects
degrading as soon as it was used on a `PersistantList` was incomplete.
In fact, the speedup degrades after it is used on objects of more than
one type.  The effect just appears immediately when used with
`PersistantList` because '() is in fact a different a
`PersistantList$EmptyList`.  Using `conj*` first in vector
implementation then results in the same inverse speedup on `Cons`s.

Even without your near-optimal speedup using Java standard library
types, I think your earlier benchmarks are enough to demonstrate that
this isn’t an issue with allocation alone.  All of the implementations
based on `reduce` with `conj` must allocate and return a new object for
each iteration.  If parallel allocation were the sole issue, I’d expect
all of the implementations to demonstrate the same behavior.

Unfortunately I have no idea what to connect from these facts:

  - Parallel allocation of `Cons` and `PersistentList` instances through
a Clojure `conj` function remains fast as long as the function only
ever returns objects of a single concrete type

  - Parallel allocation speed for `PersistentVector` instances is
unaffected by `conj` returning multiple types, and does not
demonstrate the inverse speedup seen for the previous types.

At this point I believe the symptoms point to cache contention, but I
don’t know where or why.  Using OpenJDK 7 with -XX:+UseCondMark didn’t
appear to produce any improvement.  Creating a private copy of
`PersistentList` which contained additional padding fields likewise
didn’t appear to produce any improvement.

So, Lee Spector: I think it’s possible to work around this though by
just not using `conj` on lists.  It’s suboptimal, but at least solves
the problem in your original benchmark.  Further improvements are
obviously possible, but that’s a start.

-Marshall

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Clojure raytracing scene description language

Thanks for some great links! 

I want to build a custom renderer mainly because I like an interesting 
challenge :-) Also I've written a couple of renderers before and I reckon 
it's a fairly manageable task.

I also think that to make a really good Clojure DSL it is important to 
design the underlying primitives / data structures in a way that aligns 
well with the DSL. So it might be difficult to retrofit a Clojure DSL on 
top of an existing rendering system without making some ugly compromises. 
Will have to see how that works out, and certainly I'll see what we can 
leverage from the linked code bases - Sunflow stuff is most promising since 
it is pure Java and has a compatible license (I'm thinking LGPL for 
enlight, so that people can embed it as a library more easily)


On Monday, 10 December 2012 18:37:38 UTC+8, Karsten Schmidt wrote:

 Hi Mike, you could maybe take some inspiration from Structure Synth, which 
 is based on CFDG (http://www.contextfreeart.org/)...

 http://structuresynth.sourceforge.net/learn.php

 Apart from that, wouldn't it be more worthwhile to just focus on the 
 actual scenegraph generation and scene export in different flavours instead 
 of writing a complete new renderer? There are many great open source 
 renderers apart from the slightly aged PovRay which could be (more) 
 interesting to work with, for example:

 http://luxrender.net (OpenCL, multiple light models/samplers/lenses, 
 physically correct procedural textures, network rendering etc)

 http://sunflow.sf.net (java, custom shades, could be fully wrapped)

 Just thinking out loud...

 Best, K.
 On 10 Dec 2012 09:36, Thomas th.van...@gmail.com javascript: wrote:

 A very cool idea, something I was thinking about myself as well (but 
 never had the time/knowledge for).

 Are you planning to make use of multithreading? (there is a another 
 Clojure Raytraces on githubm and that one doesn't unfortunately).

 And sticking close to the pov syntax would make sense I think, as it 
 would make porting code easier.

 Thomas

 -- 
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clo...@googlegroups.comjavascript:
 Note that posts from new members are moderated - please be patient with 
 your first post.
 To unsubscribe from this group, send email to
 clojure+u...@googlegroups.com javascript:
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en



-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Clojure raytracing scene description language

Yep, I'll probably use the trick I used in clisk and chunk the image up 
into different pieces that can be handed off to different processors. Also 
I'm planning to make the scene data structures immutable, so the concurrent 
access should work nicely.

Long term goal might be to do some distribution across machines (using 
pallet perhaps?), that gets important when you start doing multi-frame 
renders for animations.

Agree about the POV syntax from a porting / translation perspective, I'll 
try to make the syntax as close as I can while clojurizing it...

On Monday, 10 December 2012 17:36:37 UTC+8, Thomas wrote:

 A very cool idea, something I was thinking about myself as well (but never 
 had the time/knowledge for).

 Are you planning to make use of multithreading? (there is a another 
 Clojure Raytraces on githubm and that one doesn't unfortunately).

 And sticking close to the pov syntax would make sense I think, as it would 
 make porting code easier.

 Thomas


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Clojure raytracing scene description language

Hi Brandon,
I'm thinking to allow for animations via two main methods:
a) A time parameter that can be inserted in formulae to allow variation 
over frames. Good for simple animation.
b) Allowing procedural generation of multiple frames from regular Clojure 
code. So you could theoretically run a fairly complex simulation while 
doing this generation.
An open question is how much you can benefit from structural sharing of 
unchanging components across successive frames. My suspicion is that the 
answer is quite a lot but this needs some more thought.

On Monday, 10 December 2012 11:46:00 UTC+8, Brandon Bloom wrote:

 Are you only interested in static scenes? Or are you interested in 
 simulations too?

 On Sunday, December 9, 2012 4:03:36 PM UTC-8, Mikera wrote:

 Hi all,

 I'm working on a hobby project to implement a Clojure raytracer, 
 something along the lines of POV-Ray:
 https://github.com/mikera/enlight

 It's a a fairly preliminary stage right now, but I'm interested in ideas 
 on how to create the scene description language. Roughly the objectives are:
 - Allow an intuitive, declarative definition of a 3D scene
 - Allow parts of the scene to be generated programatically (e.g. randomly 
 duplicating objects!)
 - Allow mathematical functions and textures to be expressed (probably 
 using clisk - https://github.com/mikera/clisk)
 - Enable the scene to be compiled down to an optimised scene graph for 
 rendering
 - Be reasonably concise as a DSL

 I'm thinking of something like:

 [
 [:camera :position [0 0 -10] :look-at [0 0 0]]   ;; a camera for the 
 scene
 [:sphere :radius 1 :translate [0 2 0] :colour red]  ;;  a translated red 
 shere
 [:box [0 0 0] [1 1 1] :colour (function [1 x y])]   ;; a box with a 
 procedural colour function
 (for [i [2 3 4]] [:box [i 1 1] [(inc i) 2 2]])  ;; generate multiple 
 boxes with a parameterised position
 ]

 Any thoughts / ideas / feedback?



-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: STM - a request for war stories

On Dec 10, 2012, at 5:39 AM, Marko Topolnik wrote:

 The very fact that there has been no reply to this for five days may mean 
 something. I can personally attest to STM being very difficult to put to 
 real-life use because there is always that one thing you absolutely need for 
 your problem, that is mutable and not transactional. Most of the time it will 
 have to do with an existing Java library, JDK not excluded. The property of 
 STM that it is an all-or-nothing commitment has been a show-stopper for me 
 every time I tried to use it.

I'd be surprised if Paul doesn't hear from people directly; people aren't 
always keen to talk about their work publicly (and in many cases, they are 
simply barred from doing so), so one shouldn't presume that on-list responses 
(or not) are representative.

I personally have never used STM in nontrivial ways (AFAIC), but that's due 
more to the demands of the problems I run into more than anything else.  On the 
other hand, I have used, abused, and benefitted from agents in umpteen ways.  
Actually, I have often done things using agents that might otherwise been done 
using STM or other similar approaches, simply to ensure that:

(a) the processing involved can be readily parallelized, and
(b) if necessary, the system can be partitioned/distributed with minimal impact 
to the architecture, since — if you're careful about things — it doesn't matter 
whether a send is evaluated in an in-process agent or one housed in a different 
server/VM/whatever

Yes, only a subset of the things you can do with STM can be done safely with 
agents, etc.  (See: monotonic logic, and the increasingly-popular concepts of 
lattices, semilattices, and CRDTs.)  But, I've been lucky to be able to 
characterize many problems within that subset.

It's true that STM is all or nothing, but it is so over the scope of refs you 
choose.  If there's some side-effecting bit you need to do somewhere, then 
clearly that's not going to fit within a transaction…but that bit will often 
fit just fine in a send-off to an agent provoked _by_ a transaction.  And, if 
you can implement e.g. 2 of the 5 parts of your system using refs and STM, you 
just cut your thread-and-locking problems by 40%. :-P

 My guess is, if your task is something purely computational and amenable to 
 massive parallelization, you may have a go with STM; if it's just about 
 business logic accessible concurrently by many clients, you won't find it 
 workable.

If your task is purely computational and amenable to massive parallelization, 
you _should_ use agents whenever possible.  STM provides for coordination in 
order to enforce consistency; unless all of your operations are commutative (in 
which case, you should probably be using agents anyway), a program using STM 
_will_ provoke retries and other means to route around ref contention.  This is 
acceptable because STM is all about maintaining correctness in the face of 
concurrent mutation, and not necessarily about performance, aggregate 
throughput, and so on.  On the other hand, ref readers are _never_ blocked 
(regardless of what's going on on the write side), so the data in such refs is 
always accessible.  This sounds like an ideal combination for business logic 
(as nebulous a term as that is) to me.

Cheers,

- Chas

--
http://cemerick.com
[Clojure Programming from O'Reilly](http://www.clojurebook.com)

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: STM - a request for war stories



On Monday, December 10, 2012 1:56:08 PM UTC+1, Chas Emerick wrote:

 On Dec 10, 2012, at 5:39 AM, Marko Topolnik wrote: 

 I personally have never used STM in nontrivial ways (AFAIC), but that's 
 due more to the demands of the problems I run into more than anything else. 
  On the other hand, I have used, abused, and benefitted from agents in 
 umpteen ways.  Actually, I have often done things using agents that might 
 otherwise been done using STM or other similar approaches, simply to ensure 
 that: 

 (a) the processing involved can be readily parallelized, and 
 (b) if necessary, the system can be partitioned/distributed with minimal 
 impact to the architecture, since — if you're careful about things — it 
 doesn't matter whether a send is evaluated in an in-process agent or one 
 housed in a different server/VM/whatever 


The argument (b) is an even better fit (or, should we say, *perfect fit*) 
for Actors, as implemented in Erlang.
 

 It's true that STM is all or nothing, but it is so over the scope of 
 refs you choose.  If there's some side-effecting bit you need to do 
 somewhere, then clearly that's not going to fit within a transaction…but 
 that bit will often fit just fine in a send-off to an agent provoked _by_ a 
 transaction.


send-off fails to be useful whenever you need the results within the 
transaction (quite often, that is).
 

  My guess is, if your task is something purely computational and amenable 
 to massive parallelization, you may have a go with STM; if it's just about 
 business logic accessible concurrently by many clients, you won't find it 
 workable. 

 If your task is purely computational and amenable to massive 
 parallelization, you _should_ use agents whenever possible.  STM provides 
 for coordination in order to enforce consistency; unless all of your 
 operations are commutative (in which case, you should probably be using 
 agents anyway), a program using STM _will_ provoke retries and other means 
 to route around ref contention.  This is acceptable because STM is all 
 about maintaining correctness in the face of concurrent mutation, and not 
 necessarily about performance, aggregate throughput, and so on.  


But concurrency is *all* about performance and throughput. So where is the 
benefit of using correct, slow concurrent mutation? I guess in a 
write-seldom, read-often scenario.
 

 On the other hand, ref readers are _never_ blocked (regardless of what's 
 going on on the write side), so the data in such refs is always accessible. 
  This sounds like an ideal combination for business logic (as nebulous a 
 term as that is) to me. 

 
Business logic almost always involves communication with outside systems 
(since it's usually about integration of many existing systems). Even if 
not, a scalable solution must be stateless (a prerequisite for cluster 
deployment) and any durable state must go into a single datasource common 
to all cluster nodes. Again, these datasources don't participate in an STM 
transaction. Maybe this would be a major route of improvement: integrate 
the STM with external datasource transactions. But this is still quite 
removed from the present.

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: STM - a request for war stories

2012-12-10 Thread Paul Butcher

On 10 Dec 2012, at 12:56, Chas Emerick c...@cemerick.com wrote:

 I'd be surprised if Paul doesn't hear from people directly

I wish that that were true, but no, I've not had anyone get in touch off-list.

Many thanks, Marko, for resurrecting the thread - I'm still definitely keen to 
hear of first-hand experiences!

--
paul.butcher-msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: p...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: STM - a request for war stories

2012-12-10 Thread Paul Butcher

On 10 Dec 2012, at 13:37, Marko Topolnik marko.topol...@gmail.com wrote:

 But concurrency is all about performance and throughput. So where is the 
 benefit of using correct, slow concurrent mutation? I guess in a 
 write-seldom, read-often scenario.

I'm not at all sure that that's true. There are plenty of occasions where 
concurrency is about being able to do more than one thing at a time, and not 
necessarily about making something faster.

For example, your mobile 'phone is concurrent because, while it's playing music 
to you, it also wants to notice when you poke the screen and listen for 
incoming calls/messages from the network. And your IDE is concurrent so that it 
can check the syntax of your code in the background while the UI remains 
responsive. 

I'm not, of course, saying that performance isn't important - even in cases 
such as the above. It would be a major problem if everything was an order of 
magnitude slower just because I tried to do two things at the same time. But 
there are certainly plenty of occasions where we might choose to write 
concurrent code without our focus being on performance per-se.

--
paul.butcher-msgCount++

Snetterton, Castle Combe, Cadwell Park...
Who says I have a one track mind?

http://www.paulbutcher.com/
LinkedIn: http://www.linkedin.com/in/paulbutcher
MSN: p...@paulbutcher.com
AIM: paulrabutcher
Skype: paulrabutcher

On 10 Dec 2012, at 13:37, Marko Topolnik marko.topol...@gmail.com wrote:

 
 
 On Monday, December 10, 2012 1:56:08 PM UTC+1, Chas Emerick wrote:
 On Dec 10, 2012, at 5:39 AM, Marko Topolnik wrote: 
 
 I personally have never used STM in nontrivial ways (AFAIC), but that's due 
 more to the demands of the problems I run into more than anything else.  On 
 the other hand, I have used, abused, and benefitted from agents in umpteen 
 ways.  Actually, I have often done things using agents that might otherwise 
 been done using STM or other similar approaches, simply to ensure that: 
 
 (a) the processing involved can be readily parallelized, and 
 (b) if necessary, the system can be partitioned/distributed with minimal 
 impact to the architecture, since — if you're careful about things — it 
 doesn't matter whether a send is evaluated in an in-process agent or one 
 housed in a different server/VM/whatever 
 
 The argument (b) is an even better fit (or, should we say, perfect fit) for 
 Actors, as implemented in Erlang.
  
 It's true that STM is all or nothing, but it is so over the scope of refs 
 you choose.  If there's some side-effecting bit you need to do somewhere, 
 then clearly that's not going to fit within a transaction…but that bit will 
 often fit just fine in a send-off to an agent provoked _by_ a transaction.
 
 send-off fails to be useful whenever you need the results within the 
 transaction (quite often, that is).
  
  My guess is, if your task is something purely computational and amenable to 
  massive parallelization, you may have a go with STM; if it's just about 
  business logic accessible concurrently by many clients, you won't find it 
  workable. 
 
 If your task is purely computational and amenable to massive parallelization, 
 you _should_ use agents whenever possible.  STM provides for coordination in 
 order to enforce consistency; unless all of your operations are commutative 
 (in which case, you should probably be using agents anyway), a program using 
 STM _will_ provoke retries and other means to route around ref contention.  
 This is acceptable because STM is all about maintaining correctness in the 
 face of concurrent mutation, and not necessarily about performance, aggregate 
 throughput, and so on.  
 
 But concurrency is all about performance and throughput. So where is the 
 benefit of using correct, slow concurrent mutation? I guess in a 
 write-seldom, read-often scenario.
  
 On the other hand, ref readers are _never_ blocked (regardless of what's 
 going on on the write side), so the data in such refs is always accessible.  
 This sounds like an ideal combination for business logic (as nebulous a 
 term as that is) to me. 
  
 Business logic almost always involves communication with outside systems 
 (since it's usually about integration of many existing systems). Even if not, 
 a scalable solution must be stateless (a prerequisite for cluster deployment) 
 and any durable state must go into a single datasource common to all cluster 
 nodes. Again, these datasources don't participate in an STM transaction. 
 Maybe this would be a major route of improvement: integrate the STM with 
 external datasource transactions. But this is still quite removed from the 
 present.
 
 -- 
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with your 
 first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com

Re: STM - a request for war stories

On Dec 10, 2012, at 8:37 AM, Marko Topolnik wrote:

 It's true that STM is all or nothing, but it is so over the scope of refs 
 you choose.  If there's some side-effecting bit you need to do somewhere, 
 then clearly that's not going to fit within a transaction…but that bit will 
 often fit just fine in a send-off to an agent provoked _by_ a transaction.
 
 send-off fails to be useful whenever you need the results within the 
 transaction (quite often, that is).

I'm not aware of any system that provides transactional semantics in the face 
of in-transaction side-effecting actions.  If you can refer me to any, that'd 
be great.

  My guess is, if your task is something purely computational and amenable to 
  massive parallelization, you may have a go with STM; if it's just about 
  business logic accessible concurrently by many clients, you won't find it 
  workable. 
 
 If your task is purely computational and amenable to massive parallelization, 
 you _should_ use agents whenever possible.  STM provides for coordination in 
 order to enforce consistency; unless all of your operations are commutative 
 (in which case, you should probably be using agents anyway), a program using 
 STM _will_ provoke retries and other means to route around ref contention.  
 This is acceptable because STM is all about maintaining correctness in the 
 face of concurrent mutation, and not necessarily about performance, aggregate 
 throughput, and so on.  
 
 But concurrency is all about performance and throughput. So where is the 
 benefit of using correct, slow concurrent mutation? I guess in a 
 write-seldom, read-often scenario.

Fundamentally, concurrency is about simultaneous independent computation.  
Depending on the domain and computations involved, single-thread performance 
and aggregate throughput can vary significantly.

Anyway, read-heavy applications are still the norm in most industrial settings, 
despite the rise in popularity of write-scalable architectures.

 On the other hand, ref readers are _never_ blocked (regardless of what's 
 going on on the write side), so the data in such refs is always accessible.  
 This sounds like an ideal combination for business logic (as nebulous a 
 term as that is) to me. 
  
 Business logic almost always involves communication with outside systems 
 (since it's usually about integration of many existing systems). Even if not, 
 a scalable solution must be stateless (a prerequisite for cluster deployment) 
 and any durable state must go into a single datasource common to all cluster 
 nodes. Again, these datasources don't participate in an STM transaction. 
 Maybe this would be a major route of improvement: integrate the STM with 
 external datasource transactions. But this is still quite removed from the 
 present.

I'm certain that particular set of requirements holds in certain settings, but 
they are hardly universal.

If I may make a tenuous inference, it sounds like you're trying to fit every 
state transition within an application into a transaction.  If so, I'd 
recommend the opposite: decomposing applications and their processes into 
modular bags of state and treating them separately will lead to big wins — 
including potentially being able to use e.g. STM in one place, and agents in 
another, each interacting with the other as necessary.

Re: getting disparate datasources to participate in transactions, you might 
want to take a look at Avout:

http://avout.io  

I can't say I've used it, but it is at least an existence proof of the ability 
of the Clojure STM model to be distributable.

Cheers,

- Chas

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: what Jetty jars do I need for WebSockets?

2012-12-10 Thread larry google groups

Thank you. I will do that.

I find that trying to learn both Clojure and the JVM (which automatically
entails parts of the Java eco-system), is a little overwhelming at first.
But I suppose that is true of learning anything new.

On Sunday, December 9, 2012 9:04:44 PM UTC-5, Jay Fields wrote:

I don't have the answer, but I would strongly recommend webbit:
https://github.com/webbit/webbit

I've been using it for quite awhile and I've been very happy with it.

On Sun, Dec 9, 2012 at 8:55 PM, larry google groups
lawrenc...@gmail.com javascript: wrote:

I am still fairly new to Clojure, the JVM and Java, so I get lost trying
to
read some of the stuff that assumes knowledge of any of those 3. I want
to
build a Clojure app using Jetty and offering WebSocket connections.

I have already built an app with Clojure and Jetty, so that part is
easy.
But I look here:

http://wiki.eclipse.org/index.php?title=Jetty/Feature/WebSocketsoldid=297254

and it says I should download 3 jars:

wget -O jetty-all.jar --user-agent=demo \

http://repo2.maven.org/maven2/org/eclipse/jetty/aggregate/jetty-all/7.6.2.v20120308/jetty-all-7.6.2.v20120308.jar

wget -O jetty-websocket-tests.jar --user-agent=demo \

http://repo2.maven.org/maven2/org/eclipse/jetty/jetty-websocket/7.6.2.v20120308/jetty-websocket-7.6.2.v20120308-tests.jar

wget --user-agent=demo \

http://repo2.maven.org/maven2/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar

I assume I put these in project.clj? I already have stuff in there like:

:dependencies [[org.clojure/clojure 1.3.0]
[net.cgrand/moustache 1.1.0]
[ring 1.1.5]
[ring/ring-jetty-adapter 1.1.5]

etc

Searching on this topic brought me to this:

https://groups.google.com/forum/?fromgroups=#!topic/ring-clojure/JD9FLJFTVsg

But that is 2 years old. I have not found anything specific to
ring/jetty/websockets that is recent.

Can anyone point me to documentation that might lead me out of my
confusion?

--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.comjavascript:
Note that posts from new members are moderated - please be patient with
your
first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com javascript:
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: what Jetty jars do I need for WebSockets?

2012-12-10 Thread Jay Fields

The upside of using Java is that it's very widely documented. Also, I
find people on this mailing list to be very helpful. Nonetheless, I'm
sure it's frustrating to have to learn about Java when you just want
to do some Clojure.

I've previously written about using Webbit with Clojure:
http://blog.jayfields.com/2011/02/clojure-web-socket-introduction.html

afaik, the post should be up to date and still working. Feel free to
email me if you run into any issues.

On Mon, Dec 10, 2012 at 8:25 AM, larry google groups
lawrencecloj...@gmail.com wrote:

Thank you. I will do that.

On Sunday, December 9, 2012 9:04:44 PM UTC-5, Jay Fields wrote:

I don't have the answer, but I would strongly recommend webbit:
https://github.com/webbit/webbit

I've been using it for quite awhile and I've been very happy with it.

On Sun, Dec 9, 2012 at 8:55 PM, larry google groups
lawrenc...@gmail.com wrote:

I have already built an app with Clojure and Jetty, so that part is
easy.
But I look here:

http://wiki.eclipse.org/index.php?title=Jetty/Feature/WebSocketsoldid=297254

and it says I should download 3 jars:

wget -O jetty-all.jar --user-agent=demo \

http://repo2.maven.org/maven2/org/eclipse/jetty/aggregate/jetty-all/7.6.2.v20120308/jetty-all-7.6.2.v20120308.jar
wget -O jetty-websocket-tests.jar --user-agent=demo \

http://repo2.maven.org/maven2/org/eclipse/jetty/jetty-websocket/7.6.2.v20120308/jetty-websocket-7.6.2.v20120308-tests.jar
wget --user-agent=demo \

http://repo2.maven.org/maven2/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar

I assume I put these in project.clj? I already have stuff in there like:

:dependencies [[org.clojure/clojure 1.3.0]
[net.cgrand/moustache 1.1.0]
[ring 1.1.5]
[ring/ring-jetty-adapter 1.1.5]

etc

Searching on this topic brought me to this:

https://groups.google.com/forum/?fromgroups=#!topic/ring-clojure/JD9FLJFTVsg

But that is 2 years old. I have not found anything specific to
ring/jetty/websockets that is recent.

Can anyone point me to documentation that might lead me out of my
confusion?

--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with
your
first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: STM - a request for war stories



On Monday, December 10, 2012 3:15:04 PM UTC+1, Paul Butcher wrote:

 On 10 Dec 2012, at 13:37, Marko Topolnik marko.t...@gmail.comjavascript: 
 wrote:

 But concurrency is *all* about performance and throughput. So where is 
 the benefit of using correct, slow concurrent mutation? I guess in a 
 write-seldom, read-often scenario.


 I'm not at all sure that that's true. There are plenty of occasions where 
 concurrency is about being able to do more than one thing at a time, and 
 not necessarily about making something faster.

 For example, your mobile 'phone is concurrent because, while it's playing 
 music to you, it also wants to notice when you poke the screen and listen 
 for incoming calls/messages from the network. And your IDE is concurrent so 
 that it can check the syntax of your code in the background while the UI 
 remains responsive. 


My thinking always assumes the existence---and prevalence---of lock-based 
concurrency. Problems with non-critical performance are usually not too 
hard to do with locks: just use some simple, coarse locking scheme.  I'd 
need a quite convincing case where locks are an obvious disaster and an 
STM-based approach comes to rescue. These are hard to find, and personally 
I have tried several times to start out with STM, only to end up falling 
back to locks. 

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: STM - a request for war stories



On Monday, December 10, 2012 3:17:27 PM UTC+1, Chas Emerick wrote:

 On Dec 10, 2012, at 8:37 AM, Marko Topolnik wrote:

 It's true that STM is all or nothing, but it is so over the scope of 
 refs you choose.  If there's some side-effecting bit you need to do 
 somewhere, then clearly that's not going to fit within a transaction…but 
 that bit will often fit just fine in a send-off to an agent provoked _by_ a 
 transaction.


 send-off fails to be useful whenever you need the results within the 
 transaction (quite often, that is).


 I'm not aware of any system that provides transactional semantics in the 
 face of in-transaction side-effecting actions.  If you can refer me to any, 
 that'd be great.


I am comparing this with a mutex-based solution, which is still the default 
way to implement thread safety. Obviously, no problems with side effects 
there.
 


 But concurrency is *all* about performance and throughput. So where is 
 the benefit of using correct, slow concurrent mutation? I guess in a 
 write-seldom, read-often scenario.


 Fundamentally, concurrency is about simultaneous independent computation. 
  Depending on the domain and computations involved, single-thread 
 performance and aggregate throughput can vary significantly.

 Anyway, read-heavy applications are still the norm in most industrial 
 settings, despite the rise in popularity of write-scalable architectures.


So again, I would like to see the benefit of an STM over a lock-based 
solution. Read-heavy scenarios behave well with read/write locks and if 
there's not much writing around, it's usually not too complex to be kept 
under control with locks. So an STM-based solution would have to offer a) 
less complexity due to no locks and b) not incur its own complexity while 
dealing with side effects.
 



 Business logic almost always involves communication with outside systems 
 (since it's usually about integration of many existing systems). Even if 
 not, a scalable solution must be stateless (a prerequisite for cluster 
 deployment) and any durable state must go into a single datasource common 
 to all cluster nodes. Again, these datasources don't participate in an STM 
 transaction. Maybe this would be a major route of improvement: integrate 
 the STM with external datasource transactions. But this is still quite 
 removed from the present.


 I'm certain that particular set of requirements holds in certain settings, 
 but they are hardly universal.

 If I may make a tenuous inference, it sounds like you're trying to fit 
 every state transition within an application into a transaction.  If so, 
 I'd recommend the opposite: decomposing applications and their processes 
 into modular bags of state and treating them separately will lead to big 
 wins — including potentially being able to use e.g. STM in one place, and 
 agents in another, each interacting with the other as necessary.


Usually you have a unit of work to complete. If any part of it involves 
side effects, you'll need a mutex around it, and at that point the STM 
brings nothing. Another frequent problem is having any kind of time-heavy 
action, which you must make sure to execute only once (even if it is 
retryable by nature). Every new problem I start to work with, I first think 
long and hard how STM could fit into the picture; I have failed every time. 
Mind that my company is an early Clojure adopter (we remember when 
#clojure channel had only 6 people in it :)
 


 Re: getting disparate datasources to participate in transactions, you 
 might want to take a look at Avout:

 http://avout.io  

 I can't say I've used it, but it is at least an existence proof of the 
 ability of the Clojure STM model to be distributable.


I'll definitely check it out, maybe it gives me good ideas for the future. 
Thanks!
 

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Combining maps and finite domains in core.logic returns only one result

2012-12-10 Thread Frederik De Bleser

Hey,

I'm trying to combine maps with finite domains with some odd results.

A simple query using finite domains correctly returns all values:

(run* [q]
  (fresh [x]
(infd x (interval 1 3))
(== q x)))
 ;= (1 2 3)

But putting this result in a map returns only the first value:

(run* [q]
  (fresh [x]
(infd x (interval 1 3))
(== q {:foo x})))
 ;= ({:foo 1})
Am I missing something? I'm using core.logic 0.8.0-beta2 on clojure 1.4.0.

Kind regards,

Frederik

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Combining maps and finite domains in core.logic returns only one result

2012-12-10 Thread Frederik De Bleser

FYI this works with vectors:

(run* [q]
  (fresh [x]
(infd x (interval 1 3))
(== q [x])))
 ;= ([1] [2] [3])
But lcons seems to fail as well:

(run* [q]
  (fresh [x]
(infd x (interval 1 3))
(== q (lcons x 'foo
 ;= ((1 . foo))


Kind regards,

Frederik

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: ClojureScript Browser REPL Unable To Find goog.async.Deferred

2012-12-10 Thread Brent Millare

This thread may be some help
https://groups.google.com/forum/?fromgroups=#!searchin/clojure/brent/clojure/DmnPwrVvfW8/qgnp6MTVWusJ



On Sunday, December 9, 2012 11:53:16 AM UTC-5, Asim Jalis wrote:

 Hi,

 I am trying to follow the Quick Start steps to get a ClojureScript REPL in 
 a browser. Except when I try this I get errors in the REPL (see REPL 
 ERRORS). I am appending steps I am using (see QUICK START STEPS).

 Any ideas on what might be the problem and how to fix it?

 Thanks!

 Asim


 REPL ERRORS

 user= (repl/repl env) ;; start the REPL
 Type:  :cljs/quit  to quit
 ClojureScript:cljs.user Dec 9, 2012 8:36:53 AM 
 com.google.javascript.jscomp.LoggerErrorManager println
 SEVERE: 
 file:/Users/asimjalis/Proj/008-clojurescript/clojurescript/lib/goog.jar!/goog/net/xpc/crosspagechannel.js:26:
  
 ERROR - required goog.async.Deferred namespace never provided
 goog.require('goog.async.Deferred');
 ^

 Dec 9, 2012 8:36:53 AM com.google.javascript.jscomp.LoggerErrorManager 
 println
 SEVERE: 
 file:/Users/asimjalis/Proj/008-clojurescript/clojurescript/lib/goog.jar!/goog/net/xpc/nativemessagingtransport.js:26:
  
 ERROR - required goog.async.Deferred namespace never provided
 goog.require('goog.async.Deferred');
 ^

 Dec 9, 2012 8:36:53 AM com.google.javascript.jscomp.LoggerErrorManager 
 printSummary
 WARNING: 2 error(s), 0 warning(s)
 ERROR: JSC_MISSING_PROVIDE_ERROR. required goog.async.Deferred namespace 
 never provided at 
 file:/Users/asimjalis/Proj/008-clojurescript/clojurescript/lib/goog.jar!/goog/net/xpc/crosspagechannel.js
  
 line 26 : 0
 ERROR: JSC_MISSING_PROVIDE_ERROR. required goog.async.Deferred namespace 
 never provided at 
 file:/Users/asimjalis/Proj/008-clojurescript/clojurescript/lib/goog.jar!/goog/net/xpc/nativemessagingtransport.js
  
 line 26 : 0

 QUICK START STEPS 

 git clone git://github.com/clojure/clojurescript.git
 cd clojurescript
 ./script/bootstrap
 export CLOJURESCRIPT_HOME=$PWD

 cat  CLJS_END  foo.cljs
 (ns foo
   (:require [clojure.browser.repl :as repl]))
 (repl/connect http://localhost:9000/repl;)
 CLJS_END

 ./bin/cljsc foo.cljs  foo.js

 cat  HTML_END  index.html
 html
   head
 meta charset=UTF-8
 titleBrowser-connected REPL/title
   /head
   body
 div id=content
   script type=text/javascript src=out/goog/base.js/script
   script type=text/javascript src=foo.js/script
   script type=text/javascript
 goog.require('foo');
   /script
 /div
   /body
 /html
 HTML_END

 cat  REPL_PASTE_END;
 ;; Start REPL below and paste this into it
 (require '[cljs.repl :as repl])
 (require '[cljs.repl.browser :as browser])  ;; require the browser 
 implementation of IJavaScriptEnv
 (def env (browser/repl-env)) ;; create a new environment
 (repl/repl env) ;; start the REPL
 REPL_PASTE_END

 ./script/repl


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Clojure contrib datalog

2012-12-10 Thread Bronsa

There's also https://github.com/fogus/bacwn

2012/12/10 Alexander Solovyov alexan...@solovyov.net

 Hi,

 I don't think it's maintained somewhere (at least I haven't seen
 anything), but at some point in past I extracted it from sources of
 clojure-contrib and put it on github (with few updates to code, nothing
 major):

 https://github.com/piranha/datalog


 On Sun, Dec 9, 2012 at 1:15 PM, Shantanu Kumar 
 kumar.shant...@gmail.comwrote:

 Hi,

 I saw clojure-contrib datalog has not made it into modular contribs:

 https://github.com/clojure/clojure-contrib/tree/master/modules/datalog
 http://dev.clojure.org/display/design/Where+Did+Clojure.Contrib+Go
 http://dev.clojure.org/display/doc/Clojure+Contrib+Libraries

 Does anybody know if it's being maintained somewhere? Datomic supports
 datalog, so I wrongly presumed otherwise earlier.

 Shantanu

 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en




 --
 Alexander

 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: abysmal multicore performance, especially on AMD processors

2012-12-10 Thread meteorfox



  - Parallel allocation of `Cons` and `PersistentList` instances through 
 a Clojure `conj` function remains fast as long as the function only 
 ever returns objects of a single concrete type 


A possible explanation for this could be JIT Deoptimization. Deoptimization 
happens when the JIT optimized a hot code path based
on what has learned from the executing program, but no longer holds, then 
the compiler notices it has made an incorrect decision in
the previous optimization, and performs the deoptimization.

It might be the case that after a deoptimization, the JIT compiler 
reoptimizes a code path the was previously optimized. This frequently 
happens
when dealing different implementations of an interface, and/or 
inheritance and type hierarchy due to the code path execution changing 
constantly and
having to go through multiple optimizations and deoptimizations.

To identify whether deoptimization is happening, you can add the following 
flag to jvm args -XX:+PrintCompilation 
If the output contains prints made not entrant then that indicates a 
deoptimization, and that method will be interpreted until
a certain threshold of executions is surpassed, and get optimized again.


Reference:
Robust Java Benchmarking, 
http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html#do
Java Performance, Charlie Hunt and  Binu John

On Monday, December 10, 2012 6:17:47 AM UTC-5, Marshall Bockrath-Vandegrift 
wrote:

 cameron cdo...@gmail.com javascript: writes: 

  There does seem to be something unusual about conj and 
  clojure.lang.PersistentList in this parallel test case and I don't 
  think it's related to the JVMs memory allocation. 

 I’ve got a few more data-points, but still no handle on what exactly is 
 going on. 

 My last benchmark showing the `conj*` speedup for `Cons` objects 
 degrading as soon as it was used on a `PersistantList` was incomplete. 
 In fact, the speedup degrades after it is used on objects of more than 
 one type.  The effect just appears immediately when used with 
 `PersistantList` because '() is in fact a different a 
 `PersistantList$EmptyList`.  Using `conj*` first in vector 
 implementation then results in the same inverse speedup on `Cons`s. 

 Even without your near-optimal speedup using Java standard library 
 types, I think your earlier benchmarks are enough to demonstrate that 
 this isn’t an issue with allocation alone.  All of the implementations 
 based on `reduce` with `conj` must allocate and return a new object for 
 each iteration.  If parallel allocation were the sole issue, I’d expect 
 all of the implementations to demonstrate the same behavior. 

 Unfortunately I have no idea what to connect from these facts: 

   - Parallel allocation of `Cons` and `PersistentList` instances through 
 a Clojure `conj` function remains fast as long as the function only 
 ever returns objects of a single concrete type 

   - Parallel allocation speed for `PersistentVector` instances is 
 unaffected by `conj` returning multiple types, and does not 
 demonstrate the inverse speedup seen for the previous types. 

 At this point I believe the symptoms point to cache contention, but I 
 don’t know where or why.  Using OpenJDK 7 with -XX:+UseCondMark didn’t 
 appear to produce any improvement.  Creating a private copy of 
 `PersistentList` which contained additional padding fields likewise 
 didn’t appear to produce any improvement. 

 So, Lee Spector: I think it’s possible to work around this though by 
 just not using `conj` on lists.  It’s suboptimal, but at least solves 
 the problem in your original benchmark.  Further improvements are 
 obviously possible, but that’s a start. 

 -Marshall 



-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: class loaders stack constant grow in REPL

2012-12-10 Thread Colin Jones

Right, this is because nREPL uses clojure.main/repl each time it does an 
evaluation. See http://dev.clojure.org/jira/browse/NREPL-31 for a related 
issue that was addressed by modifying clojure.main/repl.

I'm not sure where a fix for this would belong (nREPL or clojure.main), but 
I went ahead and opened an nREPL JIRA issue to track 
it: http://dev.clojure.org/jira/browse/NREPL-36

- Colin




On Monday, December 10, 2012 2:46:10 AM UTC-6, Vladimir Tsichevski wrote:

 Hi,

 just found that every interaction with Clojure REPL causes one 
 more DynamicClassLoader put on the Thread context class loader chain.

 Here is how clojure.main/repl beginning looks like:

   (let [cl (.getContextClassLoader (Thread/currentThread))]
 (.setContextClassLoader (Thread/currentThread) 
 (clojure.lang.DynamicClassLoader. cl)))

 And this is how to observe it:

 nREPL server started on port 19987
 REPL-y 0.1.0-beta10
 Clojure 1.4.0
 Exit: Control+D or (exit) or (quit)
 Commands: (user/help)
 Docs: (doc function-name-here)
   (find-doc part-of-name-here)
   Source: (source function-name-here)
   (user/sourcery function-name-here)
  Javadoc: (javadoc java-object-or-class-here)
 Examples from clojuredocs.org: [clojuredocs or cdoc]
   (user/clojuredocs name-here)
   (user/clojuredocs ns-here name-here)
 user= (defn print-class-loader-stack
   ([]
  (print-class-loader-stack (.getContextClassLoader 
 (Thread/currentThread
   ([cl]
  (if cl
(do
  (pprint cl)
  (print-class-loader-stack (.getParent cl)))
(println *Top*
   #_=   #_=   #_=   #_=   #_=   #_=   #_=   #_= 
 #'user/print-class-loader-stack
 user= (print-class-loader-stack)
 #DynamicClassLoader clojure.lang.DynamicClassLoader@26a0c73f
 #DynamicClassLoader clojure.lang.DynamicClassLoader@6f603bdc
 #DynamicClassLoader clojure.lang.DynamicClassLoader@2f368c5d
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5b31fd9
 #AppClassLoader sun.misc.Launcher$AppClassLoader@4aad3ba4
 #ExtClassLoader sun.misc.Launcher$ExtClassLoader@3326b249
 *Top*
 nil
 user= 1
 1
 user= 1
 1
 user= 1
 1
 user= 1
 1
 user= (print-class-loader-stack)
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5be04861
 #DynamicClassLoader clojure.lang.DynamicClassLoader@7481933a
 #DynamicClassLoader clojure.lang.DynamicClassLoader@273f212a
 #DynamicClassLoader clojure.lang.DynamicClassLoader@4178feba
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5323961b
 #DynamicClassLoader clojure.lang.DynamicClassLoader@26a0c73f
 #DynamicClassLoader clojure.lang.DynamicClassLoader@6f603bdc
 #DynamicClassLoader clojure.lang.DynamicClassLoader@2f368c5d
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5b31fd9
 #AppClassLoader sun.misc.Launcher$AppClassLoader@4aad3ba4
 #ExtClassLoader sun.misc.Launcher$ExtClassLoader@3326b249
 *Top*
 nil
 user= 


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: abysmal multicore performance, especially on AMD processors

2012-12-10 Thread Wm. Josiah Erikson

Aha. Not only do I get a lot of made not entrant, I get a lot of made
zombie. However, I get this for both runs with map and with pmap (and with
pmapall as well)

 For instance, from a pmapall run:

33752  159 clojure.lang.Cons::next (10 bytes)   made zombie
  33752  164 clojure.lang.RT::conj (21 bytes)   made zombie
  33753  154 clojure.lang.RT::seq (32 bytes)   made zombie
  337535 %   clojure.core$reduce1::invoke @ -2 (184 bytes)
made zombie
  33753  167 clojure.core$conj::invoke (13 bytes)   made zombie
  33753  184 clojure.core$rest::invoke (7 bytes)
  33884  186 clojure.lang.Numbers$LongOps::isPos (15 bytes)
  34298  187 clojure.lang.Numbers::dec (17 bytes)
  34421  188 clojure.lang.Numbers$LongOps::dec (13 bytes)
  34897  189 clojure.core$take::invoke (24 bytes)
  34903  190 clojure.core$take$fn__4227::init (15 bytes)
  35386  191 clojure.core$iterate::invoke (39 bytes)
SNIP

  1168  154 clojure.lang.RT::seq (32 bytes)   made not entrant
   1169  168 clojure.lang.RT::seq (32 bytes)
   11715 %   clojure.core$reduce1::invoke @ -2 (184 bytes)
made not entrant
   1171  169 clojure.core$reduce1::invoke (184 bytes)
   1173  159 clojure.lang.Cons::next (10 bytes)   made not
entrant
   1173  167 clojure.core$conj::invoke (13 bytes)   made not
entrant
   1173  164 clojure.lang.RT::conj (21 bytes)   made not entrant
   1192  170 clojure.lang.PersistentList::first (5 bytes)
   1193  171 clojure.lang.PersistentList::next (18 bytes)
   1193  172 clojure.lang.Cons::next (10 bytes)
   1194  173 clojure.lang.RT::conj (21 bytes)
   1197  174 clojure.core$conj::invoke (13 bytes)
   12336 %   clojure.core$reduce1::invoke @ 0 (184 bytes)

And then, from a map run:

  1163  151 clojure.lang.RT::seq (32 bytes)   made not entrant
   1163  145 clojure.core$cons::invoke (10 bytes)   made not
entrant
   1163  168 clojure.lang.RT::seq (32 bytes)
   11654 %   clojure.core$reduce1::invoke @ 0 (184 bytes)
   1169  144 clojure.lang.RT::cons (46 bytes)   made not entrant
   3467  169 clojure.lang.RT::cons (46 bytes)
   3470   24 clojure.lang.Util::equiv (65 bytes)   made zombie
   3470   23 java.lang.String::equals (88 bytes)   made zombie
   3470   18 clojure.lang.PersistentArrayMap::createWithCheck
(80 bytes)   made zombie
   3470   26 clojure.lang.PersistentArrayMap::equalKey (6
bytes)   made zombie
   3617  170 clojure.core$cons::invoke (10 bytes)
   3622   41 clojure.lang.PersistentArrayMap::indexOf (34
bytes)   made zombie
   3622   30   ! java.net.URL::init (543 bytes)   made zombie
   3622   58 clojure.lang.Symbol::equals (49 bytes)   made
zombie
   3623   73 java.lang.Object::equals (11 bytes)   made zombie
   3623   65 clojure.lang.Util::hasheq (43 bytes)   made zombie
   4249  171  s! clojure.lang.LazySeq::sval (54 bytes)
   4259   77 clojure.lang.Util::equiv (65 bytes)   made zombie
   4259   89 clojure.lang.RT::first (35 bytes)   made zombie
   4259   88 clojure.lang.PersistentHashMap$NodeSeq::create (94
bytes)   made zombie
   4578  172 n   java.lang.Object::getClass (0 bytes)
   5380  173 clojure.lang.AFunction::init (5 bytes)
   5634  174 java.lang.Long::longValue (5 bytes)
   5785  175  s  clojure.lang.LazySeq::seq (53 bytes)
   5830  176 clojure.lang.Numbers::ops (97 bytes)
   6168  177 clojure.lang.LazySeq::init (10 bytes)
   6169   32 java.lang.AbstractStringBuilder::append (48
bytes)   made zombie
  10727  178 java.lang.Long::valueOf (36 bytes)
  10730   37 java.lang.StringBuilder::append (8 bytes)   made
zombie
  10730   49   ! sun.misc.URLClassPath$JarLoader::getResource (91
bytes)   made zombie
  10730   44 sun.misc.URLClassPath::getResource (74 bytes)
made zombie
  11121  179 clojure.lang.Numbers::num (5 bytes)
  11240  180 clojure.core$rest::invoke (7 bytes)
  11240  181 clojure.lang.RT::more (37 bytes)
  11242  144 clojure.lang.RT::cons (46 bytes)   made zombie
  11242  145 clojure.core$cons::invoke (10 bytes)   made zombie
  11242  151 clojure.lang.RT::seq (32 bytes)   made zombie




On Mon, Dec 10, 2012 at 11:21 AM, meteorfox
ctorresk8guitar@gmail.comwrote:

 -XX:+PrintCompilation

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated -

Re: [ANN] Dibble - A new way to seed databases

2012-12-10 Thread Brian Marick

Here's the SCNA talk link

http://scna.softwarecraftsmanship.org/media#videos


On Nov 11, 2012, at 5:28 PM, Michael Drogalis madrush...@gmail.com wrote:

 Interestingly enough, I did something very similar to Peano. The difference 
 is that my version used a base value and constraints are used to adjust 
 that value to something else.
 
 https://github.com/MichaelDrogalis/zombie
 
 In any case, drop the link here when you find the talk. We should take a stab 
 at merging these two concepts.
 
 On Sunday, November 11, 2012 6:23:24 PM UTC-5, Brian Marick wrote:
 
 On Nov 11, 2012, at 2:35 PM, Michael Drogalis wrote: 
  Wouldn't it be great to generate rows in a customer table without having to 
  make up names, email addresses, and balances? That's the idea of Dibble. 
 
 Vaguely related: I wrote up a proof-of-concept of using core.logic to 
 generate structured (hierarchical) test data that satisfies constraints. I 
 could imagine the two code bases being complementary. Mine is at: 
 
 https://github.com/marick/peano 
 
 I gave a talk on the idea at Software Craftsmanship North America 
 (yesterday). I believe it was recorded. 
 
 - 
 Brian Marick, Artisanal Labrador 
 Contract programming in Ruby and Clojure 
 Occasional consulting on Agile 
 Writing /Functional Programming for the Object-Oriented Programmer/: 
 https://leanpub.com/fp-oo 
 
 
 
 -- 
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with your 
 first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en


Occasional consulting on programming technique
Contract programming in Ruby and Clojure
Latest book: /Functional Programming for the Object-Oriented Programmer/
https://leanpub.com/fp-oo

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: class loaders stack constant grow in REPL

2012-12-10 Thread Vladimir Tsichevski

Thank you Colin,

I think, the main problem is nobody has ever tried to write an article 
Class loading in Clojure. If such article existed, it would make life 
much easier for many developers.

Regards,
Vladimir

On Monday, December 10, 2012 8:32:36 PM UTC+4, Colin Jones wrote:

 Right, this is because nREPL uses clojure.main/repl each time it does an 
 evaluation. See http://dev.clojure.org/jira/browse/NREPL-31 for a related 
 issue that was addressed by modifying clojure.main/repl.

 I'm not sure where a fix for this would belong (nREPL or clojure.main), 
 but I went ahead and opened an nREPL JIRA issue to track it: 
 http://dev.clojure.org/jira/browse/NREPL-36

 - Colin




 On Monday, December 10, 2012 2:46:10 AM UTC-6, Vladimir Tsichevski wrote:

 Hi,

 just found that every interaction with Clojure REPL causes one 
 more DynamicClassLoader put on the Thread context class loader chain.

 Here is how clojure.main/repl beginning looks like:

   (let [cl (.getContextClassLoader (Thread/currentThread))]
 (.setContextClassLoader (Thread/currentThread) 
 (clojure.lang.DynamicClassLoader. cl)))

 And this is how to observe it:

 nREPL server started on port 19987
 REPL-y 0.1.0-beta10
 Clojure 1.4.0
 Exit: Control+D or (exit) or (quit)
 Commands: (user/help)
 Docs: (doc function-name-here)
   (find-doc part-of-name-here)
   Source: (source function-name-here)
   (user/sourcery function-name-here)
  Javadoc: (javadoc java-object-or-class-here)
 Examples from clojuredocs.org: [clojuredocs or cdoc]
   (user/clojuredocs name-here)
   (user/clojuredocs ns-here name-here)
 user= (defn print-class-loader-stack
   ([]
  (print-class-loader-stack (.getContextClassLoader 
 (Thread/currentThread
   ([cl]
  (if cl
(do
  (pprint cl)
  (print-class-loader-stack (.getParent cl)))
(println *Top*
   #_=   #_=   #_=   #_=   #_=   #_=   #_=   #_= 
 #'user/print-class-loader-stack
 user= (print-class-loader-stack)
 #DynamicClassLoader clojure.lang.DynamicClassLoader@26a0c73f
 #DynamicClassLoader clojure.lang.DynamicClassLoader@6f603bdc
 #DynamicClassLoader clojure.lang.DynamicClassLoader@2f368c5d
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5b31fd9
 #AppClassLoader sun.misc.Launcher$AppClassLoader@4aad3ba4
 #ExtClassLoader sun.misc.Launcher$ExtClassLoader@3326b249
 *Top*
 nil
 user= 1
 1
 user= 1
 1
 user= 1
 1
 user= 1
 1
 user= (print-class-loader-stack)
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5be04861
 #DynamicClassLoader clojure.lang.DynamicClassLoader@7481933a
 #DynamicClassLoader clojure.lang.DynamicClassLoader@273f212a
 #DynamicClassLoader clojure.lang.DynamicClassLoader@4178feba
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5323961b
 #DynamicClassLoader clojure.lang.DynamicClassLoader@26a0c73f
 #DynamicClassLoader clojure.lang.DynamicClassLoader@6f603bdc
 #DynamicClassLoader clojure.lang.DynamicClassLoader@2f368c5d
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5b31fd9
 #AppClassLoader sun.misc.Launcher$AppClassLoader@4aad3ba4
 #ExtClassLoader sun.misc.Launcher$ExtClassLoader@3326b249
 *Top*
 nil
 user= 



-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: abysmal multicore performance, especially on AMD processors

2012-12-10 Thread Wm. Josiah Erikson

I tried some more performance tuning options in Java, just for kicks, and
didn't get any advantages from them: -server -XX:+TieredCompilation
-XX:ReservedCodeCacheSize=256m

Also, in case it's informative:

[josiah@compute-1-17 benchmark]$ grep entrant compilerOutputCompute-1-1.txt
| wc -l
173
[josiah@compute-1-17 benchmark]$ grep entrant
compilerOutputCompute-1-17.txt | wc -l
178
[josiah@compute-1-17 benchmark]$ grep zombie compilerOutputCompute-1-17.txt
| wc -l
163
[josiah@compute-1-17 benchmark]$ grep zombie compilerOutputCompute-1-1.txt
| wc -l
158
[josiah@compute-1-17 benchmark]$




On Mon, Dec 10, 2012 at 12:55 PM, Wm. Josiah Erikson wmjos...@gmail.comwrote:

 Aha. Not only do I get a lot of made not entrant, I get a lot of made
 zombie. However, I get this for both runs with map and with pmap (and with
 pmapall as well)

  For instance, from a pmapall run:

 33752  159 clojure.lang.Cons::next (10 bytes)   made zombie
   33752  164 clojure.lang.RT::conj (21 bytes)   made zombie
   33753  154 clojure.lang.RT::seq (32 bytes)   made zombie
   337535 %   clojure.core$reduce1::invoke @ -2 (184 bytes)
 made zombie
   33753  167 clojure.core$conj::invoke (13 bytes)   made zombie
   33753  184 clojure.core$rest::invoke (7 bytes)
   33884  186 clojure.lang.Numbers$LongOps::isPos (15 bytes)
   34298  187 clojure.lang.Numbers::dec (17 bytes)
   34421  188 clojure.lang.Numbers$LongOps::dec (13 bytes)
   34897  189 clojure.core$take::invoke (24 bytes)
   34903  190 clojure.core$take$fn__4227::init (15 bytes)
   35386  191 clojure.core$iterate::invoke (39 bytes)
 SNIP

   1168  154 clojure.lang.RT::seq (32 bytes)   made not entrant
1169  168 clojure.lang.RT::seq (32 bytes)
11715 %   clojure.core$reduce1::invoke @ -2 (184 bytes)
 made not entrant
1171  169 clojure.core$reduce1::invoke (184 bytes)
1173  159 clojure.lang.Cons::next (10 bytes)   made not
 entrant
1173  167 clojure.core$conj::invoke (13 bytes)   made not
 entrant
1173  164 clojure.lang.RT::conj (21 bytes)   made not
 entrant
1192  170 clojure.lang.PersistentList::first (5 bytes)
1193  171 clojure.lang.PersistentList::next (18 bytes)
1193  172 clojure.lang.Cons::next (10 bytes)
1194  173 clojure.lang.RT::conj (21 bytes)
1197  174 clojure.core$conj::invoke (13 bytes)
12336 %   clojure.core$reduce1::invoke @ 0 (184 bytes)

 And then, from a map run:

   1163  151 clojure.lang.RT::seq (32 bytes)   made not entrant
1163  145 clojure.core$cons::invoke (10 bytes)   made not
 entrant
1163  168 clojure.lang.RT::seq (32 bytes)
11654 %   clojure.core$reduce1::invoke @ 0 (184 bytes)
1169  144 clojure.lang.RT::cons (46 bytes)   made not
 entrant
3467  169 clojure.lang.RT::cons (46 bytes)
3470   24 clojure.lang.Util::equiv (65 bytes)   made zombie
3470   23 java.lang.String::equals (88 bytes)   made zombie
3470   18 clojure.lang.PersistentArrayMap::createWithCheck
 (80 bytes)   made zombie
3470   26 clojure.lang.PersistentArrayMap::equalKey (6
 bytes)   made zombie
3617  170 clojure.core$cons::invoke (10 bytes)
3622   41 clojure.lang.PersistentArrayMap::indexOf (34
 bytes)   made zombie
3622   30   ! java.net.URL::init (543 bytes)   made zombie
3622   58 clojure.lang.Symbol::equals (49 bytes)   made
 zombie
3623   73 java.lang.Object::equals (11 bytes)   made zombie
3623   65 clojure.lang.Util::hasheq (43 bytes)   made zombie
4249  171  s! clojure.lang.LazySeq::sval (54 bytes)
4259   77 clojure.lang.Util::equiv (65 bytes)   made zombie
4259   89 clojure.lang.RT::first (35 bytes)   made zombie
4259   88 clojure.lang.PersistentHashMap$NodeSeq::create
 (94 bytes)   made zombie
4578  172 n   java.lang.Object::getClass (0 bytes)
5380  173 clojure.lang.AFunction::init (5 bytes)
5634  174 java.lang.Long::longValue (5 bytes)
5785  175  s  clojure.lang.LazySeq::seq (53 bytes)
5830  176 clojure.lang.Numbers::ops (97 bytes)
6168  177 clojure.lang.LazySeq::init (10 bytes)
6169   32 java.lang.AbstractStringBuilder::append (48
 bytes)   made zombie
   10727  178 java.lang.Long::valueOf (36 bytes)
   10730   37 java.lang.StringBuilder::append (8 bytes)   made
 zombie
   10730   49   ! sun.misc.URLClassPath$JarLoader::getResource (91
 bytes)   made zombie
   10730   44

Re: class loaders stack constant grow in REPL

2012-12-10 Thread Tassilo Horn

Colin Jones trptco...@gmail.com writes:

Hi Colin,

 Right, this is because nREPL uses clojure.main/repl each time it does
 an evaluation. See http://dev.clojure.org/jira/browse/NREPL-31 for a
 related issue that was addressed by modifying clojure.main/repl.

 I'm not sure where a fix for this would belong (nREPL or
 clojure.main), but I went ahead and opened an nREPL JIRA issue to
 track it: http://dev.clojure.org/jira/browse/NREPL-36

Some time ago, I was also astonished by the large number of class
loaders.  But I'm pretty sure that the class loader list (i.e.,
ClassLoader.parent - ClassLoader.parent - ...) didn't grow without
bounds, but sometimes a parent becomes set to null and the previously
referenced class loader and all its parents becomes garbage collected.

I didn't do any debugging, but just used a function similar to
Vladimir's `print-class-loader-stack` every once in a while.

Bye,
Tassilo

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Starting to use Clojure, Yeah!

2012-12-10 Thread Manuel Paccagnella

Welcome! :)

Il giorno venerdì 7 dicembre 2012 04:16:28 UTC+1, Xiaodan Yuan ha scritto:

 Yeah!


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: class loaders stack constant grow in REPL

Good catch, guys.

Interesting that I never noticed this; likely because Pomegranate's 
`classloader-hierarchy` function ends up starting its walk from (RT/baseLoader) 
(which will be changing to the context classloader shortly):


https://github.com/cemerick/pomegranate/blob/master/src/main/clojure/cemerick/pomegranate.clj#L54

I'm open to suggestions.  I'm not keen on having yet another divergence in 
nREPL behaviour between 1.5.0 and all prior revs of Clojure, so an nREPL-local 
fix would be great.

My first impulse right now is to boot clojure.main/repl entirely and groom a 
replacement in-house, as it were (which I probably should have done from day 
1).

- Chas

On Dec 10, 2012, at 11:32 AM, Colin Jones wrote:

 Right, this is because nREPL uses clojure.main/repl each time it does an 
 evaluation. See http://dev.clojure.org/jira/browse/NREPL-31 for a related 
 issue that was addressed by modifying clojure.main/repl.
 
 I'm not sure where a fix for this would belong (nREPL or clojure.main), but I 
 went ahead and opened an nREPL JIRA issue to track it: 
 http://dev.clojure.org/jira/browse/NREPL-36
 
 - Colin
 
 
 
 
 On Monday, December 10, 2012 2:46:10 AM UTC-6, Vladimir Tsichevski wrote:
 Hi,
 
 just found that every interaction with Clojure REPL causes one more 
 DynamicClassLoader put on the Thread context class loader chain.
 
 Here is how clojure.main/repl beginning looks like:
 
   (let [cl (.getContextClassLoader (Thread/currentThread))]
 (.setContextClassLoader (Thread/currentThread) 
 (clojure.lang.DynamicClassLoader. cl)))
 
 And this is how to observe it:
 
 nREPL server started on port 19987
 REPL-y 0.1.0-beta10
 Clojure 1.4.0
 Exit: Control+D or (exit) or (quit)
 Commands: (user/help)
 Docs: (doc function-name-here)
   (find-doc part-of-name-here)
   Source: (source function-name-here)
   (user/sourcery function-name-here)
  Javadoc: (javadoc java-object-or-class-here)
 Examples from clojuredocs.org: [clojuredocs or cdoc]
   (user/clojuredocs name-here)
   (user/clojuredocs ns-here name-here)
 user= (defn print-class-loader-stack
   ([]
  (print-class-loader-stack (.getContextClassLoader 
 (Thread/currentThread
   ([cl]
  (if cl
(do
  (pprint cl)
  (print-class-loader-stack (.getParent cl)))
(println *Top*
   #_=   #_=   #_=   #_=   #_=   #_=   #_=   #_= 
 #'user/print-class-loader-stack
 user= (print-class-loader-stack)
 #DynamicClassLoader clojure.lang.DynamicClassLoader@26a0c73f
 #DynamicClassLoader clojure.lang.DynamicClassLoader@6f603bdc
 #DynamicClassLoader clojure.lang.DynamicClassLoader@2f368c5d
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5b31fd9
 #AppClassLoader sun.misc.Launcher$AppClassLoader@4aad3ba4
 #ExtClassLoader sun.misc.Launcher$ExtClassLoader@3326b249
 *Top*
 nil
 user= 1
 1
 user= 1
 1
 user= 1
 1
 user= 1
 1
 user= (print-class-loader-stack)
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5be04861
 #DynamicClassLoader clojure.lang.DynamicClassLoader@7481933a
 #DynamicClassLoader clojure.lang.DynamicClassLoader@273f212a
 #DynamicClassLoader clojure.lang.DynamicClassLoader@4178feba
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5323961b
 #DynamicClassLoader clojure.lang.DynamicClassLoader@26a0c73f
 #DynamicClassLoader clojure.lang.DynamicClassLoader@6f603bdc
 #DynamicClassLoader clojure.lang.DynamicClassLoader@2f368c5d
 #DynamicClassLoader clojure.lang.DynamicClassLoader@5b31fd9
 #AppClassLoader sun.misc.Launcher$AppClassLoader@4aad3ba4
 #ExtClassLoader sun.misc.Launcher$ExtClassLoader@3326b249
 *Top*
 nil
 user= 
 
 -- 
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with your 
 first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: abysmal multicore performance, especially on AMD processors

2012-12-10 Thread Marshall Bockrath-Vandegrift

Wm. Josiah Erikson wmjos...@gmail.com writes:

 Aha. Not only do I get a lot of made not entrant, I get a lot of
 made zombie. However, I get this for both runs with map and with
 pmap (and with pmapall as well)

I’m not sure this is all that enlightening.  From what I can gather,
“made not entrant” just means that a JITed version proved to be invalid
in light of later code and new invocation paths won’t use the previous
version.  And “made zombie” just means all references to an old JIT’d
version have been lost, making it available to be GC’ed.

A copy of `conj` becomes “not entrant” after being used on both vectors
and lists, but the new version gets the same speed-up when used on
vectors as a copy which has only been used on vectors.  There’s
something else going on which is specifically affecting parallel calls
to the polymorphic version when applied to instances of
`PersistentList`, and `Cons`.

-Marshall

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: abysmal multicore performance, especially on AMD processors

2012-12-10 Thread Wm. Josiah Erikson

Interesting. I tried the following:
:jvm-opts [-Xmx10g -Xms10g -XX:+AggressiveOpts -server
-XX:+TieredCompilation -XX:ReservedCodeCacheSize=256m -XX:TLABSize=1G
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseParNewGC
-XX:+ResizeTLAB -XX:+UseTLAB]

I got a slight slowdown, and the GC details are as follows:

[josiah@compute-1-1 benchmark]$ /usr/bin/time -f %E lein run
0.852: [GC 0.852: [ParNew: 2796224K-6726K(3145728K), 0.0136610 secs]
2796224K-6726K(10136256K), 0.0136930 secs] [Times: user=0.05 sys=0.02,
real=0.01 secs]
0.871: [GC 0.871: [ParNew: 2802950K-8104K(3145728K), 0.0116360 secs]
2802950K-8104K(10136256K), 0.0116720 secs] [Times: user=0.06 sys=0.01,
real=0.01 secs]
0.890: [GC 0.890: [ParNew: 2804328K-11538K(3145728K), 0.0112460 secs]
2804328K-11538K(10136256K), 0.0112720 secs] [Times: user=0.08 sys=0.00,
real=0.01 secs]
0.904: [GC 0.904: [ParNew: 2807762K-11752K(3145728K), 0.0092300 secs]
2807762K-11752K(10136256K), 0.0092550 secs] [Times: user=0.06 sys=0.00,
real=0.01 secs]
0.915: [GC 0.915: [ParNew: 2807976K-10702K(3145728K), 0.0072210 secs]
2807976K-10702K(10136256K), 0.0072480 secs] [Times: user=0.06 sys=0.00,
real=0.01 secs]
0.969: [GC 0.969: [ParNew: 2806926K-12249K(3145728K), 0.0206880 secs]
2806926K-12249K(10136256K), 0.0207160 secs] [Times: user=0.13 sys=0.01,
real=0.02 secs]
21.099: [GC 21.099: [ParNew: 2808473K-14256K(3145728K), 0.0174230 secs]
2808473K-14256K(10136256K), 0.0174580 secs] [Times: user=0.12 sys=0.00,
real=0.02 secs]
46.533: [GC 46.533: [ParNew: 2810480K-10070K(3145728K), 0.0097840 secs]
2810480K-10070K(10136256K), 0.0098140 secs] [Times: user=0.08 sys=0.00,
real=0.01 secs]
74.988: [GC 74.988: [ParNew: 2806294K-11576K(3145728K), 0.0134020 secs]
2806294K-11576K(10136256K), 0.0134330 secs] [Times: user=0.08 sys=0.00,
real=0.02 secs]
105.143: [GC 105.143: [ParNew: 2807800K-12728K(3145728K), 0.0121870 secs]
2807800K-12728K(10136256K), 0.0122240 secs] [Times: user=0.08 sys=0.00,
real=0.02 secs]
136.170: [GC 136.170: [ParNew: 2808952K-13336K(3145728K), 0.0144400 secs]
2808952K-13336K(10136256K), 0.0144720 secs] [Times: user=0.09 sys=0.00,
real=0.01 secs]
167.703: [GC 167.703: [ParNew: 2809560K-14763K(3145728K), 0.0105520 secs]
2809560K-14763K(10136256K), 0.0105830 secs] [Times: user=0.07 sys=0.00,
real=0.01 secs]
199.593: [GC 199.593: [ParNew: 2810987K-11407K(3145728K), 0.0142030 secs]
2810987K-11407K(10136256K), 0.0142350 secs] [Times: user=0.08 sys=0.00,
real=0.01 secs]
231.894: [GC 231.894: [ParNew: 2807631K-15066K(3145728K), 0.0129290 secs]
2807631K-15066K(10136256K), 0.0129630 secs] [Times: user=0.10 sys=0.01,
real=0.01 secs]
264.239: [GC 264.239: [ParNew: 2811290K-9632K(3145728K), 0.0119130 secs]
2811290K-9632K(10136256K), 0.0119850 secs] [Times: user=0.08 sys=0.00,
real=0.01 secs]
Elapsed time: 291038.415325 msecs
Heap
 par new generation   total 3145728K, used 2700935K [0x00057ae0,
0x00065035, 0x00065035)
  eden space 2796224K,  96% used [0x00057ae0, 0x00061f239bb0,
0x0006258b)
  from space 349504K,   2% used [0x00063ae0, 0x00063b768340,
0x00065035)
  to   space 349504K,   0% used [0x0006258b, 0x0006258b,
0x00063ae0)
 tenured generation   total 6990528K, used 0K [0x00065035,
0x0007fae0, 0x0007fae0)
   the space 6990528K,   0% used [0x00065035, 0x00065035,
0x000650350200, 0x0007fae0)
 compacting perm gen  total 21248K, used 11049K [0x0007fae0,
0x0007fc2c, 0x0008)
   the space 21248K,  52% used [0x0007fae0, 0x0007fb8ca638,
0x0007fb8ca800, 0x0007fc2c)
No shared spaces configured.
4:53.06
[josiah@compute-1-1 benchmark]$



On Mon, Dec 10, 2012 at 3:33 PM, Marshall Bockrath-Vandegrift 
llas...@gmail.com wrote:

 Wm. Josiah Erikson wmjos...@gmail.com writes:

  Aha. Not only do I get a lot of made not entrant, I get a lot of
  made zombie. However, I get this for both runs with map and with
  pmap (and with pmapall as well)

 I’m not sure this is all that enlightening.  From what I can gather,
 “made not entrant” just means that a JITed version proved to be invalid
 in light of later code and new invocation paths won’t use the previous
 version.  And “made zombie” just means all references to an old JIT’d
 version have been lost, making it available to be GC’ed.

 A copy of `conj` becomes “not entrant” after being used on both vectors
 and lists, but the new version gets the same speed-up when used on
 vectors as a copy which has only been used on vectors.  There’s
 something else going on which is specifically affecting parallel calls
 to the polymorphic version when applied to instances of
 `PersistentList`, and `Cons`.

 -Marshall

 --
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To

Best practices for java libraries

2012-12-10 Thread Michael Grubb

I've been searching for some best practices when it comes to wrapping 
existing Java libraries to make them more clojurized.
Unfortunately I've not found much.
While I know enough to make something that works for me, I'd like to write 
it in such a manner that it can be used (and read) by others
and not make them cringe in doing so.

My first attempt was to simply write function wrappers around the java 
interop calls.
This worked fine and was maintainable.  Then I discovered 'lein check' and 
saw a whole lot of warnings about reflection.

This led me to my second attempts.  This time I wrote multimethods to 
dispatch on type of arguments and in each defmethod argument list
I gave type hints for each argument.  This indeed resolved the reflection 
warnings, but I fear that the resulting code is rather messy and not
too pleasant to read either.  This made me wonder if all the type checking 
I'm doing in the multimethod dispatch function isn't just as bad as
the reflection that would take place.

For my purposes and this particular library I'm not really concerned with 
performance, yet someone else may be.  Can anyone offer me some
advice on how best to approach a project like this?

Kind regards,
Michael

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Is There A Way To Exit ClojureScript REPL

2012-12-10 Thread Asim Jalis

Is there a way to exit the ClojureScript REPL? None of these worked:
Ctrl-D, exit. (exit 1), quit, (quit 1). So eventually I killed the window.

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Is There A Way To Exit ClojureScript REPL

2012-12-10 Thread Devin Walters

Have you tried :cljs/quit ? 

Cheers,
-- 
'(Devin Walters)


On Monday, December 10, 2012 at 3:23 PM, Asim Jalis wrote:

 Is there a way to exit the ClojureScript REPL? None of these worked: Ctrl-D, 
 exit. (exit 1), quit, (quit 1). So eventually I killed the window. 
 
 -- 
 You received this message because you are subscribed to the Google
 Groups Clojure group.
 To post to this group, send email to clojure@googlegroups.com 
 (mailto:clojure@googlegroups.com)
 Note that posts from new members are moderated - please be patient with your 
 first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com 
 (mailto:clojure+unsubscr...@googlegroups.com)
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en 

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: leiningen and updating project versions

2012-12-10 Thread Phil Hagelberg

On Sun, Dec 9, 2012 at 12:18 AM, Dave Sann daves...@gmail.com wrote:
 Has anyone looked at how to manage snapshots and dependencies in projects
 with checkouts - where the checkouts (also snapshots) are being edited along
 with the main project?

We have so far punted on tasks that perform automated changes to
project.clj because the Clojure reader doesn't allow lossless
round-trips--any changes would have to be done using regexes or would
destroy comments that had been placed in project.clj. But now that
there's an alternate reader that can preserve comments
(https://github.com/cgrand/sjacket/) this kind of thing will be more
feasible.

-Phil

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

New System Architecture in Clojure

2012-12-10 Thread ArturoH

Everybody,

I'd like to define a systemwide data structure with Clojure. I'd like it to 
represent input data, and derived data. Some specified derived data could 
be temporary for the calculation of other derived data. I also would like 
to use functions to specify the derivation of data. The idea would be to 
specify small derivations/transformations that would eventually produce the 
desired output.

Once I have the system specified in such way, I'd like it to produce an SQL 
database. And generate code that would be 'practical' to run at the 
database level. And have the code that is best to run in Clojure to remain 
on Clojure. I understand that deciding where is best to execute the code 
may need to be a human activity. But I'd like the system to be flexible.

I think that the current basic Clojure operations may not be best suited 
for this kind of specification. But I do think clojure may be the best 
language to do this kind specification. Any ideas/opinions?

I would appreciate any feedback.

Arturo Hernandez

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: core.logic vs datomic

2012-12-10 Thread Herwig Hochleitner

In my very incomplete understanding, the main difference between datalog
and core.logic is that datalog is set-oriented and core.logic is
tuple-oriented. Also datalog is designed to join and filter on external
datasources, while in core.logic, fresh vars (i.e. join points ??) also can
be created in the query.
Hence datalog queries can be guaranteed to terminate, as long as you don't
get crazy and build a Y-combinator out of rules (gotta try that :-)

Were I to try and implement a zebra puzzle solver using datalog, I'd start
with representing an inhabitant as a set of all possible items on each
slot. Then I'd try to formulate a query that narrowed the possibilites
according to constraints and a query that propagated eliminated
possibilities between inhabitants. Then I'd try to run those queries in
lockstep until the result is stable and hopefully a definite allocation of
items to inhabitants.

That's just from my memory of hand-solving the zebra puzzle, unfortuately I
don't know how that would actually work out, I haven't had much time to
play with datomic yet. But I can see using it in my next project requiring
a database :-)

kind regards

P.S. One reason I love the clojure community the most, is that exchanges
here are conducted with the utmost amount of professional courtesy as well
as precision.

-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Best practices for java libraries

Some thoughts from various Java libraries I have wrapped:

- Normal functions are generally best for wrapping
- It can often make sense to have a protocol that dispatched on the type of 
the Java object and/or clojure params for polymorphism and extension. Your 
public functions should often call the protocol functions (after applying 
any defaults / validation / argument re-ordering etc.)
- It can make sense to turn mutable Java APIs into immutable Clojure ones 
in some cases. Depends on whether the Java API allows this and how much you 
care about performance.
- Try to type hint with interfaces or abstract base classes only. It gets 
messy if you start making special cases for specific concrete types, and is 
probably a sign of incoherent design if you feel you need to do this. Best 
to figure out the small set of Java abstractions that you want your API to 
work with / expose and stick to functions that maipulate these.
- Ensure you have zero reflection warnings. Apart from the performance 
cost, they are usually a sign of a logic error.
- I usually find the need for a few constructor functions to handle 
different use cases. Not sure that having a 1-arg constructor from a map is 
the best low level option - this will depend on the Java API. Often a 0-3 
arg constructor, a clone constructor or one that takes a ListObject seem 
to be the most common. If there are a lot of options, I'd tend to make 
these keyword params.
- Keyword args are your friend - they often translate to setXXX or addXXX 
calls in a Java API and enable you to wrap quite a lot of complexity as 
extra parameters to one function. I think these are better than map 
parameters on average, YMMV.

On Monday, 10 December 2012 13:21:09 UTC+8, Michael Grubb wrote:

 I've been searching for some best practices when it comes to wrapping 
 existing Java libraries to make them more clojurized.
 Unfortunately I've not found much.
 While I know enough to make something that works for me, I'd like to write 
 it in such a manner that it can be used (and read) by others
 and not make them cringe in doing so.

 My first attempt was to simply write function wrappers around the java 
 interop calls.
 This worked fine and was maintainable.  Then I discovered 'lein check' and 
 saw a whole lot of warnings about reflection.

 This led me to my second attempts.  This time I wrote multimethods to 
 dispatch on type of arguments and in each defmethod argument list
 I gave type hints for each argument.  This indeed resolved the reflection 
 warnings, but I fear that the resulting code is rather messy and not
 too pleasant to read either.  This made me wonder if all the type checking 
 I'm doing in the multimethod dispatch function isn't just as bad as
 the reflection that would take place.

 For my purposes and this particular library I'm not really concerned with 
 performance, yet someone else may be.  Can anyone offer me some
 advice on how best to approach a project like this?

 Kind regards,
 Michael


-- 
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: New System Architecture in Clojure