Tetris

[Zach Tellman]

I've thrown together a very simple Tetris clone, which can be found at
http://github.com/ztellman/penumbra/blob/master/src/examples/tetris.clj
.  Excluding the parts hidden away by the framework, it's a purely
functional implementation.  I figured a few people here might find it
interesting.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~--~~~~--~~--~--~---

Re: Leiningen Run ?

[Zach Tellman]

On Dec 1, 8:15 am, David Nolen  wrote:
> On Tue, Dec 1, 2009 at 1:10 AM, Phil Hagelberg  wrote:
> > David Nolen  writes:
>
> > > The problem is that JOGL needs JNIs and JNIs need to be on
> > > java.library.path or java.ext.dirs, not the classpath. In order to
> > > make life easier for people learning about clojure as well generally
> > > making lein projects simpler to play around for newbies do people see
> > > any utility in supporting something like lein run?
>
> > It sounds like you're talking about two orthogonal things: setting java
> > options for JNI and a run task. We have talked about adding a run task;
> > if you're interested we could even get it in for the 1.0.0 release. If
> > the java.library.path flag can be set in a subclassloader like we do
> > with the classpath in the deps task in the isolated-compile branch that
> > Alex just cooked up, then this could be added soon as well. But it would
> > be helpful to discuss these as two separate steps.
>
> Yes.
>
>
>
> > Also from talking on IRC it sounded like there might also be a third
> > unpacking step involved; could you explain that in more detail here? It
> > sounds like the SqliteJDBC libraries do that for you, but perhaps that
> > is not always the case with native libraries?
>
> SQLiteJDBC decides to take the runtime solution via System.load. Benefit is
> that everything stays in the jar.
>
> The other tack is to support specifying platform specific library jars and
> unpacking them into whatever folder is specified by project.clj
> library.path.
>
> David
>
>
>
> > -Phil
>

I've looked into this a bit, because right now setting up Penumbra
(http://github.com/ztellman/penumbra) is way too complex. As far as I
can see, there are two different situations to think about:

1) A standalone executable which relies on Penumbra.  This situation
is actually pretty well supported by JOGL's use of JNLP, which will
download the appropriate binaries at startup (and cache them for later
usage). This has the benefit of being more lightweight (the jar
doesn't have to contain all the binaries for all the target
platforms), so maybe there's an argument for Leiningen generating
the .jnlp file.

2) A library which relies on Penumbra, or any development work towards
a standalone executable.  If there were some generic code provided by
Leiningen which did the runtime extraction of the binaries, I don't
really see any issue with doing it that way.  The "lein run" approach
described above would also work, with the caveat that "lein swank"
would also need to have similar functionality.

With all that said, I'm certainly not an expert on the subject.  If
any of the above seems suspect, please feel free to correct me.

Zach

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Leiningen Run ?

[Zach Tellman]

On Dec 1, 3:31 pm, David Nolen  wrote:
> So just to keep the conversation going:
>
> http://download.java.net/maven/2/net/java/dev/gluegen/http://download.java.net/maven/2/net/java/dev/jogl/
>
> I note that these two maven repos specify the platform with the following:
>
> lib-{platform}-{arch}
>
> where platform is:
> macosx
> linux
> windows
>
> arch is:
> universal
> i586
>
> followed by the architecture. I'm assuming lein should resolve this
> auto-magically based on the platform Java is running on? I don't mind
> contributing some code to make this work if there's feedback on how it
> should work what it should look like.
>
> David

That maven repository isn't for JOGL 2.0, which is what I'm using. I'm
not sure a maven repo exists for that version currently. Would the
solution be to make such a repository for any JNI libraries we use?
That's not really a problem, I'm just unclear on precisely what you're
suggesting.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Leiningen Run ?

[Zach Tellman]

On Dec 2, 12:38 pm, David Nolen  wrote:
> Yeah it sounds like you'll need to package up JOGL 2 and push it to Clojars
> right?
>
>
>
> On Wed, Dec 2, 2009 at 3:20 PM, Zach Tellman  wrote:
> > On Dec 1, 3:31 pm, David Nolen  wrote:
> > > So just to keep the conversation going:
>
> >http://download.java.net/maven/2/net/java/dev/gluegen/http://download...
>
> > > I note that these two maven repos specify the platform with the
> > following:
>
> > > lib-{platform}-{arch}
>
> > > where platform is:
> > > macosx
> > > linux
> > > windows
>
> > > arch is:
> > > universal
> > > i586
>
> > > followed by the architecture. I'm assuming lein should resolve this
> > > auto-magically based on the platform Java is running on? I don't mind
> > > contributing some code to make this work if there's feedback on how it
> > > should work what it should look like.
>
> > > David
>
> > That maven repository isn't for JOGL 2.0, which is what I'm using. I'm
> > not sure a maven repo exists for that version currently. Would the
> > solution be to make such a repository for any JNI libraries we use?
> > That's not really a problem, I'm just unclear on precisely what you're
> > suggesting.
>
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Clojure" group.
> > To post to this group, send email to clojure@googlegroups.com
> > Note that posts from new members are moderated - please be patient with
> > your first post.
> > To unsubscribe from this group, send email to
> > clojure+unsubscr...@googlegroups.com > >
> > For more options, visit this group at
> >http://groups.google.com/group/clojure?hl=en

If that's what it takes, great.  Someone just needs to define what a
multi-platform JNI package looks like. I'm willing to go along with
whatever is decided upon by technomancy et al.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Trying to rewrite a loop as map/reduce

[Zach Tellman]

At first glance I don't see a clean to make this completely higher-
order, but here's a shorter (albeit a little messy) version:

(loop [a a0]
(let [[a b c d e] (reduce #(conj %1 (%2 (last %1))) [a] [f1 f2 f3
f4])
  g (f5 c)
  h (-> e f2 f5)]
  (if (or (f6? b) (<= g h))
e
(recur (f7 d b)


On Dec 15, 11:00 am, samppi  wrote:
> I'm trying to rewrite a loop to use higher-level functions instead.
> For pure functions f1, f2, f3, f4, f5, f6?, and f7, and a Clojure
> object a0, how can one rewrite the following loop to use map, reduce,
> etc.?
>
>   (loop [a a0]
>     (let [b (f1 a)
>           c (f2 b)
>           d (f3 c)
>           e (f4 d)
>           g (f5 c)
>           h (-> e f2 f5)]
>       (if (or (f6? b) (<= g h))
>         e
>         (recur (f7 d b)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: How to fund open source Clojure development

[Zach Tellman]

All of these seem to distract from the activity we're trying to fund:
the development of Clojure.  If the current approach can bring in
enough money, it strikes me as fairly ideal.  We'll just have to wait
and see if it does.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Inconsistent behavior in 1.2.0-master w.r.t. previous versions

[Zach Tellman]

At the REPL, in 1.2.0-master

> (meta (second '(a #^b c)))
nil

In 1.1.0-new (and I believe all previous versions)

> (meta (second '(a #^b c)))
{:tag b}

Is this intentional, or a bug?  Is the new type hint syntax being
introduced in 1.2?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Inconsistent behavior in 1.2.0-master w.r.t. previous versions

[Zach Tellman]

I've double-checked I have the latest from github, cleaned and
recompiled, and I'm still getting the same results.

user=> [*clojure-version* (meta (second '(a #^b c)))]
[{:interim true, :major 1, :minor 2, :incremental 0, :qualifier
"master"} nil]

Does anyone have an idea what's going on here?

On Jan 25, 8:01 pm, wlr  wrote:
> Works for me.
>
> user> [*clojure-version* (meta (second '(a #^b c)))]
> [{:interim true, :major 1, :minor 2, :incremental 0, :qualifier
> "master"} {:tag b}]

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Clojure binding for Open CL

[Zach Tellman]

A few months back I created very basic bindings for CL4Java (the code
for it still exists in Penumbra, under src/opencl).  It then
subsequently was renamed to JOCL, which was already in use by another
OpenCL library, and they started to work on combining their efforts,
and I decided to wait until everything was figured out on that end.
If anyone can point me in the direction of a clean binding in Java for
OpenCL (I don't care so much about utility functions, just the
underlying API), I'd be happy to take another crack at it.

Zach

On Feb 10, 6:58 am, atucker  wrote:
> From his todo list (1), it looks as if ztellman (2) might have
> concrete plans to include it in the (currently OpenGL) wrapper project
> penumbra (3).
>
> 1.http://wiki.github.com/ztellman/penumbra/todo
> 2.http://ideolalia.com/
> 3.http://github.com/ztellman/penumbra
>
> On Feb 9, 9:48 pm, ka  wrote:
>
>
>
> > Hi,
>
> > I was just wondering if (by now) Open CL has been 'wrapped' by higher
> > level languages. I came across these from the Khronos site 
> > (http://www.khronos.org/developers/resources/opencl/#timplementations) -
> > 1.http://ruby-opencl.rubyforge.org/
> > 2.http://planet.plt-scheme.org/display.ss?package=opencl.plt&owner=jaym...
> > 3.http://mathema.tician.de/software/pyopencl
>
> > Wondering if anyone is already working on a Open CL binding.  I would
> > love if I can code in clojure and it runs through Open CL drivers :)
> > on my Radeon!
>
> > Thanks!

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: jogl classpath not found

[Zach Tellman]

In the past, I haven't had a lot of luck putting the JOGL libraries on
the classpath.  A much better approach, I've found, is to create a
custom script to load up clojure, and put the class and library paths
as parameters to the java executable.  My library used to target JOGL
(it now uses LWJGL), and the script I suggested in the wiki can be
found at http://wiki.github.com/ztellman/penumbra/getting-started/16.
Hopefully some variation on that should work for you.

On Mar 11, 1:28 pm, strattonbrazil  wrote:
> I downloaded the new jogl 2.0 libs and can't get jogl to be recognized
> in my classpath.  It's stuck on the import.
>
> (import '(javax.media.opengl.awt GLCanvas))
>
> In my shell I have clojure aliased to 'java -jar clojure-1.0.0.jar'
>
> I have my CLASSPATH as
>
> echo $CLASSPATH
> ./jogl-2.0-linux-amd64/lib/jogl.all.jar:./jogl-2.0-linux-amd64/lib/
> nativewindow.all.jar:./jogl-2.0-linux-amd64/lib/gluegen-rt.jar
>
> I found this command online to print the classpath in clojure, which I
> assume works.
> (println (seq (.getURLs (java.lang.ClassLoader/
> getSystemClassLoader
>
> But I only get the clojure jar.
>
> (#)
>
> This is the actual error message from the import:
>
> Exception in thread "Main Thread" java.lang.ClassNotFoundException:
> javax.media.opengl.awt.GLCanvas (test.clj:0)
>         at clojure.lang.Compiler.eval(Compiler.java:4543)
>         at clojure.lang.Compiler.load(Compiler.java:4857)
>         at clojure.lang.Compiler.loadFile(Compiler.java:4824)
>         at clojure.main$load_script__5833.invoke(main.clj:206)
>         at clojure.main$script_opt__5864.invoke(main.clj:258)
>         at clojure.main$main__5888.doInvoke(main.clj:333)
>         at clojure.lang.RestFn.invoke(RestFn.java:413)
>         at clojure.lang.Var.invoke(Var.java:346)
>         at clojure.lang.AFn.applyToHelper(AFn.java:173)
>         at clojure.lang.Var.applyTo(Var.java:463)
>         at clojure.main.main(main.java:39)
> Caused by: java.lang.ClassNotFoundException:
> javax.media.opengl.awt.GLCanvas
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:247)
>         at clojure.lang.RT.classForName(RT.java:1487)
>         at clojure.core$import__4028.doInvoke(core.clj:1860)
>         at clojure.lang.RestFn.invoke(RestFn.java:426)
>         at user$eval__4.invoke(test.clj:3)
>         at clojure.lang.Compiler.eval(Compiler.java:4532)
>         ... 10 more

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

Re: Name suggestions

[Zach Tellman]

how about 'patois' or 'creole'

On Mar 17, 12:08 am, mac  wrote:
> After just a little more test and polish I plan on calling clj-native
> 1.0. But clj-native is a *really* boring name so I want to change it
> before 1.0 and I don't have very good imagination when it comes to
> these things.
> So I need your help.
> It doesn't have to have anything to do with anything, could just be
> something that sounds funny, like sasquatch, that's a funny word.
>
> Please help!
>
> /Markus
>
> On 13 mar, 19:14, mac  wrote:
>
>
>
> > Hello all.
> > I have had some time lately to work on my C FFI for Clojure and I
> > think it's pretty much feature complete now.
> > It has support for functions, callbacks, structures, unions and
> > globals.
> > For structures there is support for different alignments.
>
> > The library has two main namespaces: clj-native.direct and clj-
> > native.dynamic.
> > The direct namespace uses the "direct mapping" feature of JNA in order
> > to be as efficient and arity and type safe as possible. However, it
> > does not support varargs functions since they require reflection and
> > dynamic translation of the parameters. That's what the dynamic
> > namespace is for: It can map any C function into clojure, even vararg
> > functions like printf but at a higher cost in call time and less
> > safety (it's easy to crash the jvm by sending the wrong type or number
> > of parameters). Access to global variables through the JNA Pointer
> > class is also available in the dynamic namespace.
>
> > The library is available from github:http://github.com/bagucode/clj-native
> > and clojars:http://clojars.org/clj-native
>
> > Example usage of the direct namespace can be found 
> > here:http://github.com/bagucode/clj-native/blob/master/src/examples/c_lib.clj
>
> > Please report issues or make requests to my github account or by mail.
>
> > Enjoy!
>
> > /Markus

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

To unsubscribe from this group, send email to 
clojure+unsubscribegooglegroups.com or reply to this email with the words 
"REMOVE ME" as the subject.

defprotocol's support for variadic arguments seems broken

[Zach Tellman]

Possibly this fall out from the latest commit requiring an explicit
'this' reference (ba6cc3b), I haven't checked any versions but the
most recent.

user> (defprotocol Protocol (f [a b & c]))
Protocol
user> (def p (reify Protocol (f [a b & c] [a b c])))
#'user/p
user> (f p :a)
No single method: f of interface: user.Protocol found for function: f
of protocol: Protocol
user> (f p :a :b)
No single method: f of interface: user.Protocol found for function: f
of protocol: Protocol
user> (f p :a :b :c)
[# :a :c]
user> (f p :a :b :c :d)
No single method: f of interface: user.Protocol found for function: f
of protocol: Protocol

Even the one signature that doesn't result in an exception doesn't
seem right.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

To unsubscribe, reply using "remove me" as the subject.

Re: defprotocol's support for variadic arguments seems broken

[Zach Tellman]

You're right, that does explain it.  I assumed that variadic arguments
were supported, but maybe I was wrong.  Can anyone confirm whether or
not this was ever intended to work?


On Apr 6, 2:32 am, Jarkko Oranen  wrote:
> On Apr 6, 8:16 am, Zach Tellman  wrote:
>
> > Possibly this fall out from the latest commit requiring an explicit
> > 'this' reference (ba6cc3b), I haven't checked any versions but the
> > most recent.
>
> > user> (defprotocol Protocol (f [a b & c]))
> > Protocol
> > user> (def p (reify Protocol (f [a b & c] [a b c])))
> > #'user/p
> > user> (f p :a :b :c)
> > [# :a :c]
>
> Did protocols ever support variadic arguments? It seems to me that in
> this case & gets treated as a regular symbol and a gets bound to the
> object itself, b to :a, & to :b and c to :c, which would explain the
> weird output.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

To unsubscribe, reply using "remove me" as the subject.

[ANN] 'primitive-math', a library for predictable arithmetic operations

[Zach Tellman]

What (+ x y) compiles down is highly dependent on the surrounding context, 
including but not limited to the local type-hints and the value of 
*unchecked-math*.  Actually verifying that it's calling the primitive, 
unboxed, easily inlined clojure.lang.Numbers.add(long, long) requires 
either a profiler or a decompiler.

For a lot of use cases, this doesn't matter.  But where it does matter, it 
tends to turn into a mind-numbing cycle of adding type-hints, seeing 
whether that changed the compilation output, and then trying again.  Since 
I've gone through this more times than I'd like to admit, I rolled my 
own: https://github.com/ztellman/primitive-math.

This is not meant to be a replacement for Clojure's numeric tower, rather 
it's meant to supplement it when the range of possible inputs is 
intentionally narrow.  I expect this should be immediately useful in a 
variety of applications and libraries.

If anyone has questions, I'm happy to answer them.

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

ANN: Potemkin 0.3.0

[Zach Tellman]

Potemkin [1] is a collection of facades and utilities that I've found 
helpful when writing larger-scale libraries or applications.  I've never 
formally announced it before, but I think it's gotten to the point where 
others can benefit from it.

A few highlights:

* 'def-map-type', which allows for the definition of custom map-like 
objects with 10x less code
* 'unify-gensyms', which allows for more concise nested syntax-quotes
* 'import-vars', which allows for code sprinkled across multiple namespaces 
to be exposed via a single namespace

It's been pointed out before that ideally a library should have no 
dependencies but Clojure itself, or we risk transitive dependency conflicts 
when everyone uses different versions of a utility library.  In deference 
to this, Potemkin is licensed such that any piece of code can be simply 
pasted into your library, as long as there's a comment describing the 
origin.

If anyone has questions, I'm happy to answer them.  If anyone has moral or 
aesthetic objections to 'import-vars', you're not alone, but please 
remember you're under no obligation to use it.

Zach

[1] https://github.com/ztellman/potemkin

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Factual is hiring Clojure developers

[Zach Tellman]

At Factual, we build and constantly refine a canonical index of real world 
entities: businesses, locations, and products [1].  We also provide ways 
for other data to be resolved against our index [2] [3], allowing our 
service to act as a join table for real-world entities, breaking down 
barriers between otherwise siloed data.  

This involves a wide variety of challenges in both data analysis and 
systems engineering, many of which we solve with Clojure [4].  Working at 
Factual, you have the opportunity to solve difficult problems using 
large-scale data, and to solve them well.  You can work alongside a number 
of smart, enthusiastic engineers to build, refine, and validate our data 
pipelines until the output is, well, factual.

Factual's headquarters are in LA, and we also have offices in the SF Bay 
Area.  Remote employment is possible, but usually only when the candidate 
has significant experience in a relevant area, and is considered on a 
case-by-case basis.  Anyone who's interested in learning more should feel 
free to contact me at z...@factual.com.

[1] http://www.factual.com/
[2] http://www.factual.com/data-apis/places/resolve
[3] http://www.factual.com/data-apis/places/crosswalk
[4] http://www.factual.com/jobs/clojure

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ANN] byte-streams: a rosetta stone for all the byte representations the jvm has to offer

[Zach Tellman]

I've recently been trying to pull out useful pieces from some of my more 
monolithic libraries.  The most recent result is 'byte-streams' [1], a 
library that figures how how to convert between different byte 
representations (including character streams), and how to efficiently 
transfer bytes between various byte sources and sinks.  The net result is 
that you can do something like:

  (byte-streams/convert (File. "/tmp/foo") String {:encoding "utf-8"})

and get a string representation of the file's contents.  Of course, this is 
already possible using 'slurp', but you could also convert it to a 
CharSequence, or lazy sequence of ByteBuffers, or pretty much anything else 
you can imagine.  This is accomplished by traversing a graph of available 
conversions (don't worry, it's memoized), so simply defining a new 
conversion from some custom type to (say) a ByteBuffer will transitively 
allow you to convert it to any other type.

As an aside, this sort of conversion mechanism isn't limited to just byte 
representations, but I'm not sure if there's another large collection of 
mostly-isomorphic types out there that would benefit from this.  If anyone 
has ideas on where else this could be applied, I'd be interested to hear 
them.

Zach

[1] https://github.com/ztellman/byte-streams

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: faster zipper implementation

[Zach Tellman]

This is really cool, thanks for taking the time to do this.  I was able to 
eke out another 1.8x speedup by changing the keyword equality checks with 
'identical?' [1], and there might be some further room for improvement by 
defining inline forms for some of the smaller functions.

Zach

[1] 
https://github.com/ztellman/fast-zip/commit/ee7a64630389f36a539771658586a093369f7939

On Sunday, June 30, 2013 1:18:05 PM UTC-7, Alexander Hudek wrote:
>
> I've updated the clojure.zip implementation to use records internally. This
> achieves a speedup of roughly 2x. You can find the library below and on
> clojars:
>
> https://github.com/akhudek/fast-zip
>
> It's a drop in replacement for clojure.zip in terms of interface and 
> usage. 
> However, since the internal representation has changed, fast-zip locations 
> cannot be used with the clojure.zip implementation. E.g. you cannot
> create a zipper with fast-zip, then use clojure.zip/next on the result. 
> You 
> must use fast.zip.core/next instead. 
>
> In addition to the micro-benchmark provided, I've tried this in a larger 
> project
> that makes extensive use of zippers and found a similar speedup.
>
> Comments and suggestions welcome.
>
> Alex
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: faster zipper implementation

[Zach Tellman]

This is really cool, thanks for doing this.  I was able to eke out another 
1.8x speedup by replacing '=' with 'identical?' for the keyword comparisons 
[1]. There also might be further room for improvement by defining inline 
forms for some of the smaller functions.

Zach

[1] 
https://github.com/ztellman/fast-zip/commit/ee7a64630389f36a539771658586a093369f7939

On Sunday, June 30, 2013 1:18:05 PM UTC-7, Alexander Hudek wrote:
>
> I've updated the clojure.zip implementation to use records internally. This
> achieves a speedup of roughly 2x. You can find the library below and on
> clojars:
>
> https://github.com/akhudek/fast-zip
>
> It's a drop in replacement for clojure.zip in terms of interface and 
> usage. 
> However, since the internal representation has changed, fast-zip locations 
> cannot be used with the clojure.zip implementation. E.g. you cannot
> create a zipper with fast-zip, then use clojure.zip/next on the result. 
> You 
> must use fast.zip.core/next instead. 
>
> In addition to the micro-benchmark provided, I've tried this in a larger 
> project
> that makes extensive use of zippers and found a similar speedup.
>
> Comments and suggestions welcome.
>
> Alex
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] byte-streams: a rosetta stone for all the byte representations the jvm has to offer

[Zach Tellman]

The library defines two protocols, ByteSource and ByteSink.  They expose a 
'take-bytes!' and 'send-bytes!' method, respectively.  An unoptimized 
transfer involves shuttling bytes between these two methods, which in most 
cases means one copy and one allocation (ByteBuffer can act as a ByteSource 
and requires neither).

An optimized transform is anything which short-circuits this default 
mechanism.  Unfortunately, exactly how optimal this is depends heavily on 
what you're transferring.  Copying a File to a Channel representing a file 
descriptor will often use OS-provided zero copy mechanisms, which makes it 
a lot more "optimal" than the optimized transfer between an InputStream and 
an OutputStream, which just involves reusing an array for shuttling data 
between them.  Basically, 'optimized-transfer?' returning true means that a 
special function exists that is as optimal as I (or someone else who's 
written a def-transfer) know how to make it.

The number of steps is definitely not the number of copies.  Wrapping an 
array in a ByteArrayInputStream, for instance, is just allocating a 
wrapper.  Some are more complicated though; creating an array from a 
ByteBuffer is only zero-copy if the ByteBuffer is non-direct *and* the 
position and limit haven't been moved.  Right now my traversal is treating 
each transform as equally complex, but an obvious improvement is to allow 
each transform to be annotated with some sort of overhead metric, and 
minimize based on that.  A lot of the low-level optimizations are 
implementation-specific, though, so it's always going to be a little fuzzy.

If you have ideas on any of the above, please speak up.

Zach

On Sunday, June 30, 2013 8:46:47 PM UTC-7, Dan Burkert wrote:
>
> Very cool, I've got a couple of question: the readme references optimized 
> transfers, what qualifies as an optimized transfer?  Also, would it be 
> possible for byte-streams to give an estimation of the number of memory 
> copies that happen in a given conversion (maybe this is as simple as the 
> number of steps...)?  Thanks for releasing this!
>
> -- Dan
>
> On Saturday, June 29, 2013 1:57:58 PM UTC-4, Zach Tellman wrote:
>>
>> I've recently been trying to pull out useful pieces from some of my more 
>> monolithic libraries.  The most recent result is 'byte-streams' [1], a 
>> library that figures how how to convert between different byte 
>> representations (including character streams), and how to efficiently 
>> transfer bytes between various byte sources and sinks.  The net result is 
>> that you can do something like:
>>
>>   (byte-streams/convert (File. "/tmp/foo") String {:encoding "utf-8"})
>>
>> and get a string representation of the file's contents.  Of course, this 
>> is already possible using 'slurp', but you could also convert it to a 
>> CharSequence, or lazy sequence of ByteBuffers, or pretty much anything else 
>> you can imagine.  This is accomplished by traversing a graph of available 
>> conversions (don't worry, it's memoized), so simply defining a new 
>> conversion from some custom type to (say) a ByteBuffer will transitively 
>> allow you to convert it to any other type.
>>
>> As an aside, this sort of conversion mechanism isn't limited to just byte 
>> representations, but I'm not sure if there's another large collection of 
>> mostly-isomorphic types out there that would benefit from this.  If anyone 
>> has ideas on where else this could be applied, I'd be interested to hear 
>> them.
>>
>> Zach
>>
>> [1] https://github.com/ztellman/byte-streams
>>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] byte-streams: a rosetta stone for all the byte representations the jvm has to offer

[Zach Tellman]

Hey Mike,

Please feel free to appropriate or adapt any code you think might be
useful.  I've signed a CA, it should all be kosher.

As far as an immutable byte-data type, I'm a little skeptical it would be
useful in a wide variety of situations, since a dense array/matrix is going
to be much faster and more predictable than pretty much anything else.  If
you want to guarantee read-only semantics, you have
ByteBuffer.asReadOnly().  If you want a ByteBuffer-like interface atop an
unbounded stream with memory-efficient subsequences, you can check out the
chunked-byte-seqs in Vertigo [1], which is due for release soon.  Something
which behaves like a Clojure vector but is built atop byte[] seems like it
would only be optimal for a very narrow set of use-cases.  Maybe I'm
misunderstanding what you're proposing, though.  Please correct me if
that's the case.

Zach

[1]
https://github.com/ztellman/vertigo/blob/master/src/vertigo/bytes.clj#L212


On Tue, Jul 2, 2013 at 3:19 AM, Mikera  wrote:

> This is cool, thanks Zach!
>
> Another set of mostly-isomporphic types that this could be applied to is
> different matrix/array types in core.matrix. core.matrix already has
> generic conversion mechanisms but they probably aren't as efficient as they
> could be. I'll take a look and see if the same techniques might be
> applicable.
>
> Quick question for you and the crowd: does there exist or should we build
> a standard immutable byte data representation for Clojure?
>
> I think this is often needed: ByteBuffers and byte[] arrays work well
> enough but are mutable. Byte sequences are nice and idiomatic but have a
> lot of overhead, so people are often forced to resort to a variety of other
> techniques. And it would be nice to support some higher level operations on
> such types, e.g. production of efficient (non-copying) immutable
> subsequences.
>
> From a data structure perspective, I'm imagining something like a
> persistent data structure with byte[] data arrays at the lowest level.
>
> Given the amount of data-processing stuff people are doing, it seems like
> a reasonable thing to have in contrib at least?
>
>
> On Saturday, 29 June 2013 18:57:58 UTC+1, Zach Tellman wrote:
>>
>> I've recently been trying to pull out useful pieces from some of my more
>> monolithic libraries.  The most recent result is 'byte-streams' [1], a
>> library that figures how how to convert between different byte
>> representations (including character streams), and how to efficiently
>> transfer bytes between various byte sources and sinks.  The net result is
>> that you can do something like:
>>
>>   (byte-streams/convert (File. "/tmp/foo") String {:encoding "utf-8"})
>>
>> and get a string representation of the file's contents.  Of course, this
>> is already possible using 'slurp', but you could also convert it to a
>> CharSequence, or lazy sequence of ByteBuffers, or pretty much anything else
>> you can imagine.  This is accomplished by traversing a graph of available
>> conversions (don't worry, it's memoized), so simply defining a new
>> conversion from some custom type to (say) a ByteBuffer will transitively
>> allow you to convert it to any other type.
>>
>> As an aside, this sort of conversion mechanism isn't limited to just byte
>> representations, but I'm not sure if there's another large collection of
>> mostly-isomorphic types out there that would benefit from this.  If anyone
>> has ideas on where else this could be applied, I'd be interested to hear
>> them.
>>
>> Zach
>>
>> [1] 
>> https://github.com/**ztellman/byte-streams<https://github.com/ztellman/byte-streams>
>>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/SElE2W7bzis/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
-- 
You received this message because you are subscribed to the G

[ANN] Vertigo: fast, idiomatic C-style structs

[Zach Tellman]

Last year, I gave a talk at the Conj on my attempt to write an AI for the 
board game Go.  Two things I discovered is that it was hard to get 
predictable performance, but even once I made sure I had all the right type 
hints, there was still a lot of room at the bottom for performance 
improvements.  Towards the end [1], I mentioned a few ideas for 
improvements, one of which was simply using ByteBuffers rather than objects 
to host the data.  This would remove all the levels of indirection, giving 
much better cache coherency, and also allow for fast unsynchronized 
mutability when the situation called for it.

So, ten months and several supporting libraries [2] [3] later, here it 
is: https://github.com/ztellman/vertigo

At a high level, this library is useful whenever your datatype has a fixed 
layout and is used more than once.  Depending on your type, it will give 
you moderate to large memory savings, and if you're willing to forgo some 
of core library in favor of Vertigo's operators, you can get significant 
performance gains on batch operations.  And, in the cases where performance 
doesn't matter, it will behave exactly like any other Clojure data 
structure.

I want to point out that something like this would be more or less 
impossible in Java; reading from an offset in a ByteBuffer without the 
compile-time inference and validation provided by this library would be 
pointlessly risky.  There's not a lot of low-level Clojure libraries, but 
there's an increasing amount of production usage where people are using 
Clojure for performance-sensitive work.  I'm looking forward to seeing what 
people do with Vertigo and libraries like it.

Zach

[1] http://www.youtube.com/watch?feature=player_detailpage&v=v5dYE0CMmHQ#t=1828s
[2] https://github.com/ztellman/primitive-math
[3] https://github.com/ztellman/byte-streams

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Clojure: Elegance vs. Performance?

[Zach Tellman]

I've just released Vertigo [1], which I describe in this 
thread: https://groups.google.com/forum/#!topic/clojure/BayfuaqMzvs.  I 
suspect this has some bearing on the conversation.

Zach

[1] https://github.com/ztellman/vertigo

On Tuesday, July 9, 2013 8:11:58 AM UTC-7, Alexander Gunnarson wrote:
>
> Hello everyone! It's great to be here with all you fellow Clojurians. Just 
> so you know, this is my first post on this group, so don't shoot me if it's 
> terrible ;) 
>
> As background, I've been working through SICP and have been loving Scheme. 
> It's almost breathtaking how elegant and clean the code can be (I had some 
> moments 
> like xkcd goes off abou t). Of course, though 
> Scheme is beautiful and simple, everyone knows that it's not especially 
> practical, in general. I love the Lisp paradigms but I'm not really a fan 
> of CL so I did some looking around and stumbled upon Clojure. It seems that 
> Clojure really has a lot going for it between shockingly easy concurrency 
> support and Java interop, among other things. But one thing that's been 
> bothering me is that it seems like to optimize performance in Clojure, you 
> have to sacrifice some elegance.
>
> *Example 1: Tail-call recursion*
>
> *Scheme*
> One example would be tail-call recursion. For instance, normally in Scheme 
> I'd naively implement an iterative exponent function like this:
>
> (define (expt x n)
>
> (cond ((= 0 n) 1)
>
>   ((= 1 n) x)
>
>   (else (expt (* x x) (- n 1)
>
> Pure. Simple. Beautiful. (Not that I'm the best Scheme programmer ever, 
> but to me it looks beautiful, and it conforms well to the base of the 
> problem. You get the point.)
>
> *Clojure*
> Of course, tail-call recursion is not possible with JVM, so Clojure uses a 
> *recur* macro in place of direct recursive function calling. It avoids 
> blowing the stack as quickly but it's still not 100% "mathematically pure" 
> in the way Scheme tends to be.
>
> An added gripe is that the* else *form within *cond *in Clojure uses a 
> keyword, *:else*, instead of the more consistent parenthetical form used 
> in Scheme. I suppose that's to make it less "Lispy." But it just ends up 
> making it a little less elegant.
>
> *Example 2: Type casting*
> *
> *
> Some have said that Clojure can be somewhat slow (as with all Lisps). I'm 
> not sure how true this is, but I stumbled on an example on John Lawrence 
> Aspden's 
> blog.
>  
> He wrote a program to implement Euler's method like so:
>
> *First Solution*
>
> (defn f [t y] (- t y))
>
> (defn solveit [t0 y0 h its]
>   (if (> its 0) 
> (let [t1 (+ t0 h)
>   y1 (+ y0 (* h (f t0 y0)))]
>   (recur t1 y1 h (dec its)))
> [t0 y0 h its]))
>
>
> He points out that "if this was an assembly language program that worked the 
> way you'd expect, each loop would take 7 cycles." So he tests it for Clojure. 
> The result? On his netbook with Clojure 1.4: 2400 cycles. As he says, "We're 
> looking at a slowdown of about 300 times over what we could probably achieve 
> coding in assembler or in C with a good optimizing compiler." That's not 
> surprising, I suppose, but it's still a little disheartening. After all, you 
> want your language to be fast, right? Well, after a few iterations, he 
> manages to reduce the cycles way down - all the way down, in fact, to 37, 
> which is quite a feat. Like so:
>
>
> *Final Solution*
>
> (defn solveit-4 [t0 y0 h its]
>   (let [zero (long 0)]
> (loop [t0 (double t0) y0 (double y0) h (double h) its (long its)]
>   (if (> its zero) 
> (let [t1 (+ t0 h)
>   y1 (+ y0 (* h (- t0 y0)))]
>   (recur t1 y1 h (dec its)))
>   [t0 y0 h its]
>
>
> But the thing is, between the *recur *macro, explicit typecasting, and the 
> *loop* construct, yes, you have a very significant performance increase, but 
> it the code's gotten bigger, much less readable, and much less elegant.
>
>
> *The bottom line*
>
> *
> *
>
> My idea, which is probably very naive, but one which I'm curious about, is:
>
> *Is it possible to have some sort of set of automatic-optimizing macros that 
> work on Clojure code to preserve elegance while maximizing performance?*
>
> *
> *
>
> In theory, it would be kind of an abstraction layer. There would be one file 
> that would store the code that gets read and another output file that stores 
> the code that actually gets evaluated by the REPL, and a set of macros to 
> optimize the "front-end", "abstracted" file into the output, "nuts-and-bolts" 
> file to be evaluated by the REPL. Probably this would be a very intensive 
> process - I don't know. But maybe it's worth the trouble after all to save a 
> ton of programmer-hours by increasing readability.
>
>
> Thoughts?
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this g

Re: Lamina receive-all functionality

[Zach Tellman]

Hi Patrick,

A similar question has been asked in the Aleph mailing list, and you can 
see my answer there: 
https://groups.google.com/forum/#!topic/aleph-lib/SIO9Z8d3tdo.  For future 
reference, you're more likely to get my attention, or the attention of 
someone else who can answer your question, if you post it there.

Zach

On Thursday, July 11, 2013 12:21:20 PM UTC-7, P Martin wrote:
>
> Hi there,
>
> I am working on an application using Lamina where I register several 
> callbacks on a channel with receive-all. At some point in the application, 
> I want to cancel all current callbacks. When this occurs the channel 
> appears to close. Why does this happen? Is this a safety feature? For 
> example:
>
> (def test-channel (channel))
> (def cb1 (receive-all test-channel #(println %)))
> (def cb2 (receive-all test-channel #(println % " two!")))
>
> ... some time later
>
> (cancel-callback test-channel cb1)
> (cancel-callback test-channel cb2)
>
> (enqueue test-channel "something")
> >> :lamina/closed!
>
> Thanks!
> Patrick
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ANN] Proteus: local mutable variables for the masses

[Zach Tellman]

There was some discussion a few days ago about how the lack of local 
mutable variables were harming performance, or possibly elegance, I'm not 
sure.  Regardless, I fixed it: https://github.com/ztellman/proteus

Enjoy!

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] Proteus: local mutable variables for the masses

[Zach Tellman]

Yeah, for safety's sake I need to macroexpand everything, which wipes out 
the &env for internal macros.  There might be a gentler way to do this, but 
it's not obvious to me.  If anyone has suggestions, I'd be interested in 
hearing them.

Zach

On Thursday, July 11, 2013 6:35:21 PM UTC-7, Ben wrote:
>
> Note:
>
> proteus> (defmacro aif [test then else]
>(let [it (first (filter #(not (contains? &env %))
>(cons 'it (map #(symbol (str "it-" %)) 
> (iterate inc 1)]
>  `(let [~it ~test] (if ~it ~then ~else
> #'proteus/aif
> proteus> (aif (get {:x {:y 3}} :x)
>   (aif (get it :y)
>[it it-1]
>it)
>   nil)
> [{:y 3} 3]
> proteus> (let-mutable [x 1]
>   (aif (get {:x {:y 3}} :x)
>(aif (get it :y)
> (set! x [it it-1])
> (set! x it))
>(set! x nil))
>   x)
> CompilerException java.lang.RuntimeException: Unable to resolve symbol: 
> it-1 in this context, compiling:(NO_SOURCE_PATH:1:1) 
> proteus> 
>
>
>
>
> On Thu, Jul 11, 2013 at 6:16 PM, Zach Tellman 
> > wrote:
>
>> There was some discussion a few days ago about how the lack of local 
>> mutable variables were harming performance, or possibly elegance, I'm not 
>> sure.  Regardless, I fixed it: https://github.com/ztellman/proteus
>>
>> Enjoy!
>>
>> -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
>>
>
>
>
> -- 
> Ben Wolfson
> "Human kind has used its intelligence to vary the flavour of drinks, which 
> may be sweet, aromatic, fermented or spirit-based. ... Family and social 
> life also offer numerous other occasions to consume drinks for pleasure." 
> [Larousse, "Drink" entry]
>
>  

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] Proteus: local mutable variables for the masses

[Zach Tellman]

It looks like the macroexpansion code in conditions.free is fairly generic.
 What would you say to putting it into its own library?


On Thu, Jul 11, 2013 at 6:54 PM, Ben Wolfson  wrote:

> On Thu, Jul 11, 2013 at 6:48 PM, Zach Tellman  wrote:
>
>> Yeah, for safety's sake I need to macroexpand everything, which wipes out
>> the &env for internal macros.  There might be a gentler way to do this, but
>> it's not obvious to me.  If anyone has suggestions, I'd be interested in
>> hearing them.
>
>
> You can track the environment through expansion:
>
>
> https://github.com/bwo/conditions/blob/master/test/conditions/free_test.clj#L19
>
> The core.async internals do something similar (or did at the time of
> release, that code has been rewritten, apparently).
>
> You're also missing some binding forms:
>
> proteus> (clojure.pprint/pprint (macroexpand-1 '(let-mutable [x 1] (try (/
> 1 0) (catch Exception x (set! x 2) x)
> (clojure.core/let
>  [x (new proteus.Containers$L 1)]
>  (do
>   (try
>(. clojure.lang.Numbers (divide 1 0))
>(catch Exception (.x x) (do (.set x 2) nil) (.x x)
>
>
> proteus> (clojure.pprint/pprint (macroexpand-1 '(let-mutable [x 1] (letfn
> [(x [] 4)] (set! x 4) x
> (clojure.core/let
>  [x (new proteus.Containers$L 1)]
>  (do (letfn* [x (fn* x ([] 4))] (do (.set x 4) nil) (.x x
>
>
>
>
>>
>> Zach
>>
>>
>> On Thursday, July 11, 2013 6:35:21 PM UTC-7, Ben wrote:
>>
>>> Note:
>>>
>>> proteus> (defmacro aif [test then else]
>>>(let [it (first (filter #(not (contains? &env %))
>>>   ** (cons 'it (map #(symbol (str "it-"
>>> %)) (iterate inc 1)]
>>>  `(let [~it ~test] (if ~it ~then ~else
>>> #'proteus/aif
>>> proteus> (aif (get {:x {:y 3}} :x)
>>>   (aif (get it :y)
>>>[it it-1]
>>>it)
>>>   nil)
>>> [{:y 3} 3]
>>> proteus> (let-mutable [x 1]
>>>   (aif (get {:x {:y 3}} :x)
>>>(aif (get it :y)
>>>   **  (set! x [it it-1])
>>>   **  (set! x it))
>>>(set! x nil))
>>>   x)
>>> CompilerException java.lang.RuntimeException: Unable to resolve symbol:
>>> it-1 in this context, compiling:(NO_SOURCE_PATH:1:1)
>>> proteus>
>>>
>>>
>>>
>>>
>>> On Thu, Jul 11, 2013 at 6:16 PM, Zach Tellman  wrote:
>>>
>>>> There was some discussion a few days ago about how the lack of local
>>>> mutable variables were harming performance, or possibly elegance, I'm not
>>>> sure.  Regardless, I fixed it: 
>>>> https://github.com/**ztellman/proteus<https://github.com/ztellman/proteus>
>>>>
>>>> Enjoy!
>>>>
>>>> --
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clo...@googlegroups.com
>>>>
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+u...@**googlegroups.com
>>>>
>>>> For more options, visit this group at
>>>> http://groups.google.com/**group/clojure?hl=en<http://groups.google.com/group/clojure?hl=en>
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+u...@**googlegroups.com.
>>>>
>>>> For more options, visit 
>>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
>>>> .
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Ben Wolfson
>>> "Human kind has used its intelligence to vary the flavour of drinks,
>>> which may be sweet, aromatic, fermented or spirit-based. ... Family and
>>> social life also offer numerous other occasions to consume drinks for
>>> pleasure." [Larousse, "Drink" entry]
>>>
>>>   --
&

Re: [ANN] Vertigo: fast, idiomatic C-style structs

[Zach Tellman]

If you (vertigo.core/wrap "a-file-name"), that will use mmap under the
covers, so if no one's tried it, it's easy enough to start.

With respect to non-fixed data layouts, that could be supported by a
library which parses the framing information, and then layers Vertigo atop
the actual data.  In effect, that's what Gloss [1] is going to become, so
keep watching the skies.

Zach

[1] https://github.com/ztellman/gloss


On Sun, Jul 14, 2013 at 9:16 PM, kovas boguta wrote:

> This is pretty neat.
>
> Anyone try using this in conjunction with mmap?
>
> It would be nice to have some way to deal with strings & other
> variable-length data.
>
> I'm also curious if its possible to make the analog of this for fressian,
> basically to avoid unpacking objects that are not necessary for the
> computation at hand.
>
>
>
>
>
>
> On Tue, Jul 9, 2013 at 8:56 PM, Zach Tellman  wrote:
>
>> Last year, I gave a talk at the Conj on my attempt to write an AI for the
>> board game Go.  Two things I discovered is that it was hard to get
>> predictable performance, but even once I made sure I had all the right type
>> hints, there was still a lot of room at the bottom for performance
>> improvements.  Towards the end [1], I mentioned a few ideas for
>> improvements, one of which was simply using ByteBuffers rather than objects
>> to host the data.  This would remove all the levels of indirection, giving
>> much better cache coherency, and also allow for fast unsynchronized
>> mutability when the situation called for it.
>>
>> So, ten months and several supporting libraries [2] [3] later, here it
>> is: https://github.com/ztellman/vertigo
>>
>> At a high level, this library is useful whenever your datatype has a
>> fixed layout and is used more than once.  Depending on your type, it will
>> give you moderate to large memory savings, and if you're willing to forgo
>> some of core library in favor of Vertigo's operators, you can get
>> significant performance gains on batch operations.  And, in the cases where
>> performance doesn't matter, it will behave exactly like any other Clojure
>> data structure.
>>
>> I want to point out that something like this would be more or less
>> impossible in Java; reading from an offset in a ByteBuffer without the
>> compile-time inference and validation provided by this library would be
>> pointlessly risky.  There's not a lot of low-level Clojure libraries, but
>> there's an increasing amount of production usage where people are using
>> Clojure for performance-sensitive work.  I'm looking forward to seeing what
>> people do with Vertigo and libraries like it.
>>
>> Zach
>>
>> [1]
>> http://www.youtube.com/watch?feature=player_detailpage&v=v5dYE0CMmHQ#t=1828s
>> [2] https://github.com/ztellman/primitive-math
>> [3] https://github.com/ztellman/byte-streams
>>
>> --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/BayfuaqMzvs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

--

Re: [ANN] Vertigo: fast, idiomatic C-style structs

[Zach Tellman]

Yeah, one nice property of this is that all the underlying objects play
nicely with byte-streams [1], so it's trivial to do something like:

---

(def s (vertigo.core/wrap type "some-huge-file"))

(let [indices (sort-by #(vertigo.core/get-in s [% :foo :bar]) (range (count
s))]
  (doseq [idx indices]
(byte-streams/transfer (nth s idx) sorted-file {:append true})))

---

If there's framing information we won't be able to do this quite as
efficiently, but it shouldn't be too much more expensive.


[1] https://github.com/ztellman/byte-streams


On Mon, Jul 15, 2013 at 9:48 PM, kovas boguta wrote:

> Interesting. This seems like a pretty promising direction for the bottom
> of the big-data stack.
>
> A use case on my mind is sorting a big list of datastructures by key (or
> some set of keys/paths) .
>
> Once the data gets big, you need to do an external sort, which means tons
> of serialization round trips if implemented naively. Being able to just
> pluck out the values you need really helps in that case. Besides saving on
> the serialization overhead, it also cuts down on memory which means you can
> sort much bigger segments at a time, and complete the overall sort in fewer
> passes.
>
>
>
>
>
>
>
> On Mon, Jul 15, 2013 at 1:40 PM, Zach Tellman  wrote:
>
>> If you (vertigo.core/wrap "a-file-name"), that will use mmap under the
>> covers, so if no one's tried it, it's easy enough to start.
>>
>> With respect to non-fixed data layouts, that could be supported by a
>> library which parses the framing information, and then layers Vertigo atop
>> the actual data.  In effect, that's what Gloss [1] is going to become, so
>> keep watching the skies.
>>
>> Zach
>>
>> [1] https://github.com/ztellman/gloss
>>
>>
>> On Sun, Jul 14, 2013 at 9:16 PM, kovas boguta wrote:
>>
>>> This is pretty neat.
>>>
>>> Anyone try using this in conjunction with mmap?
>>>
>>> It would be nice to have some way to deal with strings & other
>>> variable-length data.
>>>
>>> I'm also curious if its possible to make the analog of this for
>>> fressian, basically to avoid unpacking objects that are not necessary for
>>> the computation at hand.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jul 9, 2013 at 8:56 PM, Zach Tellman  wrote:
>>>
>>>> Last year, I gave a talk at the Conj on my attempt to write an AI for
>>>> the board game Go.  Two things I discovered is that it was hard to get
>>>> predictable performance, but even once I made sure I had all the right type
>>>> hints, there was still a lot of room at the bottom for performance
>>>> improvements.  Towards the end [1], I mentioned a few ideas for
>>>> improvements, one of which was simply using ByteBuffers rather than objects
>>>> to host the data.  This would remove all the levels of indirection, giving
>>>> much better cache coherency, and also allow for fast unsynchronized
>>>> mutability when the situation called for it.
>>>>
>>>> So, ten months and several supporting libraries [2] [3] later, here it
>>>> is: https://github.com/ztellman/vertigo
>>>>
>>>> At a high level, this library is useful whenever your datatype has a
>>>> fixed layout and is used more than once.  Depending on your type, it will
>>>> give you moderate to large memory savings, and if you're willing to forgo
>>>> some of core library in favor of Vertigo's operators, you can get
>>>> significant performance gains on batch operations.  And, in the cases where
>>>> performance doesn't matter, it will behave exactly like any other Clojure
>>>> data structure.
>>>>
>>>> I want to point out that something like this would be more or less
>>>> impossible in Java; reading from an offset in a ByteBuffer without the
>>>> compile-time inference and validation provided by this library would be
>>>> pointlessly risky.  There's not a lot of low-level Clojure libraries, but
>>>> there's an increasing amount of production usage where people are using
>>>> Clojure for performance-sensitive work.  I'm looking forward to seeing what
>>>> people do with Vertigo and libraries like it.
>>>>
>>>> Zach
>>>>
>>>> [1]
>>>> http://www.youtube.com/watch?feature=player_detailpage&v=v5dYE0CMmHQ#t=1828s
>>>> [2] https://github.com/ztellma

Re: [ANN] Vertigo: fast, idiomatic C-style structs

[Zach Tellman]

I actually haven't applied it yet.  I'll post results once I have.


On Wed, Jul 17, 2013 at 11:49 AM, Daniel  wrote:

> How did this affect performance in your Go AI?
>
>
> On Tuesday, July 9, 2013 10:56:03 PM UTC-5, Zach Tellman wrote:
>>
>> Last year, I gave a talk at the Conj on my attempt to write an AI for the
>> board game Go.  Two things I discovered is that it was hard to get
>> predictable performance, but even once I made sure I had all the right type
>> hints, there was still a lot of room at the bottom for performance
>> improvements.  Towards the end [1], I mentioned a few ideas for
>> improvements, one of which was simply using ByteBuffers rather than objects
>> to host the data.  This would remove all the levels of indirection, giving
>> much better cache coherency, and also allow for fast unsynchronized
>> mutability when the situation called for it.
>>
>> So, ten months and several supporting libraries [2] [3] later, here it
>> is: 
>> https://github.com/**ztellman/vertigo<https://github.com/ztellman/vertigo>
>>
>> At a high level, this library is useful whenever your datatype has a
>> fixed layout and is used more than once.  Depending on your type, it will
>> give you moderate to large memory savings, and if you're willing to forgo
>> some of core library in favor of Vertigo's operators, you can get
>> significant performance gains on batch operations.  And, in the cases where
>> performance doesn't matter, it will behave exactly like any other Clojure
>> data structure.
>>
>> I want to point out that something like this would be more or less
>> impossible in Java; reading from an offset in a ByteBuffer without the
>> compile-time inference and validation provided by this library would be
>> pointlessly risky.  There's not a lot of low-level Clojure libraries, but
>> there's an increasing amount of production usage where people are using
>> Clojure for performance-sensitive work.  I'm looking forward to seeing what
>> people do with Vertigo and libraries like it.
>>
>> Zach
>>
>> [1] http://www.youtube.com/**watch?feature=player_**
>> detailpage&v=v5dYE0CMmHQ#t=**1828s<http://www.youtube.com/watch?feature=player_detailpage&v=v5dYE0CMmHQ#t=1828s>
>> [2] 
>> https://github.com/ztellman/**primitive-math<https://github.com/ztellman/primitive-math>
>> [3] 
>> https://github.com/**ztellman/byte-streams<https://github.com/ztellman/byte-streams>
>>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/BayfuaqMzvs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ANN] byte-transforms: methods for hashing, compressing, and encoding bytes

[Zach Tellman]

https://github.com/ztellman/byte-transforms

This is just a thin wrapper over byte-streams [1] and some best-in-class 
hashing and compression algorithms, but I figure there are at least a few 
people who, like me, have put off using Snappy or MurmurHash because they 
didn't feel like crawling through the javadocs.  Enjoy!

Zach

[1] https://github.com/ztellman/byte-streams

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ANN] byte-transforms: methods for hashing, compressing, and encoding bytes

[Zach Tellman]

This is just a thin wrapper over byte-streams [1] and some best-in-class 
hash and compression algorithms, but I figure there are at least a few 
people out there who'd like to use Snappy or MurmurHash but don't want to 
crawl through javadocs.  Enjoy.

Zach

[1] https://github.com/ztellman/byte-streams

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ANN] immutable-bitset: smalller, faster integer sets

[Zach Tellman]

https://github.com/ztellman/immutable-bitset

There's not much to describe here, this provides an implementation of an 
integer-only set which can take up three orders of magnitude less memory 
under certain conditions.  I needed this to implement a Bloom filter, but I 
figured it had applications elsewhere.  If anyone has questions, I'd be 
happy to answer them.

Zach

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] Vertigo: fast, idiomatic C-style structs

[Zach Tellman]

Hi Ezra,

This is admittedly a little confusing, but you're hinting 's' with the type
of the *element*.  Here you've created a sequence containing a single
'ints-and-floats' struct, so you'd want to do this:

user> (v/get-in s [0 :floats 4])
4.0
user> (v/get-in s [0])
{:ints (0 1 2 3 4 5 6 7 8 9), :floats (0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0
9.0)}

I'll try to make this clearer in the documentation.  Let me know if you
have any other questions.

Zach


On Wed, Jul 31, 2013 at 10:10 AM, Ezra Lee  wrote:

> Hi,
> I'm trying out vertigo and hoping you can help me figure out what I am
> missing, I get an error when I use get-in:
>
> ; nREPL 0.1.8-preview
> user> (use 'vertigo.structs)
> nil
> user> (def-typed-struct ints-and-floats :ints (array uint32 10) :floats
> (array float32 10))
> #'user/ints-and-floats
> user> (def x {:ints (range 10) :floats (map float (range 10))})
> #'user/x
> user> (require '[vertigo.core :as v])
> nil
> user> (def ^:ints-and-floats s (v/marshal-seq ints-and-floats [x]))
> #'user/s
> user> (v/get-in s [:floats 4])
> IllegalArgumentException Invalid field '4' for type ints-and-floats
>  vertigo.core/validate-lookup (core.clj:177)
> user> (v/get-in s [4 :floats])
> IllegalArgumentException   java.nio.Buffer.position (Buffer.java:216)
>
> Thanks,
> Ezra
>
> On Tuesday, July 9, 2013 11:56:03 PM UTC-4, Zach Tellman wrote:
>>
>> Last year, I gave a talk at the Conj on my attempt to write an AI for the
>> board game Go.  Two things I discovered is that it was hard to get
>> predictable performance, but even once I made sure I had all the right type
>> hints, there was still a lot of room at the bottom for performance
>> improvements.  Towards the end [1], I mentioned a few ideas for
>> improvements, one of which was simply using ByteBuffers rather than objects
>> to host the data.  This would remove all the levels of indirection, giving
>> much better cache coherency, and also allow for fast unsynchronized
>> mutability when the situation called for it.
>>
>> So, ten months and several supporting libraries [2] [3] later, here it
>> is: 
>> https://github.com/**ztellman/vertigo<https://github.com/ztellman/vertigo>
>>
>> At a high level, this library is useful whenever your datatype has a
>> fixed layout and is used more than once.  Depending on your type, it will
>> give you moderate to large memory savings, and if you're willing to forgo
>> some of core library in favor of Vertigo's operators, you can get
>> significant performance gains on batch operations.  And, in the cases where
>> performance doesn't matter, it will behave exactly like any other Clojure
>> data structure.
>>
>> I want to point out that something like this would be more or less
>> impossible in Java; reading from an offset in a ByteBuffer without the
>> compile-time inference and validation provided by this library would be
>> pointlessly risky.  There's not a lot of low-level Clojure libraries, but
>> there's an increasing amount of production usage where people are using
>> Clojure for performance-sensitive work.  I'm looking forward to seeing what
>> people do with Vertigo and libraries like it.
>>
>> Zach
>>
>> [1] http://www.youtube.com/**watch?feature=player_**
>> detailpage&v=v5dYE0CMmHQ#t=**1828s<http://www.youtube.com/watch?feature=player_detailpage&v=v5dYE0CMmHQ#t=1828s>
>> [2] 
>> https://github.com/ztellman/**primitive-math<https://github.com/ztellman/primitive-math>
>> [3] 
>> https://github.com/**ztellman/byte-streams<https://github.com/ztellman/byte-streams>
>>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/BayfuaqMzvs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
-- 
You received this message because

Re: [ANN] Vertigo: fast, idiomatic C-style structs

[Zach Tellman]

Actually, looking at the readme, I can see the code you were trying to use.
 Sorry, I'm not sure how I didn't catch that before, but I've fixed it.

Zach


On Wed, Jul 31, 2013 at 10:17 AM, Zach Tellman  wrote:

> Hi Ezra,
>
> This is admittedly a little confusing, but you're hinting 's' with the
> type of the *element*.  Here you've created a sequence containing a single
> 'ints-and-floats' struct, so you'd want to do this:
>
> user> (v/get-in s [0 :floats 4])
> 4.0
> user> (v/get-in s [0])
> {:ints (0 1 2 3 4 5 6 7 8 9), :floats (0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0
> 9.0)}
>
> I'll try to make this clearer in the documentation.  Let me know if you
> have any other questions.
>
> Zach
>
>
> On Wed, Jul 31, 2013 at 10:10 AM, Ezra Lee  wrote:
>
>> Hi,
>> I'm trying out vertigo and hoping you can help me figure out what I am
>> missing, I get an error when I use get-in:
>>
>> ; nREPL 0.1.8-preview
>> user> (use 'vertigo.structs)
>> nil
>> user> (def-typed-struct ints-and-floats :ints (array uint32 10) :floats
>> (array float32 10))
>> #'user/ints-and-floats
>> user> (def x {:ints (range 10) :floats (map float (range 10))})
>> #'user/x
>> user> (require '[vertigo.core :as v])
>> nil
>> user> (def ^:ints-and-floats s (v/marshal-seq ints-and-floats [x]))
>> #'user/s
>> user> (v/get-in s [:floats 4])
>> IllegalArgumentException Invalid field '4' for type ints-and-floats
>>  vertigo.core/validate-lookup (core.clj:177)
>> user> (v/get-in s [4 :floats])
>> IllegalArgumentException   java.nio.Buffer.position (Buffer.java:216)
>>
>> Thanks,
>> Ezra
>>
>> On Tuesday, July 9, 2013 11:56:03 PM UTC-4, Zach Tellman wrote:
>>>
>>> Last year, I gave a talk at the Conj on my attempt to write an AI for
>>> the board game Go.  Two things I discovered is that it was hard to get
>>> predictable performance, but even once I made sure I had all the right type
>>> hints, there was still a lot of room at the bottom for performance
>>> improvements.  Towards the end [1], I mentioned a few ideas for
>>> improvements, one of which was simply using ByteBuffers rather than objects
>>> to host the data.  This would remove all the levels of indirection, giving
>>> much better cache coherency, and also allow for fast unsynchronized
>>> mutability when the situation called for it.
>>>
>>> So, ten months and several supporting libraries [2] [3] later, here it
>>> is: 
>>> https://github.com/**ztellman/vertigo<https://github.com/ztellman/vertigo>
>>>
>>> At a high level, this library is useful whenever your datatype has a
>>> fixed layout and is used more than once.  Depending on your type, it will
>>> give you moderate to large memory savings, and if you're willing to forgo
>>> some of core library in favor of Vertigo's operators, you can get
>>> significant performance gains on batch operations.  And, in the cases where
>>> performance doesn't matter, it will behave exactly like any other Clojure
>>> data structure.
>>>
>>> I want to point out that something like this would be more or less
>>> impossible in Java; reading from an offset in a ByteBuffer without the
>>> compile-time inference and validation provided by this library would be
>>> pointlessly risky.  There's not a lot of low-level Clojure libraries, but
>>> there's an increasing amount of production usage where people are using
>>> Clojure for performance-sensitive work.  I'm looking forward to seeing what
>>> people do with Vertigo and libraries like it.
>>>
>>> Zach
>>>
>>> [1] http://www.youtube.com/**watch?feature=player_**
>>> detailpage&v=v5dYE0CMmHQ#t=**1828s<http://www.youtube.com/watch?feature=player_detailpage&v=v5dYE0CMmHQ#t=1828s>
>>> [2] 
>>> https://github.com/ztellman/**primitive-math<https://github.com/ztellman/primitive-math>
>>> [3] 
>>> https://github.com/**ztellman/byte-streams<https://github.com/ztellman/byte-streams>
>>>
>>  --
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+un

ANN: clj-tuple, efficient small collections

[Zach Tellman]

I just pulled some code I wrote while trying to optimize 'memoize' into its 
own library: https://github.com/ztellman/clj-tuple.  It only has the one 
function, so I assume no one will need too much explanation.  However, 
there may still be room for performance improvements, so if anyone wants to 
take a stab, pull requests are welcome.

Zach

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: clj-tuple, efficient small collections

[Zach Tellman]

I don't think so, even the existence of all the Tuple* types are an 
implementation detail, and you'd need to hint it as the right one to get 
sane performance.  (nth t n) has good performance, you should prefer that.

On Saturday, August 24, 2013 8:15:40 PM UTC-7, Ben wrote:
>
> Are the element names .e0, .e1 etc. considered part of the public 
> interface of tuple objects?
>
>
> On Sat, Aug 24, 2013 at 7:38 PM, Zach Tellman 
> > wrote:
>
>> I just pulled some code I wrote while trying to optimize 'memoize' into 
>> its own library: https://github.com/ztellman/clj-tuple.  It only has the 
>> one function, so I assume no one will need too much explanation.  However, 
>> there may still be room for performance improvements, so if anyone wants to 
>> take a stab, pull requests are welcome.
>>
>> Zach
>>
>> -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> -- 
> Ben Wolfson
> "Human kind has used its intelligence to vary the flavour of drinks, which 
> may be sweet, aromatic, fermented or spirit-based. ... Family and social 
> life also offer numerous other occasions to consume drinks for pleasure." 
> [Larousse, "Drink" entry]
>
>  

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: clj-tuple, efficient small collections

[Zach Tellman]

in my microbenchmarks I've found it to be consistently faster to unroll a 
fixed number of elements rather than iterate over them.  The difference 
might not be large enough to matter for many people's use cases, but the 
stated goal is to make a fast collection, so it's worthwhile to use the 
best known approach.

Currently tuples mimic lists, so conj adds onto the beginning, and does not 
return another tuple.  'rest', however, does return a tuple.  Also, since 
my intent wasn't to emulate vectors, 'get', 'assoc', and calling as 
function don't work with tuples.  This could obviously change, but I'd be 
curious to know why it makes a difference in someone's code.

Also, I played around with returning both a list or a vector in the 
unbounded case, but the performance is too different in each of these 
cases.  As I point out in the readme, vectors are *significantly* slower to 
create, which would make the performance of 'tuple' too inconsistent for my 
liking.  The TupleN solution I came up with was the best compromise I could 
think of.

Zach



On Saturday, August 24, 2013 10:47:11 PM UTC-7, Jozef Wagner wrote:
>
> Hi,
>
> I've did something similar for CLJS, see 
> http://dev.clojure.org/jira/browse/CLJS-453 and 
> https://groups.google.com/forum/#!searchin/clojure/arrayvector/clojure/yDvTRP0gYLA/Tb5MJC8Z0K8J
>
> In CLJS, it is comparable in performance to use array instead of 
> specialized type for each arity. Did you check if it is not the same in 
> Clojure?
>
> If I conjoin to the tuple, does it return a tuple too? Also instead of 
> tupleN, you could fall back to the PersistentVector.
>
> JW
>
>
> On Sun, Aug 25, 2013 at 4:38 AM, Zach Tellman 
> > wrote:
>
>> I just pulled some code I wrote while trying to optimize 'memoize' into 
>> its own library: https://github.com/ztellman/clj-tuple.  It only has the 
>> one function, so I assume no one will need too much explanation.  However, 
>> there may still be room for performance improvements, so if anyone wants to 
>> take a stab, pull requests are welcome.
>>
>> Zach
>>
>> -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: clj-tuple, efficient small collections

[Zach Tellman]

I'm not sure I understand the motivation behind what you're suggesting.
 Why not just (nth tuple 0)?


On Mon, Aug 26, 2013 at 10:38 AM, Kevin Downey  wrote:

> A Tuple protocol that defines get0 get1 get3 etc for fast element access
> that doesn't tie you to using field names might be a good idea.
>
> On 8/25/13 9:35 AM, Zach Tellman wrote:
> > I don't think so, even the existence of all the Tuple* types are an
> > implementation detail, and you'd need to hint it as the right one to get
> > sane performance.  (nth t n) has good performance, you should prefer
> that.
> >
> > On Saturday, August 24, 2013 8:15:40 PM UTC-7, Ben wrote:
> >>
> >> Are the element names .e0, .e1 etc. considered part of the public
> >> interface of tuple objects?
> >>
> >>
> >> On Sat, Aug 24, 2013 at 7:38 PM, Zach Tellman  
> >>> wrote:
> >>
> >>> I just pulled some code I wrote while trying to optimize 'memoize' into
> >>> its own library: https://github.com/ztellman/clj-tuple.  It only has
> the
> >>> one function, so I assume no one will need too much explanation.
>  However,
> >>> there may still be room for performance improvements, so if anyone
> wants to
> >>> take a stab, pull requests are welcome.
> >>>
> >>> Zach
> >>>
> >>> --
> >>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "Clojure" group.
> >>> To post to this group, send email to clo...@googlegroups.com
> 
> >>> Note that posts from new members are moderated - please be patient with
> >>> your first post.
> >>> To unsubscribe from this group, send email to
> >>> clojure+u...@googlegroups.com 
> >>> For more options, visit this group at
> >>> http://groups.google.com/group/clojure?hl=en
> >>> ---
> >>> You received this message because you are subscribed to the Google
> Groups
> >>> "Clojure" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send
> an
> >>> email to clojure+u...@googlegroups.com .
> >>> For more options, visit https://groups.google.com/groups/opt_out.
> >>>
> >>
> >>
> >>
> >> --
> >> Ben Wolfson
> >> "Human kind has used its intelligence to vary the flavour of drinks,
> which
> >> may be sweet, aromatic, fermented or spirit-based. ... Family and social
> >> life also offer numerous other occasions to consume drinks for
> pleasure."
> >> [Larousse, "Drink" entry]
> >>
> >>
> >
>
>
> --
> And what is good, Phaedrus,
> And what is not good—
> Need we ask anyone to tell us these things?
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ANN: clj-tuple, efficient small collections

[Zach Tellman]

I've updated the library so tuples behave like vectors, rather than lists. 
 The macros have gotten kind of elephantine, but since I've already added 
every conceivable Clojure interface at least they won't get any bigger.  If 
there are any lingering gaps, though, please let me know.

Zach 

On Saturday, August 24, 2013 7:38:25 PM UTC-7, Zach Tellman wrote:
>
> I just pulled some code I wrote while trying to optimize 'memoize' into 
> its own library: https://github.com/ztellman/clj-tuple.  It only has the 
> one function, so I assume no one will need too much explanation.  However, 
> there may still be room for performance improvements, so if anyone wants to 
> take a stab, pull requests are welcome.
>
> Zach
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ANN] riddley: code-walking without caveats

[Zach Tellman]

When I announced Proteus [1], it was rightfully pointed out that it didn't 
play nicely with macros which rely on &env, as well as a few forms like 
'letfn' that I hadn't explicitly handled.  This flaw has been shared by 
pretty much every library of this sort, and since this is a problem I've 
half-solved two or three times already, I figured something more general 
and lasting was in order.

The resulting library is called Riddley [2].  For obvious reasons, I've 
named it after a book which is written entirely in a barely-readable pidgin 
dialect. While there may be lingering issues, it's good enough to replace 
the code-walking mechanism in Proteus, which I think makes it the best game 
in town right now.  Bug reports and pull requests are welcome.

Zach

[1] 
https://groups.google.com/forum/#!searchin/clojure/proteus/clojure/7HNNiJJTte4/iMBWn8p6tZAJ
[2] https://github.com/ztellman/riddley

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

Hey Konrad, you can maybe speak with more authority as to what tools.macro
does and doesn't provide, but my reading of it is that it does expression
walking to prevent bound variables from being incorrectly
symbol-macroexpanded.  This seems only important in the context of symbol
macros, however; if you don't use symbol macros it's functionally
equivalent to clojure.walk/macroexpand-all.

This means that it suffers from all the same issues mentioned in Riddley's
readme, namely no &env and no expansion of inlined functions.  The code
walking is also only used to do expansion, no generic code walking
mechanism is exposed for more general transformations a la Proteus.

Hope that helps,
Zach


On Mon, Sep 2, 2013 at 11:20 PM, Konrad Hinsen <
googlegro...@khinsen.fastmail.net> wrote:

> --On 2 septembre 2013 13:49:01 -0700 Zach Tellman 
> wrote:
>
>  The resulting library is called Riddley [2].  For obvious reasons, I've
>> named it after a book which is written entirely in a barely-readable
>> pidgin dialect. While there may be lingering issues, it's good enough to
>> replace the code-walking mechanism in Proteus, which I think makes it the
>> best game in town right now.  Bug reports and pull requests are welcome.
>>
>
> How does this compare to mexpand-all in clojure.tools.macro?
>
> Konrad.
>
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscribe@**googlegroups.com
> For more options, visit this group at
> http://groups.google.com/**group/clojure?hl=en<http://groups.google.com/group/clojure?hl=en>
> --- You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/**
> topic/clojure/a68aThpvP4o/**unsubscribe<https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe>
> .
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscribe@**googlegroups.com
> .
> For more options, visit 
> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
> .
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

I see.  This is honestly something I hadn't considered, but since Riddley
actually uses the Clojure compiler internals to track locals, this would be
as simple as a (when-not (contains? (riddley.compiler/locals) (first expr))
...) guard in the macroexpansion.  As Ben points out, using the compiler
this way is the only way to make sure that locals are consistent
everywhere, rather than just in your own targeted use to track shadowing.

Hope that helps,
Zach


On Tue, Sep 3, 2013 at 12:41 PM, Konrad Hinsen <
googlegro...@khinsen.fastmail.net> wrote:

> --On 3 septembre 2013 02:08:23 -0700 Zach Tellman 
> wrote:
>
>  Hey Konrad, you can maybe speak with more authority as to what
>> tools.macro does and doesn't provide, but my reading of it is that it
>> does expression walking to prevent bound variables from being incorrectly
>> symbol-macroexpanded.  This seems only important in the context of
>> symbol macros, however; if you don't use symbol macros it's functionally
>> equivalent to clojure.walk/macroexpand-all.
>>
>
> Not quite. It expands only terms that are evaluated, using a built-in
> table of special forms, and it allows local macro definitions (macrolet).
> But most importantly, it tracks local bindings and expands only macros that
> are not shadowed. So if you have
>
>  (defmacro foo [] ...)
>  (let [foo (fn [] ...)]
> (foo 'bar))
>
> the form (foo 'bar) is not expanded because its local binding is a
> function. The version in clojure.walk doesn't take this into account, and
> can therefore produce incorrect code, which is a major pain to debug. I
> know because it happened to me, that's why I ended up writing my own macro
> expander. And that's why I wonder how riddley handled this.
>
>
> Konrad.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscribe@**googlegroups.com
> For more options, visit this group at
> http://groups.google.com/**group/clojure?hl=en<http://groups.google.com/group/clojure?hl=en>
> --- You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/**
> topic/clojure/a68aThpvP4o/**unsubscribe<https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe>
> .
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscribe@**googlegroups.com
> .
> For more options, visit 
> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
> .
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[ANN] clj-leveldb, idiomatic LevelDB bindings from Factual

[Zach Tellman]

In the first of what I hope will be many annoucements, we're open sourcing 
a library we've found useful at Factual: 
https://github.com/Factual/clj-leveldb.  This is just a simple wrapper 
around LevelDB, which is an in-process persistent k/v store from Google.

If anyone has questions, I'm happy to answer them.

Zach

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

I'm not sure what you mean by "complete recursive expansion".  Could you
expand on that?

As for replicating the behavior of the compiler, I'd assert that unless
&env is precisely what it would be without ahead of time macroexpansion,
the compiler's behavior isn't being replicated.  The tools.macro library
emulates an aspect of its behavior, certainly, and the fact that Clojure's
existed this long without anyone doing something like this indicates that
maybe this isn't such a huge omission, but without there remains an uncanny
valley.


On Wed, Sep 4, 2013 at 12:09 AM, Konrad Hinsen <
googlegro...@khinsen.fastmail.net> wrote:

> Zach Tellman writes:
>
>  > I see.  This is honestly something I hadn't considered, but since
>  > Riddley actually uses the Clojure compiler internals to track
>  > locals, this would be as simple as a (when-not (contains?
>  > (riddley.compiler/locals) (first expr)) ...) guard in the
>  > macroexpansion.
>
> If you don't need complete recursive expansion, that's indeed an
> approach worth exploring. For tools.macros that's not an option
> because the compiler knows nothing about local macros and symbol
> macros.
>
>  > As Ben points out, using the compiler this way is the only way to
>  > make sure that locals are consistent everywhere, rather than just
>  > in your own targeted use to track shadowing.
>
> Well, either you use the compiler or you replicate what it does.  For
> tools.macro I had to choose the second approach. I don't claim it has
> no bugs, I just claim I haven't had any bug reports ;-) (until today
> at least).
>
> Konrad
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

So "complete recursive expansion" is postwalk macroexpansion?  It seems
like that could break anaphoric macros, and likely others.  A macro has the
option of calling macroexpand-all on its own contents if it wants only
special forms, but it shouldn't be forced to take only special forms.

Also, here's a sketch of how you could do symbol macros using Riddley:
https://gist.github.com/ztellman/6439318.  Please let me know if I'm
missing something w.r.t. how symbol macros are done in tools.macros.

Zach


On Wed, Sep 4, 2013 at 2:16 AM, Konrad Hinsen <
googlegro...@khinsen.fastmail.net> wrote:

> **
> On Wed, Sep 4, 2013, at 09:25 AM, Zach Tellman wrote:
>
> I'm not sure what you mean by "complete recursive expansion".  Could
> you expand
> on that?
>
> Completely ;-)
>
> By complete recursive expansion I mean that you get a form that is fully
> reduced to
> the core language, i.e. it contains no more macro applications at any
> level.
>
> If you leave macro expansion to the compiler, it does it when it arrives
> at the
> macro during evaluation. Then it does a plain non-recursive macroexpand
> and goes on
> evaluating. Any macro thus has access to the unexpanded contents of its
> form, but
> not to what it eventually expands to. For many applications that's just
> fine, which
> is why this approach has been the default in the Lisp world for a long
> time.
>
> As for replicating the behavior of the compiler, I'd assert that
> unless &env is
> precisely what it would be without ahead of time macroexpansion, the
> compiler's
> behavior isn't being replicated.
>
> I agree. tools.macro predates &env, which is why it is not supported.
> Since I have
> never need &env support and nobody ever asked for it (before now), it's
> not there.
> I don't see any reason why it couldn't be supported.
>
> Konrad.
>
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

Actually, postwalk expansion (if that is in fact what you were describing)
would ignore any binding forms created by the outer macro.  This means that
something simple like:

(defmacro with-db [db & body]
  `(with-open [~db (create-db)]
 ~@body))

would be expanded without any knowledge of the 'db' local variable, since
that would only get turned into a let form later.  Prewalk is pretty much
the only way this works.


On Wed, Sep 4, 2013 at 9:50 AM, Ben Wolfson  wrote:

> Postwalk expansion would break macros that inspect their argument forms
> for e.g. writing special-purpose queries, if they *also* adopt the symbols
> "and" and "or" for conjunction or disjunction. Korma's "where", for
> instance, does this; one can write
>
> (select my-table (where (and (...) (...
>
> And the "where" detects the "and".
>
> Arguably this is wrongheaded behavior from the get-go (it can be somewhat
> confusing and makes it necessary to use something like clojure.core/and
> within "where" if you want normal clojure-land "and" semantics), but it's a
> style of non-anaphoric macro that relies on receiving an unexpanded form.
>
>
> On Wed, Sep 4, 2013 at 9:27 AM, Zach Tellman  wrote:
>
>> So "complete recursive expansion" is postwalk macroexpansion?  It seems
>> like that could break anaphoric macros, and likely others.  A macro has the
>> option of calling macroexpand-all on its own contents if it wants only
>> special forms, but it shouldn't be forced to take only special forms.
>>
>> Also, here's a sketch of how you could do symbol macros using Riddley:
>> https://gist.github.com/ztellman/6439318.  Please let me know if I'm
>> missing something w.r.t. how symbol macros are done in tools.macros.
>>
>> Zach
>>
>>
>> On Wed, Sep 4, 2013 at 2:16 AM, Konrad Hinsen <
>> googlegro...@khinsen.fastmail.net> wrote:
>>
>>> **
>>> On Wed, Sep 4, 2013, at 09:25 AM, Zach Tellman wrote:
>>>
>>> I'm not sure what you mean by "complete recursive expansion".  Could
>>> you expand
>>> on that?
>>>
>>> Completely ;-)
>>>
>>> By complete recursive expansion I mean that you get a form that is fully
>>> reduced to
>>> the core language, i.e. it contains no more macro applications at any
>>> level.
>>>
>>> If you leave macro expansion to the compiler, it does it when it arrives
>>> at the
>>> macro during evaluation. Then it does a plain non-recursive macroexpand
>>> and goes on
>>> evaluating. Any macro thus has access to the unexpanded contents of its
>>> form, but
>>> not to what it eventually expands to. For many applications that's just
>>> fine, which
>>> is why this approach has been the default in the Lisp world for a long
>>> time.
>>>
>>> As for replicating the behavior of the compiler, I'd assert that
>>> unless &env is
>>> precisely what it would be without ahead of time macroexpansion, the
>>> compiler's
>>> behavior isn't being replicated.
>>>
>>> I agree. tools.macro predates &env, which is why it is not supported.
>>> Since I have
>>> never need &env support and nobody ever asked for it (before now), it's
>>> not there.
>>> I don't see any reason why it couldn't be supported.
>>>
>>> Konrad.
>>>
>>>
>>> --
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Clojure" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> clojure+unsubscr...@googlegroups.com.
>>>
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>  --
>> --
>> You receive

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

I guess I'm confused, then.  You contrast "complete recursive expansion"
with what the compiler does, and then say it's recursive prewalk expansion,
which is exactly what the compiler does.  Can you clarify the difference
between what you're doing and what the compiler does?


On Wed, Sep 4, 2013 at 11:54 AM, Konrad Hinsen <
googlegro...@khinsen.fastmail.net> wrote:

> --On 4 septembre 2013 09:27:12 -0700 Zach Tellman 
> wrote:
>
>  So "complete recursive expansion" is postwalk macroexpansion?  It seems
>> like that could break anaphoric macros, and likely others.  A macro has
>> the option of calling macroexpand-all on its own contents if it wants
>> only special forms, but it shouldn't be forced to take only special forms.
>>
>
> Recursive macro expansion still works from outside in, so each macro gets
> to see the unexpanded form. It's only after the macro has done its
> transformation that the inner forms get expanded.
>
>
>  Also, here's a sketch of how you could do symbol macros using
>> Riddley: 
>> https://gist.github.**com/ztellman/6439318<https://gist.github.com/ztellman/6439318>.
>>  Please let me know
>> if I'm missing something w.r.t. how symbol macros are done in
>> tools.macros.
>>
>
> It's on my reading list for tomorrow!
>
>
> Konrad.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscribe@**googlegroups.com
> For more options, visit this group at
> http://groups.google.com/**group/clojure?hl=en<http://groups.google.com/group/clojure?hl=en>
> --- You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/**
> topic/clojure/a68aThpvP4o/**unsubscribe<https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe>
> .
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscribe@**googlegroups.com
> .
> For more options, visit 
> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
> .
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Handling name collisions with clojure.core

[Zach Tellman]

It is probably instructive to look at how (use-primitive-operators) works 
in primitive-math [1], though maybe not something you want to emulate.  The 
basic mechanism is pretty simple: use 'ns-unmap' to get rid of the 
operators you want to shadow, and bring in the operators from the alternate 
namespace.

Unfortunately, the next time you load the namespace, you'll get the same 
mess of collision warnings you were trying to avoid, because you haven't 
added the ':refer-clojure :exclude' clause to the ns declaration.  To get 
around this, use-primitive-operator hijacks 'ns' via alter-var-root so that 
this is automatically added if the alternate operators are detected [2], 
but otherwise leaves it unchanged.  This hijacking is undone if 
clojure.core is ever reloaded (usually by a :reload-all somewhere), 
however.  This could be fixed by adding a watcher to #'clojure.core/ns and 
forcing it back whenever it changes, but I haven't done that yet because it 
doesn't really affect me, and it feels hokey enough already.

Despite all that, though, it works pretty well.  Feel free to use this 
approach if you like, or create a less questionable variant if you can 
think of one.

Zach

[1] 
https://github.com/ztellman/primitive-math/blob/master/src/primitive_math.clj#L154
[2] 
https://github.com/ztellman/primitive-math/blob/master/src/primitive_math.clj#L135

On Wednesday, September 4, 2013 6:22:08 PM UTC-7, Mikera wrote:
>
> Hi all,
>
> While building the API for core.matrix, I've fun into a few cases where 
> the "best" name is a direct clash with clojure.core.
>
> Examples are "+", "zero?", "vector?", "=="
>
> In many of these cases, the core.matrix behaviour is a natural extension 
> of the clojure.core function (i.e. it extends the same functionality to 
> arbitrary N-dimensional arrays). 
>
> I'm not very happy with any of the options I can see for handling this:
>
> A) Use the good names in the "clojure.core.matrix" namespace. Problem: 
> that gives you a ton of nasty warnings of the type "WARNING: + already 
> refers to: #'clojure.core/+ in namespace: test.blank, being replaced by: 
> #'clojure.core.matrix/+". Significant boilerplate must be maintained by the 
> user in their ns declaration to prevent these warnings. I don't like 
> forcing users to maintain boilerplate, and I think that normal idiomatic 
> usage should be warning-free.
>
> B) Separate the name-clashing functions into separate namespaces - e.g. 
> "clojure.core.matrix.operators". Problem: that's something of an artificial 
> division, and again it forces users to do extra ns-management work to 
> access the functions they want.
>
> C) Use different names. Problem: names would be worse, and this would be 
> inconsistent and confusing, especially for functions that do effectively 
> the same thing.
>
> D) Encourage users to use aliases. Problem: that's horrendously ugly and 
> inconvenient for numerical code. Users with any sense of elegance in their 
> coding style would quite rightly throw their hands up in disgust. 
>
> Currently we're doing B), I'd prefer to do A) but can't figure out a way 
> to automatically suppress the warnings.
>
> Any better ideas?
>
>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

Hi Konrad,

Okay, I think I was just being dense.  I thought you were talking about a
different macroexpansion strategy, rather than just doing full
macroexpansion without interleaved compilation.  Thanks for your patience
in explaining what you meant.

I will note, though, that &env is an implicit argument to the macros, so
anything which works "exactly" like the compiler needs to mimic that as
well.

Zach


On Thu, Sep 5, 2013 at 3:09 AM, Konrad Hinsen <
googlegro...@khinsen.fastmail.net> wrote:

> **
> Zach Tellman writes:
>
>  > I guess I'm confused, then.  You contrast "complete recursive
>  > expansion" with what the compiler does, and then say it's recursive
>  > prewalk expansion, which is exactly what the compiler does.  Can
>  > you clarify the difference between what you're doing and what the
>  > compiler does?
>
> Here's an example:
>
>(defmacro foo [x]
>  `(list ~x ~x))
>
>(defmacro bar [x]
>  `[~x ~x])
>
> Now let's work on the form
>
>(foo (bar 'baz))
>
> Plain macroexpand returns
>
>(list (bar 'baz) (bar 'baz))
>
> whereas tools.macro/mexpand-all gives
>
>(list ['baz 'baz] ['baz 'baz])
>
> It does this by first calling macroexpand, so foo gets called exactly
> as during Clojure compilation and returns
>
>(list (bar 'baz) (bar 'baz))
>
> mexpand-all then goes through that form and expands the two subforms
> (bar 'baz).
>
> So mexpand-all does exactly what the compiler does, in particular it
> calls the macros with exactly the same arguments. But the compiler
> interleaves macro expansion with compilation, so it never gives you
> access to the fully expanded but uncompiled form which is
>
>(list ['baz 'baz] ['baz 'baz])
>
> Konrad
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

Sorry, that documentation reflected 0.1.1-SNAPSHOT, which I've just
released as 0.1.1.  Let me know if you have any other issues.

Zach


On Thu, Sep 5, 2013 at 6:26 AM, Stathis Sideris  wrote:

> Thanks for this library Zach,
>
> It seems that the released version is a bit behind in comparison to the
> generated documentation [1]. For example, walk-exprs is advertised as being
> able to accept a special-forms parameter, but that's not the case in the
> jar that leiningen retrieved when I used [riddley "0.1.0"] in my
> project.clj.
>
> Thanks,
>
> Stathis
>
> [1] http://ideolalia.com/riddley/
>
>
>
> On Monday, 2 September 2013 21:49:01 UTC+1, Zach Tellman wrote:
>>
>> When I announced Proteus [1], it was rightfully pointed out that it
>> didn't play nicely with macros which rely on &env, as well as a few forms
>> like 'letfn' that I hadn't explicitly handled.  This flaw has been shared
>> by pretty much every library of this sort, and since this is a problem I've
>> half-solved two or three times already, I figured something more general
>> and lasting was in order.
>>
>> The resulting library is called Riddley [2].  For obvious reasons, I've
>> named it after a book which is written entirely in a barely-readable pidgin
>> dialect. While there may be lingering issues, it's good enough to replace
>> the code-walking mechanism in Proteus, which I think makes it the best game
>> in town right now.  Bug reports and pull requests are welcome.
>>
>> Zach
>>
>> [1] https://groups.google.com/**forum/#!searchin/clojure/**
>> proteus/clojure/7HNNiJJTte4/**iMBWn8p6tZAJ<https://groups.google.com/forum/#!searchin/clojure/proteus/clojure/7HNNiJJTte4/iMBWn8p6tZAJ>
>> [2] 
>> https://github.com/**ztellman/riddley<https://github.com/ztellman/riddley>
>>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [ANN] riddley: code-walking without caveats

[Zach Tellman]

Hi Nils,

Thanks for the link, I hadn't seen that before.  Happily, what you describe
is pretty much exactly what Riddley does, in this case by accessing the
compiler internals.  An example of how macrolet or symbol-macrolet could be
implemented using this is linked above in my response to Konrad.  This
means that macros which rely on the clojure.lang.Compiler$LocalBinding
(which includes [1] and [2]) will still work.  As far as I can tell, this
is not the case in either tools.macro or core.async.

If you know of any ways to make the code-walking more complete, please let
me know.

Zach

[1] https://github.com/ztellman/proteus/blob/master/src/proteus.clj#L60
[2]
https://github.com/flatland/useful/blob/develop/src/flatland/useful/datatypes.clj#L62


On Thu, Sep 5, 2013 at 3:31 AM, bertschi
wrote:

> Hi Zach,
>
> you might want to look at this paper explaining how to write a correct
> macroexpand-all (which requires a code walker) in Common Lisp:
> http://www.merl.com/publications/TR1993-017/
>
> The compiler certainly has to do something like that, but might not do all
> of the macroexpansion before starting any compilation as Konrad explained.
> What the compiler needs to do is track the lexical environment while
> walking down the source forms. When a code walker wants to introduce
> additional bindings, such as macrolet (for local macros) or symbol-macrolet
> (for new symbols) it needs to be able to extend the environment
> accordingly. So, you either have to access the compiler internals,
> especially its environment handling, or track the environment yourself (as
> Konrad suggested).
> As an aside: The problem in Common Lisp is mainly that the environment
> handling is not exposed in the standard, thus you cannot write a portable
> code walker without doing some environment handling yourself.
>
> You might also want to look at core.async, which uses a code walker to
> transform go blocks into state machines. I have not (yet) checked its
> restrictions (someone told me, that it cannot even look into anonymous fn
> forms within its body!), but it is generally very hard to write a code
> walker that can handle all special forms (in Common Lisp I don't know any).
>
> +10 for having a library that supports writing correct and (almost)
> complete code walkers
>
> Best,
>
>Nils
>
> On Thursday, September 5, 2013 12:09:28 PM UTC+2, Konrad Hinsen wrote:
>>
>> Zach Tellman writes:
>>
>>  > I guess I'm confused, then.  You contrast "complete recursive
>>  > expansion" with what the compiler does, and then say it's recursive
>>  > prewalk expansion, which is exactly what the compiler does.  Can
>>  > you clarify the difference between what you're doing and what the
>>  > compiler does?
>>
>> Here's an example:
>>
>>(defmacro foo [x]
>>  `(list ~x ~x))
>>
>>(defmacro bar [x]
>>  `[~x ~x])
>>
>> Now let's work on the form
>>
>>(foo (bar 'baz))
>>
>> Plain macroexpand returns
>>
>>(list (bar 'baz) (bar 'baz))
>>
>> whereas tools.macro/mexpand-all gives
>>
>>(list ['baz 'baz] ['baz 'baz])
>>
>> It does this by first calling macroexpand, so foo gets called exactly
>> as during Clojure compilation and returns
>>
>>(list (bar 'baz) (bar 'baz))
>>
>> mexpand-all then goes through that form and expands the two subforms
>> (bar 'baz).
>>
>> So mexpand-all does exactly what the compiler does, in particular it
>> calls the macros with exactly the same arguments. But the compiler
>> interleaves macro expansion with compilation, so it never gives you
>> access to the fully expanded but uncompiled form which is
>>
>>(list ['baz 'baz] ['baz 'baz])
>>
>> Konrad
>>
>  --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/a68aThpvP4o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
>

Re: Handling name collisions with clojure.core

[Zach Tellman]

Not doing actual replacement via code-walking will prevent any functions
with inline definitions actually being able to benefit from this.  I'm not
sure if those are used in core.matrix, though.


On Thu, Sep 5, 2013 at 4:54 AM, Colin Fleming
wrote:

> This is actually probably not a bad solution. You wouldn't even need to
> rewrite, couldn't you just expand to a let?
>
> (let [* clojure.core.matrix.*
>   + clojure.core.matrix.+]
>   (+ ... (* ...)))
>
> Although thinking about it, you'd have to let-bind all possible operators
> every time, and the compiler doesn't do any dead-code elimination (although
> Hotspot might).
>
>
> On 5 September 2013 17:35, Dave Ray  wrote:
>
>> Maybe this is a dumb idea, but could you have a macro that rewrites code
>> to use your ops?
>>
>>   (require '[clojure.core.matrix :as m])
>>   (m/with-ops (+ ... (* ...) ...))
>>
>> and then all the "special" symbols get rewritten/qualified with
>> clojure.core.matrix?
>>
>> Dave
>>
>>
>>
>> On Wed, Sep 4, 2013 at 10:26 PM, Sean Corfield wrote:
>>
>>> You only get the warning if you 'use' the namespace or 'refer all'
>>> tho', correct?
>>>
>>> And we've recently seen a lot of discussion that basically says "don't
>>> do that" so it seems that either users of core.matric are going to
>>> have two "approved" choices:
>>> * require core.matrix with an alias, or choose to rename colliding
>>> names however you want
>>> * exclude the colliding symbols via refer-clojure and require them
>>> from core.matrix as referred symbols
>>>
>>> That's seems right to me: it is explicit and provides no surprises; it
>>> gives the user control over how to manage things.
>>>
>>> Sean
>>>
>>> On Wed, Sep 4, 2013 at 6:22 PM, Mikera 
>>> wrote:
>>> > Hi all,
>>> >
>>> > While building the API for core.matrix, I've fun into a few cases
>>> where the
>>> > "best" name is a direct clash with clojure.core.
>>> >
>>> > Examples are "+", "zero?", "vector?", "=="
>>> >
>>> > In many of these cases, the core.matrix behaviour is a natural
>>> extension of
>>> > the clojure.core function (i.e. it extends the same functionality to
>>> > arbitrary N-dimensional arrays).
>>> >
>>> > I'm not very happy with any of the options I can see for handling this:
>>> >
>>> > A) Use the good names in the "clojure.core.matrix" namespace. Problem:
>>> that
>>> > gives you a ton of nasty warnings of the type "WARNING: + already
>>> refers to:
>>> > #'clojure.core/+ in namespace: test.blank, being replaced by:
>>> > #'clojure.core.matrix/+". Significant boilerplate must be maintained
>>> by the
>>> > user in their ns declaration to prevent these warnings. I don't like
>>> forcing
>>> > users to maintain boilerplate, and I think that normal idiomatic usage
>>> > should be warning-free.
>>> >
>>> > B) Separate the name-clashing functions into separate namespaces - e.g.
>>> > "clojure.core.matrix.operators". Problem: that's something of an
>>> artificial
>>> > division, and again it forces users to do extra ns-management work to
>>> access
>>> > the functions they want.
>>> >
>>> > C) Use different names. Problem: names would be worse, and this would
>>> be
>>> > inconsistent and confusing, especially for functions that do
>>> effectively the
>>> > same thing.
>>> >
>>> > D) Encourage users to use aliases. Problem: that's horrendously ugly
>>> and
>>> > inconvenient for numerical code. Users with any sense of elegance in
>>> their
>>> > coding style would quite rightly throw their hands up in disgust.
>>> >
>>> > Currently we're doing B), I'd prefer to do A) but can't figure out a
>>> way to
>>> > automatically suppress the warnings.
>>> >
>>> > Any better ideas?
>>> >
>>> >
>>> >
>>> > --
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups "Clojure" group.
>>> > To post to this group, send email to clojure@googlegroups.com
>>> > Note that posts from new members are moderated - please be patient
>>> with your
>>> > first post.
>>> > To unsubscribe from this group, send email to
>>> > clojure+unsubscr...@googlegroups.com
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/clojure?hl=en
>>> > ---
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "Clojure" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an
>>> > email to clojure+unsubscr...@googlegroups.com.
>>> > For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>> --
>>> Sean A Corfield -- (904) 302-SEAN
>>> An Architect's View -- http://corfield.org/
>>> World Singles, LLC. -- http://worldsingles.com/
>>>
>>> "Perfection is the enemy of the good."
>>> -- Gustave Flaubert, French realist novelist (1821-1880)
>>>
>>> --
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patien

Re: Use core.async executor with aleph

[Zach Tellman]

Hi Robin,

You can absolutely specify an executor of :none if you're sure you won't be 
doing any blocking in your request handler.  If everything's wrapped by a 
go-block, that's certainly the case, and is probably the most efficient 
approach.  However, Aleph just needs some java.util.concurrent.Executor, so 
using the core.async executor is also a valid approach.

By the way, I'm more likely to notice these sorts of questions if you ask 
them on the Aleph mailing 
list: https://groups.google.com/forum/#!forum/aleph-lib

Zach

On Tuesday, February 17, 2015 at 1:35:53 AM UTC-8, Robin Heggelund Hansen 
wrote:
>
> From what I can see, aleph allows me to set a executor to handle client 
> requests. I'm already using core.async pretty heavily. Is there any reason 
> why I shouldn't pass core.async's executor to aleph? I see I can also make 
> every client request start on aleph's dispatch thread. Considering 
> absolutely every request spawns a go-block, might it even be a good idea to 
> not run aleph with an executor at all?
>
> Thanks!
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN} Aleph 0.4.0 released, plus Manifold, Dirigiste, and a whole host of other libraries

[Zach Tellman]

Hey all,

In preparation for Clojure/West, I'm formally releasing the latest Aleph 
and the libraries that surround it.  Aleph 0.4.0 has been running in 
production at Factual for half a year now, and across a variety of services 
is handling at peak 600k HTTP requests/sec (spread across 15-20 machines).  

Since the landscape of Clojure HTTP servers is pretty crowded these days, 
it's worth taking some time to explain how Aleph differs.  To be clear, 
most Clojure deployments likely use Jetty, and should continue to do so. 
 However, Aleph has some unique properties:

* It uses the Netty library, which is a high-performance and very 
battle-tested network layer for the JVM
* It is the only HTTP server that has *ubiquitous* asynchronous streams 
wherever data can be received or sent (all other libraries can only 
represent streaming requests using InputStreams, or like http-kit don't 
support streaming HTTP requests at all)
* It is the only server that has a WebSocket implementation with any 
support for per-connection backpressure.  I won't make this post even 
longer by going into why this is important, but this will be a central 
theme of my talk at Clojure/West next week if you're interested in hearing 
more.
* It uses consistent abstractions to represent network connections over a 
variety of protocols, which makes it straightforward to use the same 
application logic for all of them.

Again, none of these points mean you should immediately drop whatever 
you're using and move over to Aleph instead.  However, I do feel it 
represents the only (current) good option for using core.async or a similar 
stream abstraction to represent network data, which is an idea a number of 
people seem to be playing with lately.  Some examples of this can be found 
at http://ideolalia.com/aleph/literate.html.

A full list of the libraries:

aleph - https://github.com/ztellman/aleph - uses the excellent Netty 
library to expose HTTP, TCP, and UDP using a consistent asynchronous stream 
representation.

manifold - https://github.com/ztellman/manifold - an unopinionated stream 
representation designed to cleanly interoperate with other stream 
representations (Clojure's seqs, core.async channels, Java's 
BlockingQueues, and others).  This is the base stream representation for 
all network sources and sinks in Aleph.

dirigiste -  https://github.com/ztellman/dirigiste - a pure-Java library 
that provides instrumented, dynamically sized thread and object pools. 
 This is used for thread pools in Aleph's HTTP server, and for connection 
pools in Aleph's HTTP client.

byte-streams -  https://github.com/ztellman/byte-streams - a means of 
translating any byte representation into another.  Want to turn a 
core.async channel that emits byte-arrays into an InputStream, or maybe the 
other way around?  Look no further.  The library's conversion mechanism is 
extensible, which is used in Aleph to make Netty's custom byte 
representations interoperable with more familiar representations.

byte-transforms -  https://github.com/ztellman/byte-transforms - a curated 
collection of byte compression, hashing, and encoding mechanisms, which can 
work on anything byte-streams can convert.

While all these libraries are used in concert to create Aleph, I've been 
very careful to make sure any of them can be used by themselves.  If anyone 
has questions about them, the best place to get my attention is the Aleph 
mailing list: https://groups.google.com/forum/#!forum/aleph-lib.

I will be mentioning some of these libraries at my upcoming Clojure/West 
talk (http://clojurewest.org/speakers#ztellman), but I've also set aside an 
Unsession for specifically discussing these 
libraries: https://github.com/clojurewest/clojurewest2015/wiki/Unsessions. 
 If you're interested, please add your name to the list.

Zach

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN} Aleph 0.4.0 released, plus Manifold, Dirigiste, and a whole host of other libraries

[Zach Tellman]

Hey Dmitri,

I haven't used any sort of dev-mode before (I just update stuff in the REPL
when necessary), but it seems like something like that belongs in
middleware, not the server.  The server is just calling a function, it
shouldn't care if something else is changing that function's behavior.

If other servers have this behavior embedded, maybe it can be extracted
into a standalone library?

Zach

On Fri, Apr 17, 2015 at 3:47 PM, Dmitri  wrote:

> I'd like to add Aleph to the Luminus template and I was wondering if
> there's an equivalent of dev mode available for other servers where it
> watches for changes in source and reloads them. I did a cursory look but
> didn't spot anything like a -dev option.
>
> On Friday, April 17, 2015 at 5:06:30 PM UTC-4, Zach Tellman wrote:
>
>> Hey all,
>>
>> In preparation for Clojure/West, I'm formally releasing the latest Aleph
>> and the libraries that surround it.  Aleph 0.4.0 has been running in
>> production at Factual for half a year now, and across a variety of services
>> is handling at peak 600k HTTP requests/sec (spread across 15-20 machines).
>>
>> Since the landscape of Clojure HTTP servers is pretty crowded these days,
>> it's worth taking some time to explain how Aleph differs.  To be clear,
>> most Clojure deployments likely use Jetty, and should continue to do so.
>> However, Aleph has some unique properties:
>>
>> * It uses the Netty library, which is a high-performance and very
>> battle-tested network layer for the JVM
>> * It is the only HTTP server that has *ubiquitous* asynchronous streams
>> wherever data can be received or sent (all other libraries can only
>> represent streaming requests using InputStreams, or like http-kit don't
>> support streaming HTTP requests at all)
>> * It is the only server that has a WebSocket implementation with any
>> support for per-connection backpressure.  I won't make this post even
>> longer by going into why this is important, but this will be a central
>> theme of my talk at Clojure/West next week if you're interested in hearing
>> more.
>> * It uses consistent abstractions to represent network connections over a
>> variety of protocols, which makes it straightforward to use the same
>> application logic for all of them.
>>
>> Again, none of these points mean you should immediately drop whatever
>> you're using and move over to Aleph instead.  However, I do feel it
>> represents the only (current) good option for using core.async or a similar
>> stream abstraction to represent network data, which is an idea a number of
>> people seem to be playing with lately.  Some examples of this can be found
>> at http://ideolalia.com/aleph/literate.html.
>>
>> A full list of the libraries:
>>
>> aleph - https://github.com/ztellman/aleph - uses the excellent Netty
>> library to expose HTTP, TCP, and UDP using a consistent asynchronous stream
>> representation.
>>
>> manifold - https://github.com/ztellman/manifold - an unopinionated
>> stream representation designed to cleanly interoperate with other stream
>> representations (Clojure's seqs, core.async channels, Java's
>> BlockingQueues, and others).  This is the base stream representation for
>> all network sources and sinks in Aleph.
>>
>> dirigiste -  https://github.com/ztellman/dirigiste - a pure-Java library
>> that provides instrumented, dynamically sized thread and object pools.
>> This is used for thread pools in Aleph's HTTP server, and for connection
>> pools in Aleph's HTTP client.
>>
>> byte-streams -  https://github.com/ztellman/byte-streams - a means of
>> translating any byte representation into another.  Want to turn a
>> core.async channel that emits byte-arrays into an InputStream, or maybe the
>> other way around?  Look no further.  The library's conversion mechanism is
>> extensible, which is used in Aleph to make Netty's custom byte
>> representations interoperable with more familiar representations.
>>
>> byte-transforms -  https://github.com/ztellman/byte-transforms - a
>> curated collection of byte compression, hashing, and encoding mechanisms,
>> which can work on anything byte-streams can convert.
>>
>> While all these libraries are used in concert to create Aleph, I've been
>> very careful to make sure any of them can be used by themselves.  If anyone
>> has questions about them, the best place to get my attention is the Aleph
>> mailing list: https://groups.google.com/forum/#!forum/aleph-lib.
>>
>> I will be mentioning some of t

Re: [ANN} Aleph 0.4.0 released, plus Manifold, Dirigiste, and a whole host of other libraries

[Zach Tellman]

I suspect it would, I think I was just letting the mechanics of Manifold's
let-flow macro color my judgment. Happy to accept any pull requests which
make my core.async examples more idiomatic.
On Apr 18, 2015 8:33 AM, "Matthias Lange"  wrote:

> In your examples, you put a let around the reads from timeouts.
>
> (let [_ (a/
> As far as i know, that is not neccessary. So your first example could be:
>
> (defn delayed-hello-world-handler
>   [req]
>   (d/->deferred
> (a/go
>   (a/   (hello-world-handler req
>
> Would that not work?
>
>  --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/KH8Js8URKwA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] lein-jammin

[Zach Tellman]

https://github.com/ztellman/lein-jammin

This one's pretty simple: you put `lein jammin ` in front of any 
other Leiningen task, and if it gets stuck for the specified duration, it 
prints out a thread dump.  This especially useful for tests, and 
extra-especially useful for tests in a CI environment where using `jstack` 
or similar tools is difficult or impossible.

Enjoy,
Zach

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] data.int-map 0.2.0

[Zach Tellman]

https://github.com/clojure/data.int-map

The 0.1.0 release of this library was a faithful Clojure implementation of 
Okasaki's "Fast Mergeable Integer Maps" paper [1].  While it was much 
faster than Clojure's sorted-maps, it was in some cases slower than 
hash-maps, which was mostly due to the int-maps being a binary tree, and 
hash-maps being a 32-ary tree.  It also could only take non-negative 
integers, which was a limitation of the approach used in the paper.  In the 
0.2.0 release, I reimplemented the core data structure in Java, but more 
importantly adapted the data structure to use 16-ary trees under the 
covers.  This has greatly improved performance (by a factor of 2-5x, 
depending on the benchmark), and has also allowed the map to have negative 
integers for keys.

I recommend anyone using an earlier version of this library upgrade to the 
latest, which will be a drop-in replacement.

Zach

[1] http://ittc.ku.edu/~andygill/papers/IntMap98.pdf

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Opinion on core.async vs callbacks in abstract APIs?

[Zach Tellman]

The problem with using bare callbacks is that there's no way for the 
invoked callback to exert backpressure, except by blocking or passing in a 
CPS-style callback to the callback, neither of which is ideal.  If you're 
looking for a "neutral" choice, I'd suggest an infinite lazy-seq over 
callbacks.  If you absolutely need it to be async, something like Manifold 
might be warranted, in that it allows people to consume it however they 
like (though they will have to be at least a little familiar with Manifold 
to do so).

Zach

On Monday, June 1, 2015 at 12:18:19 PM UTC-7, Christopher Small wrote:
>
> Greetings
>
> I imagine most of us here would rather use core.async channels over 
> callbacks in their application code, particularly with more complicated 
> applications. But is it okay/preferable for Clojure libraries to force 
> their users to use core.async channels as part of an API (an event channel, 
> for example)? 
>
> As much as I love core.async, I can't help but wonder whether sticking 
> with callbacks for an API isn't a simpler/better design strategy. It's easy 
> enough to drop messages on a channel in a callback, and this let's users 
> opt-in. But if one expects core.async channels are what most would prefer 
> anyway, is it okay to foist them upon everyone?
>
> As a follow up, does your opinion on the matter change if implementations 
> of an API become simpler using core.async channels?
>
>
> Looking forward to your thoughts :-)
>
> Chris Small
>
>
>
> PS I'm asking because I'm working on a physical computing API (
> https://github.com/clj-bots/pin-ctrl) and debating between using channels 
> vs callbacks for the edge detection functionality (if you're not familiar, 
> edge detection let's you asynchronously handle changes in pin state, such 
> as button pushes). If you're interested in this question as it applies 
> specifically to this application, feel free to join the discussion on our 
> gitter channel: https://gitter.im/clj-bots/chat
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RejectedExecutionException? I don't think I was using anything concurrent?

[Zach Tellman]

A RejectedExecutionException is thrown when a thread-pool's queue is full. 
 Try using 'jstack' to take a thread dump, and see who has spun up a thread 
pool under the covers.  Possible culprits include 'at-at' and 'carmine'.  

On Friday, July 17, 2015 at 11:01:05 AM UTC-7, sugarste...@gmail.com wrote:
>
>
> Okay, this error seems to be triggered by a NullPointerException which 
> seems to arise when I try to fetch a non-existent document out of Redis. I 
> am a bit of a noob, so perhaps someone can explain how this works. I'm not 
> aware of using any of Clojure's concurrency tools in this (very simple) 
> app, so how might the NullPointerException lead to 
> RejectedExecutionException?  
>
>
> On Friday, July 17, 2015 at 1:17:47 PM UTC-4, sugarste...@gmail.com wrote:
>>
>> My app starts up, runs for a few minutes, and then: 
>>
>> #> java.util.concurrent.RejectedExecutionException: Task 
>> java.util.concurrent.FutureTask@4fe887e6 rejected from 
>> java.util.concurrent.ThreadPoolExecutor@aea107f[Terminated, pool size = 0, 
>> active threads = 0, queued tasks = 0, completed tasks = 3]>
>>
>> This is a very simple app, and I am not aware of using any concurrency. I 
>> am not using Futures or Promises or pmap or anything else I can think of. 
>> My first hunch would be this comes from some library I am using, but I am 
>> not sure what. I searched on Google but none of the reported problems seem 
>> to apply to me. This is what I am using, do any of these seem likely to 
>> throw this error? 
>>
>> [org.clojure/clojure "1.6.0"]
>>  [com.taoensso/timbre "4.0.2"]
>>  [dire "0.5.1"]
>>  [slingshot "0.12.2"]
>>  [org.clojure/data.json "0.2.5"]
>>  [org.clojure/tools.namespace "0.2.4"]
>>  [me.raynes/fs "1.4.4"]
>>  [clj-stacktrace "0.2.8"]
>>  [overtone/at-at "1.2.0"]
>>  [org.clojure/core.cache "0.6.4"]
>>  [cheshire "5.5.0"]
>>  [com.taoensso/carmine "2.11.1"]
>>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Clojure 1.8.0-alpha2

[Zach Tellman]

You're also going to have to target [clj-tuple "0.2.2"], since something 
else seems to be shadowing that depedency.  Sorry for all the fuss.

On Sunday, July 19, 2015 at 5:02:13 PM UTC-7, Sean Corfield wrote:
>
> Bumping clj-http to 2.0.0 got me past that problem (and it uses Potemkin 
> 0.4.1) but then I hit the problem below, which I’ve run into several times 
> trying to use updated versions of clj-http over the last year.
>
> Messagejava.lang.RuntimeException: No such var: clj-tuple/vector, 
> compiling:(potemkin/utils.clj:110:10)Cause
> clojure.lang.Compiler$CompilerExceptionStacktraceThe Error Occurred in
> *core.clj: line 5866* 
> *called from* core.clj: line 5865 
> *called from* core.clj: line 5671 
> *called from* core.clj: line 5711 
> *called from* core.clj: line 5710 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5753 
> *called from* core.clj: line 634 
> *called from* core.clj: line 5843 
> *called from* collections.clj: line 1 
> *called from* collections.clj: line 1 
> *called from* core.clj: line 5866 
> *called from* core.clj: line 5865 
> *called from* core.clj: line 5671 
> *called from* core.clj: line 5711 
> *called from* core.clj: line 5710 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5753 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5832 
> *called from* potemkin.clj: line 1 
> *called from* potemkin.clj: line 1 
> *called from* core.clj: line 5866 
> *called from* core.clj: line 5865 
> *called from* core.clj: line 5671 
> *called from* core.clj: line 5711 
> *called from* core.clj: line 5710 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5749 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5832 
> *called from* headers.clj: line 1 
> *called from* headers.clj: line 1 
> *called from* core.clj: line 5866 
> *called from* core.clj: line 5865 
> *called from* core.clj: line 5671 
> *called from* core.clj: line 5711 
> *called from* core.clj: line 5710 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5749 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5832 
> *called from* core.clj: line 1 
> *called from* core.clj: line 1 
> *called from* core.clj: line 5866 
> *called from* core.clj: line 5865 
> *called from* core.clj: line 5671 
> *called from* core.clj: line 5711 
> *called from* core.clj: line 5710 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5749 
> *called from* core.clj: line 632 
> *called from* core.clj: line 5832 
> *called from* client.clj: line 1 
>
>
> On Jul 19, 2015, at 4:53 PM, Michael Blume  > wrote:
>
> Looks like it has, pinning to Potemkin 0.4.1 should probably sort you out
>
> On Sun, Jul 19, 2015 at 4:50 PM Michael Blume  > wrote:
>
>> Sean, I think that was identified as a bug in Potemkin. The pull was 
>> merged but I'm not sure if there's been a release since. Zack?
>>
>> https://github.com/ztellman/potemkin/pull/40
>>
>>
>> http://dev.clojure.org/jira/browse/CLJ-1208?focusedCommentId=39632&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-39632
>>
>> On Sun, Jul 19, 2015 at 4:47 PM Sean Corfield > > wrote:
>>
>>> On Jul 18, 2015, at 10:33 PM, Sean Corfield >> > wrote:
>>> > Wow, that's a fast timeline. Thank you. We'll upgrade to Alpha 2 this 
>>> week. We may go to production with it fairly quickly.
>>>
>>> Switched out 1.7.0 for 1.8.0-alpha2 and got the exception below. Posting 
>>> here in case anyone knows immediately what the cause is, while I go 
>>> debugging...
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError: IllegalName: 
>>> compile__stub.clj_http.headers.clj-http.headers/HeaderMap, 
>>> compiling:(clj_http/headers.clj:105:1)
>>> at clojure.lang.Compiler.analyzeSeq(Compiler.java:6798)
>>> at clojure.lang.Compiler.analyze(Compiler.java:6592)
>>> at clojure.lang.Compiler.analyze(Compiler.java:6553)
>>> at 
>>> clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5929)
>>> at clojure.lang.Compiler$LetExpr$Parser.parse(Compiler.java:6247)
>>> at clojure.lang.Compiler.analyzeSeq(Compiler.java:6791)
>>> at clojure.lang.Compiler.analyze(Compiler.java:6592)
>>> at clojure.lang.Compiler.analyze(Compiler.java:6553)
>>> at 
>>> clojure.lang.Compiler$BodyExpr$Parser.parse(Compiler.java:5929)
>>> at clojure.lang.Compiler$FnMethod.parse(Compiler.java:5359)
>>> at clojure.lang.Compiler$FnExpr.parse(Compiler.java:3959)
>>> at clojure.lang.Compiler.analyzeSeq(Compiler.java:6789)
>>> at clojure.lang.Compiler.analyze(Compiler.java:6592)
>>> at clojure.lang.Compiler.eval(Compiler.java:6847)
>>> at clojure.lang.Compiler.eval(Compiler.java:6839)
>>> at clojure.lang.Compiler.load(Compiler.java:7295)
>>> at clojure.lang.RT.loadResourceScript(RT.java:372)
>>> at clojure.lang.RT.loadResourceScript(RT.java:363)
>

Re: [ANN] Clojure 1.8.0-alpha2

[Zach Tellman]

A similar issue was reported earlier in the thread, target [potemkin 
"0.4.1"] and see if that fixes it.

On Tuesday, July 21, 2015 at 12:24:43 PM UTC-7, Rangel Spasov wrote:
>
> Hey guys,
>
> Getting this error with 1.8.0-alpha2, I think related to aleph (using 
> 0.4.0, latest version at the moment).
>
> #error {
>
>  :cause IllegalName: 
> compile__stub.aleph.http.core.aleph.http.core/HeaderMap
>
>  :via
>
>  [{:type clojure.lang.Compiler$CompilerException
>
>:message java.lang.NoClassDefFoundError: IllegalName: 
> compile__stub.aleph.http.core.aleph.http.core/HeaderMap, 
> compiling:(aleph/http/core.clj:81:1)
>
>:at [clojure.lang.Compiler analyzeSeq Compiler.java 6798]}
>
>   {:type java.lang.NoClassDefFoundError
>
>:message IllegalName: 
> compile__stub.aleph.http.core.aleph.http.core/HeaderMap
>
>:at [java.lang.ClassLoader preDefineClass ClassLoader.java 654]}]
>
>  :trace
>
>  [[java.lang.ClassLoader preDefineClass ClassLoader.java 654]
>
>   [java.lang.ClassLoader defineClass ClassLoader.java 758]
>
>   [java.lang.ClassLoader defineClass ClassLoader.java 642]
>
>   [clojure.lang.DynamicClassLoader defineClass DynamicClassLoader.java 46]
>
>   [clojure.lang.Compiler$NewInstanceExpr compileStub Compiler.java 7815]
>
>   [clojure.lang.Compiler$NewInstanceExpr build Compiler.java 7680]
>
>   [clojure.lang.Compiler$NewInstanceExpr$DeftypeParser parse Compiler.java 
> 7590]
>
>   [clojure.lang.Compiler analyzeSeq Compiler.java 6791]
>
>   [clojure.lang.Compiler analyze Compiler.java 6592]
>
>   [clojure.lang.Compiler analyze Compiler.java 6553]
>
>   [clojure.lang.Compiler$BodyExpr$Parser parse Compiler.java 5929]
>
>   [clojure.lang.Compiler$LetExpr$Parser parse Compiler.java 6247]
>
>   [clojure.lang.Compiler analyzeSeq Compiler.java 6791]
>
>   [clojure.lang.Compiler analyze Compiler.java 6592]
>
>   [clojure.lang.Compiler analyze Compiler.java 6553]
>
>   [clojure.lang.Compiler$BodyExpr$Parser parse Compiler.java 5929]
>
>   [clojure.lang.Compiler$FnMethod parse Compiler.java 5359]
>
>   [clojure.lang.Compiler$FnExpr parse Compiler.java 3959]
>
>   [clojure.lang.Compiler analyzeSeq Compiler.java 6789]
>
>   [clojure.lang.Compiler analyze Compiler.java 6592]
>
>   [clojure.lang.Compiler eval Compiler.java 6847]
>
>   [clojure.lang.Compiler eval Compiler.java 6839]
>
>   [clojure.lang.Compiler load Compiler.java 7295]
>
>   [clojure.lang.RT loadResourceScript RT.java 372]
>
>   [clojure.lang.RT loadResourceScript RT.java 363]
>
>   [clojure.lang.RT load RT.java 453]
>
>   [clojure.lang.RT load RT.java 419]
>
>   [clojure.core$load$fn__5448 invoke core.clj 5866]
>
>   [clojure.core$load doInvoke core.clj 5865]
>
>   [clojure.lang.RestFn invoke RestFn.java 408]
>
>   [clojure.core$load_one invoke core.clj 5671]
>
>   [clojure.core$load_lib$fn__5397 invoke core.clj 5711]
>
>   [clojure.core$load_lib doInvoke core.clj 5710]
>
>   [clojure.lang.RestFn applyTo RestFn.java 142]
>
>   [clojure.core$apply invoke core.clj 632]
>
>   [clojure.core$load_libs doInvoke core.clj 5749]
>
>   [clojure.lang.RestFn applyTo RestFn.java 137]
>
>   [clojure.core$apply invoke core.clj 632]
>
>   [clojure.core$require doInvoke core.clj 5832]
>
>   [clojure.lang.RestFn invoke RestFn.java 551]
>
>   [aleph.http.server$eval9251$loading__5340__auto9252 invoke 
> server.clj 1]
>
>   [aleph.http.server$eval9251 invoke server.clj 1]
>
>   [clojure.lang.Compiler eval Compiler.java 6850]
>
>   [clojure.lang.Compiler eval Compiler.java 6839]
>
>   [clojure.lang.Compiler load Compiler.java 7295]
>
>   [clojure.lang.RT loadResourceScript RT.java 372]
>
>   [clojure.lang.RT loadResourceScript RT.java 363]
>
>   [clojure.lang.RT load RT.java 453]
>
>   [clojure.lang.RT load RT.java 419]
>
>   [clojure.core$load$fn__5448 invoke core.clj 5866]
>
>   [clojure.core$load doInvoke core.clj 5865]
>
>   [clojure.lang.RestFn invoke RestFn.java 408]
>
>   [clojure.core$load_one invoke core.clj 5671]
>
>   [clojure.core$load_lib$fn__5397 invoke core.clj 5711]
>
>   [clojure.core$load_lib doInvoke core.clj 5710]
>
>   [clojure.lang.RestFn applyTo RestFn.java 142]
>
>   [clojure.core$apply invoke core.clj 632]
>
>   [clojure.core$load_libs doInvoke core.clj 5753]
>
>   [clojure.lang.RestFn applyTo RestFn.java 137]
>
>   [clojure.core$apply invoke core.clj 632]
>
>   [clojure.core$require doInvoke core.clj 5832]
>
>   [clojure.lang.RestFn invoke RestFn.java 457]
>
>   [aleph.http$eval1594$loading__5340__auto1595 invoke http.clj 1]
>
>   [aleph.http$eval1594 invoke http.clj 1]
>
>   [clojure.lang.Compiler eval Compiler.java 6850]
>
>   [clojure.lang.Compiler eval Compiler.java 6839]
>
>   [clojure.lang.Compiler load Compiler.java 7295]
>
>   [clojure.lang.RT loadResourceScript RT.java 372]
>
>   [clojure.lang.RT loadResourceScript RT.java 363]
>
>   [clojure.lang.RT load RT.java 453]
>
>   [clojure.lang.RT load RT.java 419]
>
>   [clojure.core$load$fn__5448 invoke core.clj 5866]
>
>   [clojure.core$load doInvoke core.clj 5865]
>
>

Re: Howto Use Automat ?

[Zach Tellman]

Hi Tim,

Glad to see you're trying Automat out.

I'm not sure what  you mean by "forks in the road".  If your FSM is defined 
as (a/or [1 3] [1 2 3]), then you can do any of the following:

(reduce #(a/advance fsm %1 %2) nil [1 2 3])

(a/advance-stream fsm nil [1 2 3] ::rejected)

(let [adv (partial a/advance fsm)]
  (-> nil (adv 1) (adv 2) (adv 3)))

Does that answer your first two questions?  If not, you'll have to clarify 
what you're asking.

Automat doesn't allow you to name your states, because that breaks your 
ability to freely compose automata.  If you want to differentiate between 
two different paths, you can use actions:

  (view (a/or [:sent :acknowledged :accepted] [:sent :rejected (a/$ 
:not-accepted)]))

I can go into more depth on this, but the README gives a fairly complete 
explanation of how actions can be used.

Hope that helps,
Zach




On Tuesday, August 26, 2014 11:42:24 AM UTC-7, frye wrote:
>
> I'm trying to use Automat to map out transitions that are either of the 
> paths below. 
>
>- There can be many claims sent to an external service. Each claim, 
>after being sent, can be :acknowledged or :rejected. 
>- If a claim is :acknowledged, it can then either be :accepted or 
>:rejected. 
>- There will be many such claims 
>
> [:sent :acknowledged :accepted]
> [:sent :acknowledged :rejected]
> [:sent :rejected]
>
>
> I'm trying to figure out how to use Automat to handle this use case. 
>
>- The only way to advance through your FSM, seems to be to call 
>fsm/advance (or fsm/advance-stream). How do we deal with forks in the road?
>- Looking at the documentation 
> and tests 
>
> , 
>it's not clear to me if I can pass multiple tokens (or claims) through 1 
>FSM. Are we meant to use 1 FSM per token (claim in this case)? 
>- The semantics for Automat have you declare transitions, instead of 
>states. States are implicit numbers between the transitions. Pallet-fsm 
> seems to follow the state 
>declaration pattern, but hasn't been updated in a few years. Can I name 
> the 
>states and transitions between them? 
>- There always seems to be one output state, even if they come from 2 
>different transitions. So the below 2 FSMs are equivalent. However, they 
>both go to the same end state. Is there a way to define different output 
>states?  
>
> => (require '[automat.viz :refer (view)])
> => (require '[automat.core :as a])
>
> => (view (a/or [:sent :acknowledged :accepted] [:sent :acknowledged 
> :rejected] [:sent :rejected]))
> => (view [:sent (a/or [:acknowledged (a/or :accepted :rejected)] 
> [:rejected])])
>
>
>
> Tim Washington 
> Interruptsoftware.com  
>
>  

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] data.int-map 0.1.0

[Zach Tellman]

https://github.com/clojure/data.int-map

This contrib library represents the union of two other libraries [1] [2], 
which are now both deprecated.  There's nothing too surprising here, but 
I'm happy to answer any questions.

Zach

[1] https://github.com/ztellman/immutable-int-map
[2] https://github.com/ztellman/immutable-bitset

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] async-sockets - work with sockets using core.async channels

[Zach Tellman]

If I'm reading this correctly, you're using non-blocking thread pools for 
blocking operations on the sockets.  Given more than N connections (last 
time I looked the thread pool's size was 42), you risk deadlock or at the 
very least poor average throughput.

On Sunday, October 5, 2014 7:06:56 PM UTC-7, Brian Guthrie wrote:
>
> Hi all,
>
> I'm releasing a little library for working with sockets. Feedback and pull 
> requests gratefully appreciated.
>
> The skinny
> ---
>
> This library allows you to create socket servers and socket clients and 
> interact with them asynchronously using channels. Servers return a record 
> with a :connections field, a channel which yields one socket per incoming 
> connection. Clients return the same socket record. Socket records each have 
> an :in and :out channel each which allow you to receive and send data 
> respectively on a line-by-line basis. The raw java.net.Socket is also 
> available (as :socket).
>
> Servers and clients are defined using the Component framework and must be 
> explicitly started using (component/start ), though 
> sockets will clean up after themselves if they are terminated for some 
> reason.
>
> Further information is available on Github here: 
> https://github.com/bguthrie/async-sockets
>
> Releases
> --
>
> This is the first release, which I've tagged for now as 0.0.1-SNAPSHOT. 
> Leiningen dependency: [com.gearswithingears/async-sockets "0.0.1-SNAPSHOT"].
>
> If this is useful to you, please let me know, but any and all feedback is 
> great.
>
> Happy hacking,
>
> Brian
> @bguthrie
> btgu...@gmail.com 
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Re: Handling java streams..

[Zach Tellman]

Calling (-> stream .iterator iterator-seq) should give the desired result.

On Wednesday, October 1, 2014 6:39:11 AM UTC-7, José Ricardo wrote:
>
> By nicer I meant something like being able to use, for example, map and 
> filter, just like I can do on a java.util.ArrayList (in clojure) and on a 
> Stream (in java).
>
> On Saturday, September 27, 2014 1:47:38 AM UTC-4, Sean Corfield wrote:
>>
>> On Fri, Sep 26, 2014 at 11:51 AM, José Ricardo  
>> wrote: 
>> > Hi, I'm not sure if resurrecting this thread is the right approach, but 
>> what 
>> > about Java 8 Streams (java.util.stream)? 
>> > 
>> > Are there any libraries out there for making java 8 streams handling 
>> nicer? 
>> > :) 
>>
>> Well, this thread is certainly a blast from the past :) 
>>
>> Can you be a bit more specific about what you'd like to see "nicer" 
>> with Java 8 streams, and what you'd like to see in a Clojure wrapper? 
>>
>> I ask because Java 8 is the first version that I've been interested in 
>> since I felt it went off the rails with Java 5 - and streams are part 
>> of what makes Java 8 attractive again :) 
>> -- 
>> Sean A Corfield -- (904) 302-SEAN 
>> An Architect's View -- http://corfield.org/ 
>> World Singles, LLC. -- http://worldsingles.com/ 
>>
>> "Perfection is the enemy of the good." 
>> -- Gustave Flaubert, French realist novelist (1821-1880) 
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] async-sockets - work with sockets using core.async channels

[Zach Tellman]

Please note that if you use core.async with java.nio, you need to make sure 
backpressure is properly propagated (this happens automatically with 
java.io, assuming you have a thread per connection).

On Sunday, October 5, 2014 9:10:24 PM UTC-7, adrian...@mail.yu.edu wrote:
>
> Zach makes an excellent point; I've used AsyncSocketChannels and its irk (
> http://docs.oracle.com/javase/8/docs/api/java/nio/channels/AsynchronousServerSocketChannel.html),
>  
> with core.async in the past. Perhaps replacing your direct java.net.Sockets 
> with nio classes that can be given CompletionHandlers (
> http://docs.oracle.com/javase/7/docs/api/java/nio/channels/CompletionHandler.html)
>  
> would be a better fit. 
>
> On Sunday, October 5, 2014 11:57:18 PM UTC-4, Zach Tellman wrote:
>>
>> If I'm reading this correctly, you're using non-blocking thread pools for 
>> blocking operations on the sockets.  Given more than N connections (last 
>> time I looked the thread pool's size was 42), you risk deadlock or at the 
>> very least poor average throughput.
>>
>> On Sunday, October 5, 2014 7:06:56 PM UTC-7, Brian Guthrie wrote:
>>>
>>> Hi all,
>>>
>>> I'm releasing a little library for working with sockets. Feedback and 
>>> pull requests gratefully appreciated.
>>>
>>> The skinny
>>> ---
>>>
>>> This library allows you to create socket servers and socket clients and 
>>> interact with them asynchronously using channels. Servers return a record 
>>> with a :connections field, a channel which yields one socket per incoming 
>>> connection. Clients return the same socket record. Socket records each have 
>>> an :in and :out channel each which allow you to receive and send data 
>>> respectively on a line-by-line basis. The raw java.net.Socket is also 
>>> available (as :socket).
>>>
>>> Servers and clients are defined using the Component framework and must 
>>> be explicitly started using (component/start ), though 
>>> sockets will clean up after themselves if they are terminated for some 
>>> reason.
>>>
>>> Further information is available on Github here: 
>>> https://github.com/bguthrie/async-sockets
>>>
>>> Releases
>>> --
>>>
>>> This is the first release, which I've tagged for now as 0.0.1-SNAPSHOT. 
>>> Leiningen dependency: [com.gearswithingears/async-sockets "0.0.1-SNAPSHOT"].
>>>
>>> If this is useful to you, please let me know, but any and all feedback 
>>> is great.
>>>
>>> Happy hacking,
>>>
>>> Brian
>>> @bguthrie
>>> btgu...@gmail.com
>>>
>>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] async-sockets - work with sockets using core.async channels

[Zach Tellman]

The reason the thread-per-connection approach is nice is because it 
correctly propagates backpressure.  If we're copying data from a source to 
a sink (let's say reading it in from the network and writing to a file), 
it's possible that the production of data may outstrip the consumption.  If 
this happens, we need to make sure the producer slows down, or we risk 
running out of memory.  In Java, the producer is typically connected to the 
consumer via a blocking queue, and if the queue fills up the producer can't 
send anything more to the consumer.  A Java socket is one such queue, and 
if it fills up it will exert backpressure via TCP.  This will work no 
matter how many queues or other mechanisms separate the producer and 
consumer.

However, every attempt I've seen to marry core.async to an async network 
stack has been fundamentally broken, in that it doesn't do this.  Often, 
they'll just use 'put!', which works fine until the channel's queue fills 
up, and 1024 pending puts are accumulated, and finally the channel throws 
an exception.  Alternately, they'll use a blocking put on the channel, 
which means that any backpressure will also extend to whatever other 
connections are sharing that thread or the thread pool.  Note that the 
software that uses core.async in this way may work flawlessly in a wide 
variety of cases, but there's still an intractable failure mode lying in 
wait.

In some cases, such as http-kit's websocket mechanism, there's no way to 
even exert backpressure (you register a callback, and have no way to 
indicate in your callback that you can't handle more messages).  This means 
that any attempt to use http-kit in conjunction with core.async will be 
subtly but fundamentally broken.  Arguably, even without core.async in the 
equation it's broken.  This is not a good state of affairs.  I'll admit 
that it took me a few failures in production to realize how important 
correct handling of backpressure is, but this isn't something that our 
ecosystem can afford to ignore, especially as Clojure is used for 
larger-scale projects.

I will note that I am working on a solution to this, in the form of the 
upcoming Aleph release [1].  This will model every network connection via 
streams that can trivially be converted into core.async channels [2], and 
which exert backpressure over TCP wherever necessary without requiring a 
thread per connection.  A formal beta should be available in the near 
future (it's already handling billions of requests a day in production 
without issue).

Zach

[1] https://github.com/ztellman/aleph/tree/0.4.0
[2] https://github.com/ztellman/manifold



On Tuesday, October 7, 2014 1:36:16 PM UTC-7, adrian...@mail.yu.edu wrote:
>
> It's not about 'safety' (depending on what that means in this context), 
> but as Zach pointed out, if you aren't careful about backpressure you can 
> run into performance bottlenecks with unrestrained async IO operations 
> because although they let you code as if you could handle an unlimited 
> amount of connections, obviously that isn't true. There is only a finite 
> amount of data that can be buffered in and out of any network according to 
> its hardware. When you don't regulate that, your system will end up 
> spending an inordinate amount of time compensating for this. You don't need 
> to worry about this with "regular io" because the "thread per connection" 
> abstraction effectively bounds your activity within the acceptable physical 
> constraints of the server. 
>
> On Tuesday, October 7, 2014 2:49:30 PM UTC-4, Brian Guthrie wrote:
>>
>>
>> On Mon, Oct 6, 2014 at 12:10 AM,  wrote:
>>
>>> Zach makes an excellent point; I've used AsyncSocketChannels and its irk 
>>> (
>>> http://docs.oracle.com/javase/8/docs/api/java/nio/channels/AsynchronousServerSocketChannel.html),
>>>  
>>> with core.async in the past. Perhaps replacing your direct java.net.Sockets 
>>> with nio classes that can be given CompletionHandlers (
>>> http://docs.oracle.com/javase/7/docs/api/java/nio/channels/CompletionHandler.html)
>>>  
>>> would be a better fit. 
>>>
>>
>> Once I do some performance instrumentation I'll give that a shot. I admit 
>> that I'm not familiar with all the implications of using the nio classes; 
>> were I to switch, is it safe to continue using go blocks, or is it worth 
>> explicitly allocating a single thread per socket?
>>
>> Brian
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to c

Re: [ANN] async-sockets - work with sockets using core.async channels

[Zach Tellman]

I wasn't aware of hermod, that's interesting.  I would still characterize
its approach to backpressure as "broken", though, since when the queues get
full it silently drops messages on the ground.  In fairness, this is very
clearly documented, so it's less pernicious than some of the other cases
out there.

Both core.async buffers and SelectableChannels have very particular
semantics, and I would be very surprised if they could be combined in that
way.  It's perfectly possible to feed one into the other and handle
backpressure properly (again, I'm doing just that with Aleph 0.4.0, using
Netty), but it's a nuanced integration and easy to get wrong.

On Wed, Oct 8, 2014 at 7:12 AM,  wrote:

> Check out https://github.com/halgari/com.tbaldridge.hermod for an
> interesting take on this.
>
> On Wednesday, October 8, 2014 1:17:11 AM UTC-4, Sun Ning wrote:
>>
>>  BTW, is there any network based core.async channel available now?
>>
>> On 10/08/2014 04:36 AM, adrian...@mail.yu.edu wrote:
>>
>>  It's not about 'safety' (depending on what that means in this context),
>> but as Zach pointed out, if you aren't careful about backpressure you can
>> run into performance bottlenecks with unrestrained async IO operations
>> because although they let you code as if you could handle an unlimited
>> amount of connections, obviously that isn't true. There is only a finite
>> amount of data that can be buffered in and out of any network according to
>> its hardware. When you don't regulate that, your system will end up
>> spending an inordinate amount of time compensating for this. You don't need
>> to worry about this with "regular io" because the "thread per connection"
>> abstraction effectively bounds your activity within the acceptable physical
>> constraints of the server.
>>
>> On Tuesday, October 7, 2014 2:49:30 PM UTC-4, Brian Guthrie wrote:
>>>
>>>
>>> On Mon, Oct 6, 2014 at 12:10 AM,  wrote:
>>>
 Zach makes an excellent point; I've used AsyncSocketChannels and its
 irk (http://docs.oracle.com/javase/8/docs/api/java/nio/channels/
 AsynchronousServerSocketChannel.html), with core.async in the past.
 Perhaps replacing your direct java.net.Sockets with nio classes that can be
 given CompletionHandlers (http://docs.oracle.com/
 javase/7/docs/api/java/nio/channels/CompletionHandler.html) would be a
 better fit.

>>>
>>> Once I do some performance instrumentation I'll give that a shot. I
>>> admit that I'm not familiar with all the implications of using the nio
>>> classes; were I to switch, is it safe to continue using go blocks, or is it
>>> worth explicitly allocating a single thread per socket?
>>>
>>>  Brian
>>>
>>  --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>   --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/TVMQJwaij1U/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] rmap - define lazy, recursive maps

[Zach Tellman]

Hi Arnout,

This is interesting, but may be a lot less useful than you think without 
filling in all the other methods that a normal map has.  For instance, you 
cannot do an equality check with another map (no use of 
clojure.lang.MapEquivalence or implementation of equals and equiv), nor use 
it in a set or key in another map (no implementation of hashCode and 
hasheq), etc. etc.  Unless you and everyone else using it is certain that 
every line of code in your application and the libraries you depend on 
don't do this, this is a ticking timebomb.

For an example of a full map implementation, it might be helpful to take a 
look at 
https://github.com/ztellman/potemkin/blob/master/src/potemkin/collections.clj, 
and will definitely be useful to 
run https://github.com/ztellman/collection-check against your map 
implementation.  One issue that collection-check would uncover is the fact 
that the underlying LinkedHashMap that you're using has different key 
equality semantics than Clojure does, so calling (-> m (assoc 1 :foo) 
(dissoc 1N)) would likely break a bunch of stuff.  

Please don't be discouraged by this, making custom map-like data structures 
is much harder right now than it should be.  I just want to make sure no 
one gets an unpleasant surprise in production somewhere down the line.

Best,
Zach

On Monday, October 6, 2014 1:45:02 AM UTC-7, Arnout Roemers wrote:
>
> Thank you for your kind responses! I will look into writing a post that 
> shows what the motivation was (i.e. how I use it), how it works, and how it 
> compares to Prismatic's Graph. 
>
> Cheers,
> Arnout
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] async-sockets - work with sockets using core.async channels

[Zach Tellman]

The documentation for Manifold can explain the API better than I can here.
The point where that interacts with Netty w.r.t. backpressure is here:
https://github.com/ztellman/aleph/blob/0.4.0/src/aleph/netty.clj#L109.
Here the stream represents data coming off the wire, and if the put onto
the stream is not immediately successful, backpressure is enabled until the
put completes.  No blocking required anywhere.

On Wed, Oct 8, 2014 at 10:10 AM, Jozef Wagner 
wrote:

> If you want to handle multiple TCP connections and async channels in one
> thread, you need a way how to block on both connections (wait for new input
> to arrive) and channels (wait for a free space in a buffer). Blocking only
> on connections will get you a busy loop if channels are full. If you could
> point me to the part of Aleph sources that handles this issue, I would be
> very grateful. I'm not familiar with netty API nor manifold's concepts, so
> I'm having trouble navigating in the Aleph sources.
>
> Thanks,
> Jozef
>
> On Wednesday, October 8, 2014 6:15:47 PM UTC+2, Zach Tellman wrote:
>>
>> I wasn't aware of hermod, that's interesting.  I would still characterize
>> its approach to backpressure as "broken", though, since when the queues get
>> full it silently drops messages on the ground.  In fairness, this is very
>> clearly documented, so it's less pernicious than some of the other cases
>> out there.
>>
>> Both core.async buffers and SelectableChannels have very particular
>> semantics, and I would be very surprised if they could be combined in that
>> way.  It's perfectly possible to feed one into the other and handle
>> backpressure properly (again, I'm doing just that with Aleph 0.4.0, using
>> Netty), but it's a nuanced integration and easy to get wrong.
>>
>> On Wed, Oct 8, 2014 at 7:12 AM,  wrote:
>>
>>> Check out https://github.com/halgari/com.tbaldridge.hermod for an
>>> interesting take on this.
>>>
>>> On Wednesday, October 8, 2014 1:17:11 AM UTC-4, Sun Ning wrote:
>>>>
>>>>  BTW, is there any network based core.async channel available now?
>>>>
>>>> On 10/08/2014 04:36 AM, adrian...@mail.yu.edu wrote:
>>>>
>>>>  It's not about 'safety' (depending on what that means in this
>>>> context), but as Zach pointed out, if you aren't careful about backpressure
>>>> you can run into performance bottlenecks with unrestrained async IO
>>>> operations because although they let you code as if you could handle an
>>>> unlimited amount of connections, obviously that isn't true. There is only a
>>>> finite amount of data that can be buffered in and out of any network
>>>> according to its hardware. When you don't regulate that, your system will
>>>> end up spending an inordinate amount of time compensating for this. You
>>>> don't need to worry about this with "regular io" because the "thread per
>>>> connection" abstraction effectively bounds your activity within the
>>>> acceptable physical constraints of the server.
>>>>
>>>> On Tuesday, October 7, 2014 2:49:30 PM UTC-4, Brian Guthrie wrote:
>>>>>
>>>>>
>>>>> On Mon, Oct 6, 2014 at 12:10 AM,  wrote:
>>>>>
>>>>>> Zach makes an excellent point; I've used AsyncSocketChannels and its
>>>>>> irk (http://docs.oracle.com/javase/8/docs/api/java/nio/channels/
>>>>>> AsynchronousServerSocketChannel.html), with core.async in the past.
>>>>>> Perhaps replacing your direct java.net.Sockets with nio classes that can 
>>>>>> be
>>>>>> given CompletionHandlers (http://docs.oracle.com/javase
>>>>>> /7/docs/api/java/nio/channels/CompletionHandler.html) would be a
>>>>>> better fit.
>>>>>>
>>>>>
>>>>> Once I do some performance instrumentation I'll give that a shot. I
>>>>> admit that I'm not familiar with all the implications of using the nio
>>>>> classes; were I to switch, is it safe to continue using go blocks, or is 
>>>>> it
>>>>> worth explicitly allocating a single thread per socket?
>>>>>
>>>>>  Brian
>>>>>
>>>>  --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clo...@googlegroups.com
>

Re: [ANN] async-sockets - work with sockets using core.async channels

[Zach Tellman]

Sorry, didn't cover converse case.  That's handled by the ChannelSink
directly underneath.  Note that each write returns a Netty ChannelFuture
representing completion of the write, which is transformed into a Manifold
deferred.  Any time a Manifold put returns an unrealized deferred, that
creates upstream backpressure.

On Wed, Oct 8, 2014 at 10:16 AM, Zach Tellman  wrote:

> The documentation for Manifold can explain the API better than I can
> here.  The point where that interacts with Netty w.r.t. backpressure is
> here:
> https://github.com/ztellman/aleph/blob/0.4.0/src/aleph/netty.clj#L109.
> Here the stream represents data coming off the wire, and if the put onto
> the stream is not immediately successful, backpressure is enabled until the
> put completes.  No blocking required anywhere.
>
> On Wed, Oct 8, 2014 at 10:10 AM, Jozef Wagner 
> wrote:
>
>> If you want to handle multiple TCP connections and async channels in one
>> thread, you need a way how to block on both connections (wait for new input
>> to arrive) and channels (wait for a free space in a buffer). Blocking only
>> on connections will get you a busy loop if channels are full. If you could
>> point me to the part of Aleph sources that handles this issue, I would be
>> very grateful. I'm not familiar with netty API nor manifold's concepts, so
>> I'm having trouble navigating in the Aleph sources.
>>
>> Thanks,
>> Jozef
>>
>> On Wednesday, October 8, 2014 6:15:47 PM UTC+2, Zach Tellman wrote:
>>>
>>> I wasn't aware of hermod, that's interesting.  I would still
>>> characterize its approach to backpressure as "broken", though, since when
>>> the queues get full it silently drops messages on the ground.  In fairness,
>>> this is very clearly documented, so it's less pernicious than some of the
>>> other cases out there.
>>>
>>> Both core.async buffers and SelectableChannels have very particular
>>> semantics, and I would be very surprised if they could be combined in that
>>> way.  It's perfectly possible to feed one into the other and handle
>>> backpressure properly (again, I'm doing just that with Aleph 0.4.0, using
>>> Netty), but it's a nuanced integration and easy to get wrong.
>>>
>>> On Wed, Oct 8, 2014 at 7:12 AM,  wrote:
>>>
>>>> Check out https://github.com/halgari/com.tbaldridge.hermod for an
>>>> interesting take on this.
>>>>
>>>> On Wednesday, October 8, 2014 1:17:11 AM UTC-4, Sun Ning wrote:
>>>>>
>>>>>  BTW, is there any network based core.async channel available now?
>>>>>
>>>>> On 10/08/2014 04:36 AM, adrian...@mail.yu.edu wrote:
>>>>>
>>>>>  It's not about 'safety' (depending on what that means in this
>>>>> context), but as Zach pointed out, if you aren't careful about 
>>>>> backpressure
>>>>> you can run into performance bottlenecks with unrestrained async IO
>>>>> operations because although they let you code as if you could handle an
>>>>> unlimited amount of connections, obviously that isn't true. There is only 
>>>>> a
>>>>> finite amount of data that can be buffered in and out of any network
>>>>> according to its hardware. When you don't regulate that, your system will
>>>>> end up spending an inordinate amount of time compensating for this. You
>>>>> don't need to worry about this with "regular io" because the "thread per
>>>>> connection" abstraction effectively bounds your activity within the
>>>>> acceptable physical constraints of the server.
>>>>>
>>>>> On Tuesday, October 7, 2014 2:49:30 PM UTC-4, Brian Guthrie wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 6, 2014 at 12:10 AM,  wrote:
>>>>>>
>>>>>>> Zach makes an excellent point; I've used AsyncSocketChannels and its
>>>>>>> irk (http://docs.oracle.com/javase/8/docs/api/java/nio/channels/
>>>>>>> AsynchronousServerSocketChannel.html), with core.async in the past.
>>>>>>> Perhaps replacing your direct java.net.Sockets with nio classes that 
>>>>>>> can be
>>>>>>> given CompletionHandlers (http://docs.oracle.com/javase
>>>>>>> /7/docs/api/java/nio/channels/CompletionHandler.html) would be a
>>>>>>> better fit.
>>>&g

Re: [ANN] async-sockets - work with sockets using core.async channels

[Zach Tellman]

Yes, I didn't mean to imply that there are no blocking operations anywhere,
only that nothing in the worker threads (where the put! occurs) will block,
and that backpressure isn't exerted by causing an active thread to hang.

As to the second point, I'm not sure why you'd think that.  These aren't
futures that are being accessed via blocking dereference, so there's no
reason any thread would block as a result of an unrealized future/deferred.



On Wed, Oct 8, 2014 at 12:22 PM, Jozef Wagner 
wrote:

> Thank you! Using put! callback to control backpressure is a very elegant
> solution. BTW there always has to be blocking somewhere. Netty uses
> selector to block at [1] and .setAutoRead causes respective channel to
> deregister itself from a selector [2], until put! completes.
>
> Regarding the other way around, it seems to me from the quick look (sorry
> if I misinterpreted) that using futures to represent write completion will
> cause the eventual backpressure to block the thread, causing analogous
> drawback as a blocking put! in the "reading from multiple connection" case.
>
> [1]
> https://github.com/netty/netty/blob/220660e351b2a22112b19c4af45e403eab1f73ab/transport/src/main/java/io/netty/channel/nio/NioEventLoop.java#L625
> [2]
> https://github.com/netty/netty/blob/220660e351b2a22112b19c4af45e403eab1f73ab/transport/src/main/java/io/netty/channel/nio/AbstractNioChannel.java#L173
>
> Jozef
>
> On Wednesday, October 8, 2014 7:16:44 PM UTC+2, Zach Tellman wrote:
>>
>> The documentation for Manifold can explain the API better than I can
>> here.  The point where that interacts with Netty w.r.t. backpressure is
>> here: https://github.com/ztellman/aleph/blob/0.4.0/src/
>> aleph/netty.clj#L109.  Here the stream represents data coming off the
>> wire, and if the put onto the stream is not immediately successful,
>> backpressure is enabled until the put completes.  No blocking required
>> anywhere.
>>
>> On Wed, Oct 8, 2014 at 10:10 AM, Jozef Wagner 
>> wrote:
>>
>>> If you want to handle multiple TCP connections and async channels in one
>>> thread, you need a way how to block on both connections (wait for new input
>>> to arrive) and channels (wait for a free space in a buffer). Blocking only
>>> on connections will get you a busy loop if channels are full. If you could
>>> point me to the part of Aleph sources that handles this issue, I would be
>>> very grateful. I'm not familiar with netty API nor manifold's concepts, so
>>> I'm having trouble navigating in the Aleph sources.
>>>
>>> Thanks,
>>> Jozef
>>>
>>> On Wednesday, October 8, 2014 6:15:47 PM UTC+2, Zach Tellman wrote:
>>>>
>>>> I wasn't aware of hermod, that's interesting.  I would still
>>>> characterize its approach to backpressure as "broken", though, since when
>>>> the queues get full it silently drops messages on the ground.  In fairness,
>>>> this is very clearly documented, so it's less pernicious than some of the
>>>> other cases out there.
>>>>
>>>> Both core.async buffers and SelectableChannels have very particular
>>>> semantics, and I would be very surprised if they could be combined in that
>>>> way.  It's perfectly possible to feed one into the other and handle
>>>> backpressure properly (again, I'm doing just that with Aleph 0.4.0, using
>>>> Netty), but it's a nuanced integration and easy to get wrong.
>>>>
>>>> On Wed, Oct 8, 2014 at 7:12 AM,  wrote:
>>>>
>>>>> Check out https://github.com/halgari/com.tbaldridge.hermod for an
>>>>> interesting take on this.
>>>>>
>>>>> On Wednesday, October 8, 2014 1:17:11 AM UTC-4, Sun Ning wrote:
>>>>>>
>>>>>>  BTW, is there any network based core.async channel available now?
>>>>>>
>>>>>> On 10/08/2014 04:36 AM, adrian...@mail.yu.edu wrote:
>>>>>>
>>>>>>  It's not about 'safety' (depending on what that means in this
>>>>>> context), but as Zach pointed out, if you aren't careful about 
>>>>>> backpressure
>>>>>> you can run into performance bottlenecks with unrestrained async IO
>>>>>> operations because although they let you code as if you could handle an
>>>>>> unlimited amount of connections, obviously that isn't true. There is 
>>>>>> only a
>>>>>> finite amount of data th

Re: [ANN] async-sockets - work with sockets using core.async channels

[Zach Tellman]

A slightly more straightforward implementation can be found 
at https://gist.github.com/ztellman/fb64e81d1d7f0b261ccd.  I'm fairly sure 
it's equivalent to Cristophe's, but I may be missing some nuance.  At any 
rate, I've found using the async/put! and callback mechanism to be a much 
more straightforward way to do interop with non-core.async code.

And yes, reimplementing a TCP-like ack mechanism on top of WebSockets is 
not something you want to do.  The existing stack will do it better and 
faster than you can.  Just to be clear, this is a large part of why I wrote 
Manifold [1], which can easily be turned into a core.async channel, but 
provides an API which is designed for interop with other stream mechanisms 
(including synchronous ones like Java's BlockingQueues).  core.async is a 
framework, meaning it brings not only a stream representation, but an 
entire execution model; using core.async should be an application-level 
decision, not one made for you by your libraries

Zach

[1] https://github.com/ztellman/manifold


On Sunday, October 12, 2014 9:42:32 AM UTC-7, Ryan Waters wrote:
>
> I was just starting to use Sente [1] (which relies on httpkit [2]) and 
> this conversation is a real eye opener.  Unless a person uses a library 
> that supports backpressure, as mentioned earlier, your 
> transport-concern-made-opaque-because-of-core-async must become an 
> application-level concern.  The far-side of the communication would have to 
> respond with an application level acknowledgement for local sends and the 
> local side would need to not send data unless acks were received for 
> previously sent data.
>
> E.g. this could be implemented with core.async by using a pair of channels 
> (instead of a single channel) for all 'sends' where one channel is used for 
> data while the other channel waits for acknowledgement of put data (a 
> 'control' channel).  This would have the unfortunate side effect of hurting 
> throughput.  A better system would be to allow for a certain number of 
> unacknowledged sends before backing off.  Of course, now a person is 
> implementing what was created for TCP at the level of their application.
>
> Christophe's approach means you at least wouldn't have to do the above, 
> replacing it instead with a per backend implementation.  I hope somebody 
> else is able to explain it better.
>
> Looking forward to an Aleph rewrite!!
>
> - - -
> [1] https://github.com/ptaoussanis/sente
> [2] https://github.com/http-kit/http-kit
>
> On Sat, Oct 11, 2014 at 8:01 PM, Julian 
> > wrote:
>
>> Hi Zach, 
>>
>> Thanks for the clarity of thought that went into this post. 
>>
>> Perhaps it is obvious to everyone but me, but I saw this post by 
>> Christophe Grande yesterday that appears to address these concerns:
>> "Back-pressurized interop for core.async" 
>> https://twitter.com/cgrand/status/520566182194450432
>> https://gist.github.com/cgrand/767673242b7f7c27f35a
>>
>> I'm interested to hear if this solves your problem or is about something 
>> else. 
>>
>> Cheers
>> Julian
>>
>>
>> On Wednesday, 8 October 2014 17:00:02 UTC+11, Zach Tellman wrote:
>>>
>>> The reason the thread-per-connection approach is nice is because it 
>>> correctly propagates backpressure.  If we're copying data from a source to 
>>> a sink (let's say reading it in from the network and writing to a file), 
>>> it's possible that the production of data may outstrip the consumption.  If 
>>> this happens, we need to make sure the producer slows down, or we risk 
>>> running out of memory.  In Java, the producer is typically connected to the 
>>> consumer via a blocking queue, and if the queue fills up the producer can't 
>>> send anything more to the consumer.  A Java socket is one such queue, and 
>>> if it fills up it will exert backpressure via TCP.  This will work no 
>>> matter how many queues or other mechanisms separate the producer and 
>>> consumer.
>>>
>>> However, every attempt I've seen to marry core.async to an async network 
>>> stack has been fundamentally broken, in that it doesn't do this.  Often, 
>>> they'll just use 'put!', which works fine until the channel's queue fills 
>>> up, and 1024 pending puts are accumulated, and finally the channel throws 
>>> an exception.  Alternately, they'll use a blocking put on the channel, 
>>> which means that any backpressure will also extend to whatever other 
>>> connections are sharing that thread or the thread pool.  Note that the 
>>> software 

updated web server benchmarks

[Zach Tellman]

New benchmarks have been run for a variety of Clojure web servers, and the 
results can be found 
at 
https://github.com/ptaoussanis/clojure-web-server-benchmarks#ubuntu-1404--2x-xeon-x5650.
 
 The top-performing servers are based on Undertow (ring-undertow, Immutant 
v2), Netty (Aleph), and nginx (nginx-clojure) respectively.

The benchmark hardware was dual-socket 2010-era Xeons, which is the 
hardware profile of choice at Factual.  This means that there were 24 
hardware threads, but each core was noticeably slower than those on modern 
desktop processors.  This caused http-kit, which only uses a single thread 
for I/O, to perform worse than it would have on a system with fewer, faster 
cores.  

As always, these benchmarks are very contrived, and not representative of 
most real-world applications.  Adding something as simple as Ring's 
`wrap-keyword-params` middleware on Clojure 1.6.0 would reduce throughput 
on all servers by a factor of 3-4x.  Luckily keyword creation will be much 
cheaper in Clojure 1.7.0, so this particular problem won't exist for much 
longer, but it's important to remember that performance is a property of 
your entire application, not just the underlying web server.

With that said, it's great to see that so many people are taking server 
performance in Clojure seriously.  However contrived these benchmarks are, 
a lot of people use them as a proxy for a language's "maturity", and 
further effort in this space can only help Clojure's adoption.

Zach

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] s3-journal, a library for high-throughput log journaling

[Zach Tellman]

Today Factual is open sourcing two libraries, the first of which is a 
relatively simple library that journals data to Amazon's S3 service [1], 
and buffers data on disk using durable-queue [2], making it much more 
robust than most S3 clients to process failure, network issues, and a host 
of other issues.  We have used it to write over a trillion entries this 
year, and expect it will become equally indispensable for others in the 
Clojure community.

A blog post describing it in more detail can be found at 
http://blog.factual.com/how-factual-uses-persistent-storage-for-its-real-time-services.
 
 The library itself can be found at https://github.com/factual/s3-journal.

Zach 

[1] http://aws.amazon.com/s3/
[2] https://github.com/factual/durable-queue

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] riffle, a high-performance write-once key/value storage engine

[Zach Tellman]

The second of the libraries Factual is open sourcing today is Riffle, which 
is a write-once key/value storage engine.  By disallowing random writes, it 
allows for O(1) on-disk reads, and a read throughput is both faster and 
significantly more stable than LevelDB and other, similar databases.  Of 
course, any changes to the database require merging two existing databases 
or writing another one from scratch, which can be done either in-process or 
using Hadoop.  

A blog post describing the library can be found 
at 
http://blog.factual.com/how-factual-uses-persistent-storage-for-its-real-time-services,
 
and the library itself can be found at https://github.com/factual/riffle. 
 We look forward to see how it's used by others.

Zach

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Server Sent Events under Http-Kit

[Zach Tellman]

I'm not sure about http-kit's streaming response implementation, but in 
Aleph [1] if you use a stream/channel as a response body each message from 
the stream will be immediately sent as an HTTP chunk.

Best,
Zach

[1] https://github.com/ztellman/aleph

On Saturday, December 6, 2014 9:03:41 AM UTC-8, Lars Ole Avery Simonsen 
wrote:
>
> Hi guys
>
> I am trying to implement Server Sent Event support in a small hobby 
> project based on the http-kit server framework and compojure.
>
> I am still quite new to clojure in general, so it may very well be that I 
> am missing something obvious, but this SSE detail has me stumped.
>
> What I have tried is to take inspiration from the Eventual 
>  library for the SSE implementation 
> (which is aimed at jetty). I have changed the Eventual implementation so 
> that it uses the http-kit async-channel implementation for data 
> transmission. What I am seeing is that everything seems to run without 
> errors except for the fact that no data is being received at the 
> EventSource in the browser.
>
> If I allow the sse channel to close immediately after opening (which is 
> pretty useless), the initial message is received at the browser side, but 
> if I keep the connection open, nothing gets received.
>
> Having had a look at the http-kit internals, my suspicion is that the 
> socket doesn't get flushed and so all my attempts at communication are 
> sitting in some buffer somewhere.
>
> All SSE implementations I have looked at for other platforms have explicit 
> calls to flush the socket buffers on the long lived connection after each 
> message transmission.
>
> Does anyone have any suggestions as to a better way to achieve SSE support 
> in a http-kit based server? Or maybe ideas as to what I might be doing 
> wrong?
>
> Thanks in advance.
>
> LOAS
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Server Sent Events under Http-Kit

[Zach Tellman]

Does `send!` block if it needs to exert backpressure?  If so, the `go-loop`
seems dangerous.  If not, how does backpressure work?

On Wed, Dec 17, 2014 at 8:07 AM, Malcolm Sparks  wrote:
>
> Here is a solution for SSE over http-kit channels
>
> (defn server-event-source [ch] ;; Adding a mult here is dangerous because
> it bleeds the underlying ;; channel dry. We should provide the mult via
> modular.async (let [m (async/mult ch)] (fn [req] (let [ch (async/chan 16)]
> (async/tap m ch) (with-channel req net-ch (on-close net-ch (fn [_]
> (async/untap m ch) (async/close! ch))) (send! net-ch {:headers headers}
> false) (go-loop [] (when-let [data ( (send! net-ch (->message data) false) (recur
>
>
> https://github.com/juxt/modular/blob/master/modules/http-kit-events/src/modular/http_kit/events.clj
>
>
> On Saturday, 6 December 2014 17:03:41 UTC, Lars Ole Avery Simonsen wrote:
>>
>> Hi guys
>>
>> I am trying to implement Server Sent Event support in a small hobby
>> project based on the http-kit server framework and compojure.
>>
>> I am still quite new to clojure in general, so it may very well be that I
>> am missing something obvious, but this SSE detail has me stumped.
>>
>> What I have tried is to take inspiration from the Eventual
>>  library for the SSE implementation
>> (which is aimed at jetty). I have changed the Eventual implementation so
>> that it uses the http-kit async-channel implementation for data
>> transmission. What I am seeing is that everything seems to run without
>> errors except for the fact that no data is being received at the
>> EventSource in the browser.
>>
>> If I allow the sse channel to close immediately after opening (which is
>> pretty useless), the initial message is received at the browser side, but
>> if I keep the connection open, nothing gets received.
>>
>> Having had a look at the http-kit internals, my suspicion is that the
>> socket doesn't get flushed and so all my attempts at communication are
>> sitting in some buffer somewhere.
>>
>> All SSE implementations I have looked at for other platforms have
>> explicit calls to flush the socket buffers on the long lived connection
>> after each message transmission.
>>
>> Does anyone have any suggestions as to a better way to achieve SSE
>> support in a http-kit based server? Or maybe ideas as to what I might be
>> doing wrong?
>>
>> Thanks in advance.
>>
>> LOAS
>>
>>  --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/DbBzM3AG9wM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Server Sent Events under Http-Kit

[Zach Tellman]

In Aleph, you simply return the channel as the body of the response [1],
and messages are consumed only as they're able to be sent over the
network.  This means that backpressure "just works" without any need to
poll whether downstream buffers are full.  Obviously you can poll using the
put-with-timeout mechanism in core.async et al if you want, though.

[1] https://gist.github.com/ztellman/613629a2a32908d096d5

On Wed, Dec 17, 2014 at 10:42 AM, Malcolm Sparks  wrote:
>
> http-kit's send! doesn't block normally -
>
> From http://www.http-kit.org/server.html :-
>
> send!: Sends data to client and returns true if the data was successfully
> written to the output queue, or false if the channel is closed. Normally,
> checking the returned value is not needed. This function returns
> immediately (does not block).
>
> http-kit will attempt to write to the output queue, otherwise will place
> the message in a pending queue backed by a
> java.util.LinkedList, so there's no back-pressure (except for
> that exerted by an OutOfMemory exception)
>
> AFAICT there's no way of finding whether http-kit's output channel is
> empty or not, that would certainly be a feature I'd like to see.
>
> Does aleph support this?
>
>
>
>
> On 17 December 2014 at 18:23, Zach Tellman  wrote:
>>
>> Does `send!` block if it needs to exert backpressure?  If so, the
>> `go-loop` seems dangerous.  If not, how does backpressure work?
>>
>> On Wed, Dec 17, 2014 at 8:07 AM, Malcolm Sparks  wrote:
>>>
>>> Here is a solution for SSE over http-kit channels
>>>
>>> (defn server-event-source [ch] ;; Adding a mult here is dangerous
>>> because it bleeds the underlying ;; channel dry. We should provide the
>>> mult via modular.async (let [m (async/mult ch)] (fn [req] (let [ch
>>> (async/chan 16)] (async/tap m ch) (with-channel req net-ch (on-close
>>> net-ch (fn [_] (async/untap m ch) (async/close! ch))) (send! net-ch {
>>> :headers headers} false) (go-loop [] (when-let [data (>> data! " data) (send! net-ch (->message data) false) (recur
>>>
>>>
>>> https://github.com/juxt/modular/blob/master/modules/http-kit-events/src/modular/http_kit/events.clj
>>>
>>>
>>> On Saturday, 6 December 2014 17:03:41 UTC, Lars Ole Avery Simonsen wrote:
>>>>
>>>> Hi guys
>>>>
>>>> I am trying to implement Server Sent Event support in a small hobby
>>>> project based on the http-kit server framework and compojure.
>>>>
>>>> I am still quite new to clojure in general, so it may very well be that
>>>> I am missing something obvious, but this SSE detail has me stumped.
>>>>
>>>> What I have tried is to take inspiration from the Eventual
>>>> <https://github.com/ninjudd/eventual> library for the SSE
>>>> implementation (which is aimed at jetty). I have changed the Eventual
>>>> implementation so that it uses the http-kit async-channel implementation
>>>> for data transmission. What I am seeing is that everything seems to run
>>>> without errors except for the fact that no data is being received at the
>>>> EventSource in the browser.
>>>>
>>>> If I allow the sse channel to close immediately after opening (which is
>>>> pretty useless), the initial message is received at the browser side, but
>>>> if I keep the connection open, nothing gets received.
>>>>
>>>> Having had a look at the http-kit internals, my suspicion is that the
>>>> socket doesn't get flushed and so all my attempts at communication are
>>>> sitting in some buffer somewhere.
>>>>
>>>> All SSE implementations I have looked at for other platforms have
>>>> explicit calls to flush the socket buffers on the long lived connection
>>>> after each message transmission.
>>>>
>>>> Does anyone have any suggestions as to a better way to achieve SSE
>>>> support in a http-kit based server? Or maybe ideas as to what I might be
>>>> doing wrong?
>>>>
>>>> Thanks in advance.
>>>>
>>>> LOAS
>>>>
>>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, s

Re: Newbie Gloss questions - dynamic buffer structure

[Zach Tellman]

Hey Tzach,

If I understand what you're trying to do, you want something like this:

[ :uint32
  :ubyte ;; or bit-seq, depending
  (delimited-block (prefix uint24 #(- % 5) #(+ % 5))) ]

And then you'll need to parse the final block separately.  I've left 
defining the uint24 as an exercise for the reader (sorry, long day), but 
using (compile-frame [:uint16 :uint8] ...) should be pretty straightforward.

Hope that helps, happy to answer any followup questions.

Zach

On Saturday, January 3, 2015 11:51:14 PM UTC-8, Tzach wrote:
>
> I'm trying to work with Gloss binary encoder/decoder, and need some help 
> to kick start.
>
> My first task is simple(not for me ;)
> I have the following binary buffer to read/write:
>
>- 4 byte (32 bit) - code (uint)
>- 8 bit - misc flags
>- 3 byte (24 bit) - the entire buffer length
>- 4 byte (32 bit) uint value -  optional, depending on on of the flags.
>- List of 4 byte (uints) data elements - size dependent on the overall 
>size 
>
> How should I represent this structure?
> Clearly I need to use *prefix* and *repeated* for the last element, but I 
> failed to understand how.
> I also struggle with the optional element and how to represent a 3 byte 
> element.
>  
> Thanks!
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A (foolish) plan to re-invent IO on top of core.async

[Zach Tellman]

Aleph has always allowed that, but the latest release allows the asynchrony 
to be modeled using core.async channels if that's to your taste.

On Wednesday, January 7, 2015 at 9:52:53 AM UTC-8, Paul deGrandis wrote:
>
> There was another discussion on this list regarding async IO and web 
> servers.  It may be rather informative to you: 
> https://groups.google.com/d/msg/clojure/rKqT13Ofy4k/H9xvkZA9Yy4J
>
> To my knowledge, Pedestal is the only web library that let's you go async 
> all the way down to the wire (potentially the latest Aleph also allows for 
> this).  The benefit provided is that Pedestal manages the NIO integration 
> directly with the container for you - optimized to the specific container.  
> You may thumb through the implementation for some ideas.
>
> You also have to be very mindful about back-pressure when using core.async 
> in certain combinations.  Zach Tellman has covered the major points here: 
> https://groups.google.com/d/msg/clojure/TVMQJwaij1U/dQxyBxxbIjQJ
>
> Cheers,
> Paul
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Newbie Gloss questions - dynamic buffer structure

[Zach Tellman]

I'm sorry, that should have been :ubyte. Being able to insert arbitrary
data into a codec is a feature, though.

On Mon, Jan 12, 2015 at 6:27 AM, Tzach  wrote:

> Hi Zach
> Thanks for the detailed response
>
> On Wednesday, January 7, 2015 at 7:28:06 AM UTC+2, Zach Tellman wrote:
>>
>> Hey Tzach,
>>
>> If I understand what you're trying to do, you want something like this:
>>
>> [ :uint32
>>   :ubyte ;; or bit-seq, depending
>>   (delimited-block (prefix uint24 #(- % 5) #(+ % 5))) ]
>>
>> And then you'll need to parse the final block separately.  I've left
>> defining the uint24 as an exercise for the reader (sorry, long day), but
>> using (compile-frame [:uint16 :uint8] ...) should be pretty straightforward.
>>
> This is what work for me, in case anyone is interested or want to comment
>
> (defn valid-length [l]
>  (and (< l 16777216)
>   (>= l 0)))
>
> (def uint24 (compile-frame [:ubyte :uint16]
>(fn [u]
>  {:pre  [(valid-length u)]}
>  (let [u8 (bit-shift-right u 16)
>u16 (- u (bit-shift-left u8 16))]
>[u8 u16]))
>(fn [[u8 u16]]
>  {:post [(valid-length %)]}
>  (+ (bit-shift-left u8 16) u16
>
> (def avp
>   (compile-frame
>[:uint32
> :ubyte
> (repeated :int32 :prefix (prefix uint24 #(- % 5) #(+ % 5))) ]
> ))
>
> My next step is to parse the bits from the header :ubyte and base on it,
> read an extra uint32 before the repeated part.
> The header is probably the answer.
>
>
>>
>> Hope that helps, happy to answer any followup questions.
>>
>
> It did help! Thanks
>
> BTW, I spent too much time trying to use (compile-frame [:uint16 :uint8])
> before realizing :uint8 is not defined.
> The result is simply encoding of the :uint8 symbol.
> I wonder if its a bug or a feature.
>
>
>
>>
>> Zach
>>
>> On Saturday, January 3, 2015 11:51:14 PM UTC-8, Tzach wrote:
>>>
>>> I'm trying to work with Gloss binary encoder/decoder, and need some help
>>> to kick start.
>>>
>>> My first task is simple(not for me ;)
>>> I have the following binary buffer to read/write:
>>>
>>>- 4 byte (32 bit) - code (uint)
>>>- 8 bit - misc flags
>>>- 3 byte (24 bit) - the entire buffer length
>>>- 4 byte (32 bit) uint value -  optional, depending on on of the
>>>flags.
>>>- List of 4 byte (uints) data elements - size dependent on the
>>>overall size
>>>
>>> How should I represent this structure?
>>> Clearly I need to use *prefix* and *repeated* for the last element, but
>>> I failed to understand how.
>>> I also struggle with the optional element and how to represent a 3 byte
>>> element.
>>>
>>> Thanks!
>>>
>>  --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/NxsCPBk12mE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Virgil 0.1.6

[Zach Tellman]

I figured it was worth reminding everyone that this library 
exists: https://github.com/ztellman/virgil.  It now seems to work on Java 
projects of arbitrary size and structure (the in-process compiler is very 
fussy about compile order, you need to topologically sort the classes), and 
I've been using it extensively this last month while writing a bunch of 
Java [1].  I can tweak some Java, wait a second, and then test that code at 
the REPL.  It's been pretty game-changing for my workflow.

If anyone has questions, I'm happy to answer them.

Zach

[1] https://github.com/lacuna/bifurcan

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Virgil 0.1.6

[Zach Tellman]

The code is split into a library and a minimal Leiningen plugin which runs 
it at startup.  If someone wants to contribute a Boot equivalent, I'm happy 
to accept that PR.

On Thursday, March 16, 2017 at 11:18:16 AM UTC-7, Gregg Reynolds wrote:
>
>
>
> On Mar 16, 2017 1:13 PM, "Zach Tellman" > 
> wrote:
>
> I figured it was worth reminding everyone that this library exists: 
> https://github.com/ztellman/virgil.  It now seems to work on Java 
> projects of arbitrary size and structure (the in-process compiler is very 
> fussy about compile order, you need to topologically sort the classes), and 
> I've been using it extensively this last month while writing a bunch of 
> Java [1].  I can tweak some Java, wait a second, and then test that code at 
> the REPL.  It's been pretty game-changing for my workflow.
>
> If anyone has questions, I'm happy to answer them.
>
>
> wow!  looks great, when's the boot version due out? 😊
>
>
> Zach
>
> [1] https://github.com/lacuna/bifurcan
>
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com 
> Note that posts from new members are moderated - please be patient with 
> your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com 
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+u...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN and RFC] Bifurcan: impure functional data strucures

[Zach Tellman]

This is a slightly irregular announcement, because it's not for a Clojure 
library.  Rather, it's for a library written purely in 
Java: https://github.com/lacuna/bifurcan.

This is a collection of mutable and immutable data structures, designed to 
address some of my personal frustrations with what's available in the 
Clojure and Java ecosystems.  Notably, they have pluggable equality 
semantics, so while they *can* use Clojure's expensive hash and equality 
checks, they don't *have* to.  They also provide high-performance mutable 
variants of the data structure which share an API with their immutable 
cousins.  

I'm posting it here to ask for people's thoughts on how, if at all, this 
should be exposed as a Clojure library.  It would be simple to simply wrap 
them in the Clojure interfaces and make them behave identically to 
Clojure's own data structures, but that kind of obviates the point. 
 However, creating an entirely new set of accessors means that we can't 
leverage Clojure's standard library.

It's possible that I'm alone in my frustrations, and no Clojure wrapper is 
necessary.  But if this does solve a problem you have, I'd like to hear 
more about what it is, and how you think Bifurcan might help.  Please feel 
free to reply here, or to grab me at Clojure/West and talk about it there.

Thanks in advance,
Zach

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN and RFC] Bifurcan: impure functional data strucures

[Zach Tellman]

Benchmarks are available here, and the Clojure benchmarks make use of
transients wherever possible:
https://github.com/lacuna/bifurcan/blob/master/doc/benchmarks.md.

More generally, while transients are often used in practice to quickly
construct a read-only data structure, the more formal definition is that
they provide an O(1) mechanism for transforming between immutable and
mutable forms.  This isn't possible with purely mutable data structures
like Java's HashMap or Bifurcan's LinearMap.  So while wrapping these data
structures in the Clojure API would provide better performance for
construction and lookups, it wouldn't be quite the same thing as a
transient.



On Mon, Mar 27, 2017 at 10:05 AM Michael Gardner 
wrote:

>
> > On Mar 27, 2017, at 09:51, Zach Tellman  wrote:
> >
> > They also provide high-performance mutable variants of the data
> structure which share an API with their immutable cousins.
>
> How does their performance compare to Clojure's transients? Transients are
> slower than Java's native mutable collections, so if the mutable
> collections in this library deliver the same performance as the latter,
> they could act as a drop-in replacement for the former (given a compatible
> Clojure wrapper).
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/1m_I7IrDGb0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN and RFC] Bifurcan: impure functional data strucures

[Zach Tellman]

Both the 'List' and 'Map' data structures in Bifurcan use innovative
approaches that were published after Clojure's original release [1] [2].
In the case of the immutable map, you get faster iteration and the
structural invariants allow for some clever stuff w.r.t. equality checks
and set operations.  In the case of the immutable list/vector, you get fast
concatenation, the ability to add and remove from both ends of the
collection, and a `subvec` that doesn't hold onto the entire underlying
data structure.

All of this is MIT licensed, so please feel welcome to open a PR against
Clojure to change the core data structures using my code, but I'd rate the
chance of that being accepted as somewhere between "low" and
"nonexistent".  Also, it should be noted that Clojure's implementation is
much more battle-tested than my own at this point.  But if anyone wants to
tilt at that particular windmill, feel free to ask me any questions you may
have about the implementation.

Zach

[1] https://michael.steindorfer.name/publications/oopsla15.pdf
[2] https://infoscience.epfl.ch/record/169879/files/RMTrees.pdf

On Mon, Mar 27, 2017 at 12:30 PM Luke Burton  wrote:

>
> I'm not well versed enough in these data structures to know this without
> asking (apologies if it's really obvious to some people): is there
> opportunity to improve Clojure's built-in data structures with Bifurcan
> rather than trying to wrap Bifurcan's structures in Clojure?
>
> As an aside, I want to draw people's attention to the sweet little
> criterium + gnuplot setup you have there for generating benchmarking plots.
> Nice!
>
>
> On Mar 27, 2017, at 10:13 AM, Zach Tellman  wrote:
>
> Benchmarks are available here, and the Clojure benchmarks make use of
> transients wherever possible:
> https://github.com/lacuna/bifurcan/blob/master/doc/benchmarks.md.
>
> More generally, while transients are often used in practice to quickly
> construct a read-only data structure, the more formal definition is that
> they provide an O(1) mechanism for transforming between immutable and
> mutable forms.  This isn't possible with purely mutable data structures
> like Java's HashMap or Bifurcan's LinearMap.  So while wrapping these data
> structures in the Clojure API would provide better performance for
> construction and lookups, it wouldn't be quite the same thing as a
> transient.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/1m_I7IrDGb0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN and RFC] Bifurcan: impure functional data strucures

[Zach Tellman]

To be clear, my intention was always to wrap the implementations in the
appropriate Clojure interfaces, and I don't believe that will cause much,
if any, of a performance hit (inlining is magic).  However, there are some
real questions regarding how to expose non-standard equality semantics, and
whether transients should be represented using the immutable or mutable
collection variants.

For what it's worth, I have about 1/3 of an implementation of
Clojure-compatible versions of these data structures, I just wanted to mull
on the above questions a bit before going further.  I'm happy to discuss
them here in more depth if you have any questions or opinions.

Zach

On Tue, Apr 18, 2017 at 6:53 AM Dave Dixon  wrote:

> Stared at this a bit yesterday. Seems like if you want to leverage spec
> while using bifurcan, then the bifurcan types need to have the Clojure
> wrapper. The alternative appears to be re-implementing at least a large
> subset of collection-related spec code, which is a lot to bite off. Also
> tried updating some existing code to use bifurcan. Similar to spec, there
> are going to be cases which are less perf sensitive, where it would be nice
> to use code that is polymorphic for collections, and drop down to the fast
> interface in perf-sensitive parts.
>
>
> On Monday, April 17, 2017 at 1:52:39 PM UTC-7, Dave Dixon wrote:
>>
>> What is the issue with wrapping in Clojure interfaces? Added overhead of
>> function calls?
>>
>> I'm finding myself in the process of doing some of this, at least for
>> constructors. Also thinking of generating predicates/generators for use
>> with spec.
>>
>> On Monday, March 27, 2017 at 9:51:46 AM UTC-7, Zach Tellman wrote:
>>
>>> This is a slightly irregular announcement, because it's not for a
>>> Clojure library.  Rather, it's for a library written purely in Java:
>>> https://github.com/lacuna/bifurcan.
>>>
>>> This is a collection of mutable and immutable data structures, designed
>>> to address some of my personal frustrations with what's available in the
>>> Clojure and Java ecosystems.  Notably, they have pluggable equality
>>> semantics, so while they *can* use Clojure's expensive hash and equality
>>> checks, they don't *have* to.  They also provide high-performance mutable
>>> variants of the data structure which share an API with their immutable
>>> cousins.
>>>
>>> I'm posting it here to ask for people's thoughts on how, if at all, this
>>> should be exposed as a Clojure library.  It would be simple to simply wrap
>>> them in the Clojure interfaces and make them behave identically to
>>> Clojure's own data structures, but that kind of obviates the point.
>>> However, creating an entirely new set of accessors means that we can't
>>> leverage Clojure's standard library.
>>>
>>> It's possible that I'm alone in my frustrations, and no Clojure wrapper
>>> is necessary.  But if this does solve a problem you have, I'd like to hear
>>> more about what it is, and how you think Bifurcan might help.  Please feel
>>> free to reply here, or to grab me at Clojure/West and talk about it there.
>>>
>>> Thanks in advance,
>>> Zach
>>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/1m_I7IrDGb0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN and RFC] Bifurcan: impure functional data strucures

[Zach Tellman]

Sure, happy to elaborate.  Bifurcan offers potential performance wins a few
different ways:

* We can use standard Java equality semantics, bypassing all the overhead
of the hash calculations and enhanced numeric equality checks (this can
lead to moderate performance gains)
* We can use a mutable data structure as long as it never escapes a local
context (this can lead to significant performance gains)
* We can use the extra capabilities the data structures expose, like
concatenation, slicing, set operations, etc. (this is too dependent on the
use case to really quantify)

it would be easy to have a `map` and `map*` method that expose Clojure and
Java equality semantics, respectively, but that puts a big onus on the
developer to determine if the latter is safe for their use case.  I've been
bit by this when I've used j.u.c.ConcurrentHashMap before, so I expect
people will suffer similarly weird bugs.

However, I think there's a way to use the mutable data structures.
Technically, transient data structures allow arbitrary persistent data
structures to be batch updated, but in practice they tend to be empty, and
after they're populated they tend to be treated as read-only.

If we're convinced this is common enough, every empty transient data
structure could be mutable, and when we make it persistent we could wrap it
in a "virtual" collection [1] which allows updates without touching the
base collection.  This would allow for faster writes, faster reads, and
only marginally slower updates if those are required.

This is all predicated on a bunch of assumptions that are hard to validate,
but if this describes enough real-world use cases, it could lead to a big,
easy performance win.  It's even possible to automatically replace the base
Clojure collections with these alternatives using something like Sleight
[2].

Anyway, that's what I've been mulling over.  If anyone has opinions, I'm
happy to hear them.

Zach

[1]
https://github.com/lacuna/bifurcan/blob/master/src/io/lacuna/bifurcan/Maps.java#L103
[2] https://github.com/ztellman/sleight

On Thu, Apr 20, 2017 at 8:55 AM Dave Dixon  wrote:

> Sounds great. If you have time, I'd certainly like to hear your thoughts
> on the issues of equality semantics and transients, maybe I can ponder and
> make some suggestions based on my target use-case.
>
>
> On Tuesday, April 18, 2017 at 9:32:32 AM UTC-7, Zach Tellman wrote:
>
>> To be clear, my intention was always to wrap the implementations in the
>> appropriate Clojure interfaces, and I don't believe that will cause much,
>> if any, of a performance hit (inlining is magic).  However, there are some
>> real questions regarding how to expose non-standard equality semantics, and
>> whether transients should be represented using the immutable or mutable
>> collection variants.
>>
>> For what it's worth, I have about 1/3 of an implementation of
>> Clojure-compatible versions of these data structures, I just wanted to mull
>> on the above questions a bit before going further.  I'm happy to discuss
>> them here in more depth if you have any questions or opinions.
>>
>> Zach
>>
>> On Tue, Apr 18, 2017 at 6:53 AM Dave Dixon  wrote:
>>
> Stared at this a bit yesterday. Seems like if you want to leverage spec
>>> while using bifurcan, then the bifurcan types need to have the Clojure
>>> wrapper. The alternative appears to be re-implementing at least a large
>>> subset of collection-related spec code, which is a lot to bite off. Also
>>> tried updating some existing code to use bifurcan. Similar to spec, there
>>> are going to be cases which are less perf sensitive, where it would be nice
>>> to use code that is polymorphic for collections, and drop down to the fast
>>> interface in perf-sensitive parts.
>>>
>>>
>>> On Monday, April 17, 2017 at 1:52:39 PM UTC-7, Dave Dixon wrote:
>>>>
>>>> What is the issue with wrapping in Clojure interfaces? Added overhead
>>>> of function calls?
>>>>
>>>> I'm finding myself in the process of doing some of this, at least for
>>>> constructors. Also thinking of generating predicates/generators for use
>>>> with spec.
>>>>
>>>> On Monday, March 27, 2017 at 9:51:46 AM UTC-7, Zach Tellman wrote:
>>>>
>>>>> This is a slightly irregular announcement, because it's not for a
>>>>> Clojure library.  Rather, it's for a library written purely in Java:
>>>>> https://github.com/lacuna/bifurcan.
>>>>>
>>>>> This is a collection of mutable and immutable data structures,
>>>>> designed to address some of my 

Re: [ANN and RFC] Bifurcan: impure functional data strucures

[Zach Tellman]

Are you relying on the immutability of these structures, or are they
effectively always transient?
On Sun, Apr 23, 2017 at 11:02 AM Dave Dixon  wrote:

> FWIW, the use-case I have essentially involves Monte Carlo simulations. So
> we start with a state (non-empty map), and then make a series of
> modifications to it. Various statistics are held in hash-maps keyed by the
> state, so there's a lot of lookups and modifications in those maps.
>
> That said, I'm not sure if for this particular case I care too much using
> Clojure idioms vs. direct API access. The algorithms tend to be
> hand-tweaked for performance anyway. The big win for me in wrapping
> bifurcan would be the ability to use spec without having to write
> specialized specs, generators, etc.
>
>
> On Thursday, April 20, 2017 at 9:53:56 PM UTC-7, Zach Tellman wrote:
>
>> Sure, happy to elaborate.  Bifurcan offers potential performance wins a
>> few different ways:
>>
>> * We can use standard Java equality semantics, bypassing all the overhead
>> of the hash calculations and enhanced numeric equality checks (this can
>> lead to moderate performance gains)
>> * We can use a mutable data structure as long as it never escapes a local
>> context (this can lead to significant performance gains)
>> * We can use the extra capabilities the data structures expose, like
>> concatenation, slicing, set operations, etc. (this is too dependent on the
>> use case to really quantify)
>>
>
>> it would be easy to have a `map` and `map*` method that expose Clojure
>> and Java equality semantics, respectively, but that puts a big onus on the
>> developer to determine if the latter is safe for their use case.  I've been
>> bit by this when I've used j.u.c.ConcurrentHashMap before, so I expect
>> people will suffer similarly weird bugs.
>>
>> However, I think there's a way to use the mutable data structures.
>> Technically, transient data structures allow arbitrary persistent data
>> structures to be batch updated, but in practice they tend to be empty, and
>> after they're populated they tend to be treated as read-only.
>>
>> If we're convinced this is common enough, every empty transient data
>> structure could be mutable, and when we make it persistent we could wrap it
>> in a "virtual" collection [1] which allows updates without touching the
>> base collection.  This would allow for faster writes, faster reads, and
>> only marginally slower updates if those are required.
>>
>> This is all predicated on a bunch of assumptions that are hard to
>> validate, but if this describes enough real-world use cases, it could lead
>> to a big, easy performance win.  It's even possible to automatically
>> replace the base Clojure collections with these alternatives using
>> something like Sleight [2].
>>
>> Anyway, that's what I've been mulling over.  If anyone has opinions, I'm
>> happy to hear them.
>>
>> Zach
>>
>> [1]
>> https://github.com/lacuna/bifurcan/blob/master/src/io/lacuna/bifurcan/Maps.java#L103
>> [2] https://github.com/ztellman/sleight
>>
>> On Thu, Apr 20, 2017 at 8:55 AM Dave Dixon  wrote:
>>
>>> Sounds great. If you have time, I'd certainly like to hear your thoughts
>>> on the issues of equality semantics and transients, maybe I can ponder and
>>> make some suggestions based on my target use-case.
>>>
>>>
>>> On Tuesday, April 18, 2017 at 9:32:32 AM UTC-7, Zach Tellman wrote:
>>>
>>>> To be clear, my intention was always to wrap the implementations in the
>>>> appropriate Clojure interfaces, and I don't believe that will cause much,
>>>> if any, of a performance hit (inlining is magic).  However, there are some
>>>> real questions regarding how to expose non-standard equality semantics, and
>>>> whether transients should be represented using the immutable or mutable
>>>> collection variants.
>>>>
>>>> For what it's worth, I have about 1/3 of an implementation of
>>>> Clojure-compatible versions of these data structures, I just wanted to mull
>>>> on the above questions a bit before going further.  I'm happy to discuss
>>>> them here in more depth if you have any questions or opinions.
>>>>
>>>> Zach
>>>>
>>>> On Tue, Apr 18, 2017 at 6:53 AM Dave Dixon  wrote:
>>>>
>>> Stared at this a bit yesterday. Seems like if you want to leverage spec
>>>>> while using bifurcan, the

[ANN] A complete draft of "Elements of Clojure" is available

[Zach Tellman]

Almost exactly two years ago, I announced on this mailing list that I was 
working on an intermediate-level book on Clojure [1], and released the 
first chapter.  I've confined updates since then to a book-specific list, 
but I feel like it's fair to post here again to say that the fourth and 
final chapter has been released.  Details can be found on the book's 
mailing 
list: https://groups.google.com/forum/#!topic/elements-of-clojure/t_Uqc8F0Ch0.

If you haven't heard about the book before, please check out its 
website: http://elementsofclojure.com/

Zach


[1] https://groups.google.com/forum/#!msg/clojure/Nh_Z0XaxhTA/P-lPdf2NDQAJ

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] 'Elements of Clojure' is complete

[Zach Tellman]

I'm very happy to announce, only two and a half years after the release of 
the first chapter, that Elements of Clojure is completely finished.  
Further details can be found 
here: https://groups.google.com/forum/#!topic/elements-of-clojure/UUJjqU1rllU.

If you've never heard of the book before, please check out its website: 
http://elementsofclojure.com/

Zach

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Are you using Aleph in production? Tell us about it.

[Zach Tellman]

We're in the process of figuring out long-term plans for governance of 
Aleph [1], and it would like to understand who the current users are, and 
how they're using the library.  If you are using it, please respond to this 
issue: https://github.com/ztellman/aleph/issues/450.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] data.int-map 0.2.1

[Zach Tellman]

https://github.com/clojure/data.int-map

This is a small but important update.  Where 0.2.0 added the ability to use 
negative integers, it did *not* retain the ordered key invariant when 
negative numbers were used (negative numbers came after the positive 
numbers, because binary representations are a continual delight).  I've 
pushed an update such that the ordering is correct, and also pushed some of 
the int-set implementation into Java, yielding a 20-30% performance 
improvement.  This should be a drop-in replacement for anyone using earlier 
versions of the library.

Zach

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Manifold streams/deferreds raising errors downstream of a sink

[Zach Tellman]

Hi Andy,

To give you some context on this, `put!` will return true if the thing 
directly downstream accepts the value (in this case, the stream).  The fact 
that the callback in `consume` fails is between the stream and the 
callback, and is not propagated all the way back (though the exception will 
cancel the `consume`, which will cause the stream to close).  The approach 
described by Atamert is a reasonable one to get the desired result.

Zach

On Sunday, August 9, 2015 at 9:13:30 PM UTC-7, Andy Chambers wrote:
>
> Hey All,
>
> I'm trying to build a library that exposes manifold streams for data 
> producing applications. The intention
> would be for these apps to s/put! to a sink stream returned by the 
> producer function. However, I need to
> take the value that is put in and pipe it into some other function before 
> I know whether there is an error
> and if there is an error, I'd like to propagate it back to the caller.
>
> I understand that the value returned by put! is a deferred, and how to set 
> the error state of a deferred but I
> don't understand how I can arrange for the deferred to be manipulated 
> downstream of where it has been
> taken off the stream.
>
> (defn producer []
>   (let [in (s/stream)]
>
> (s/consume (fn [msg]
>  (d/future
>(prn "oops, going to error now")
>(throw (Exception. (str msg) in)
> 
> in))
>
> @(s/put! (producer) 42)
>
> As written, you get a success response even though an exception was 
> triggered by the function that consumes
> the "in" stream. Is it possible to re-write this so that the exception 
> bubbles up to where the put! is deref'd?
>
> Cheers,
> Andy
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What does Zach Tellman mean by "factored out for greater inlining joy"

[Zach Tellman]

The inlining part is explained very well by this blog 
post http://normanmaurer.me/blog/2014/05/15/Inline-all-the-Things/

As for why I left all the repetition in there, I tend to let code expand 
before getting annoyed and compacting it.  Sometimes there's a commit 
between those two events, sometimes there's not.  In this case, you get to 
see how the macro/abstraction sausage is made.

Happy to answer any other questions,
Zach

On Sunday, August 9, 2015 at 9:38:51 AM UTC-7, Lawrence Krubner wrote:
>
>
> Reid, thank you. I think you answer half the question. You make a good 
> point about giving the JVM a way to better optimize a hot path. I think you 
> are right about that. But, given the large amount of repetition between 
> "chain'-" and "chain-" I'm wondering why this wasn't done with a macro? 
>
>
>
> On Sunday, August 9, 2015 at 2:08:47 AM UTC-4, Reid McKenzie wrote:
>>
>> -BEGIN PGP SIGNED MESSAGE- 
>> Hash: SHA256 
>>
>> Lawrence, 
>>
>> This is just a theory, but in the interests of response time, the JVM 
>> uses a large number of heuristics to determine what optimizations will 
>> likely prove profitable. One of them is a budget for method size. I 
>> would guess that lifting this code out into a separate fn made the JVM 
>> see that it was optimizing a hot path between the main body and 
>> several small but tightly related methods thus giving itself more 
>> leeway to inline and optimize in ways that it would otherwise presume 
>> are more expensive and not pursue. 
>>
>> Reid 
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What does Zach Tellman mean by "factored out for greater inlining joy"

[Zach Tellman]

It's fluid, but the output that Norman uses in his post (and that motivated 
the changes to Aleph) were from the JVM explicitly saying "I would have 
inlined this, but it was too big".  That may not be true under other 
circumstances, but you're only helping by factoring out uncommon, large 
code branches.

On Tuesday, August 11, 2015 at 2:01:59 PM UTC-7, Colin Fleming wrote:
>
> That's a really interesting post, thanks for that. I actually cornered 
> Cliff Click at Curry On because I was interested in knowing how well the 
> JVM inlined var indirection. The short answer is "it's complicated". But if 
> the JVM does inline your method, then the var indirection shouldn't cost 
> you as long as the var content is stable.
>
> Tom Crayford's great talk at EuroClojure also pointed out that another 
> thing which affects inlining is the stack depth, which can mean you're 
> damned if you do and you're damned if you don't when trying to refactor to 
> help with inlining. However Cliff said that all these limits are pretty 
> fluid - it's not like a method over 325 bytecodes will never be inlined, if 
> it's identified as hot that limit goes way up, and a relatively cold method 
> can still be inlined even if it's small.
>
> I'd actually love to sit down and test this with Clojure sometime, but I 
> never seem to find time for it.
>
> On 11 August 2015 at 21:00, Zach Tellman > 
> wrote:
>
>> The inlining part is explained very well by this blog post 
>> http://normanmaurer.me/blog/2014/05/15/Inline-all-the-Things/
>>
>> As for why I left all the repetition in there, I tend to let code expand 
>> before getting annoyed and compacting it.  Sometimes there's a commit 
>> between those two events, sometimes there's not.  In this case, you get to 
>> see how the macro/abstraction sausage is made.
>>
>> Happy to answer any other questions,
>> Zach
>>
>>
>> On Sunday, August 9, 2015 at 9:38:51 AM UTC-7, Lawrence Krubner wrote:
>>>
>>>
>>> Reid, thank you. I think you answer half the question. You make a good 
>>> point about giving the JVM a way to better optimize a hot path. I think you 
>>> are right about that. But, given the large amount of repetition between 
>>> "chain'-" and "chain-" I'm wondering why this wasn't done with a macro? 
>>>
>>>
>>>
>>> On Sunday, August 9, 2015 at 2:08:47 AM UTC-4, Reid McKenzie wrote:
>>>>
>>>> -BEGIN PGP SIGNED MESSAGE- 
>>>> Hash: SHA256 
>>>>
>>>> Lawrence, 
>>>>
>>>> This is just a theory, but in the interests of response time, the JVM 
>>>> uses a large number of heuristics to determine what optimizations will 
>>>> likely prove profitable. One of them is a budget for method size. I 
>>>> would guess that lifting this code out into a separate fn made the JVM 
>>>> see that it was optimizing a hot path between the main body and 
>>>> several small but tightly related methods thus giving itself more 
>>>> leeway to inline and optimize in ways that it would otherwise presume 
>>>> are more expensive and not pursue. 
>>>>
>>>> Reid 
>>>>
>>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com 
>> 
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Manifold questions: Creating sinks/sources

[Zach Tellman]

Hi Atamert,

For future reference, posting these questions 
to https://groups.google.com/forum/#!forum/aleph-lib will ensure I'll see 
them sooner.  

The `source-only` method is just a way to make sure that the chance for 
confusion is minimized, it doesn't prevent the underlying object from being 
used normally.  As for your scenario, I'm not sure I completely understand. 
 In a standard stream, the flow of data is:

[ sink -> source ]

So if you `put!` something into the sink, you can `take!` it from the 
source.  If you use `source-only` and connect that to some other sink, then 
the flow of data will be

[ sink -> source ] -> sink

So the source is only connected to one sink, not two.  Maybe I'm 
misunderstanding you, though.

Zach

On Thursday, August 27, 2015 at 4:13:35 AM UTC-7, Atamert Ölçgen wrote:
>
> Hi,
>
> AFAIK the only way to create (just) a source (or sink) is:
>
> (def my-source (s/source-only (s/stream ...)))
>
> This results in creating a stream and then wrapping it with a SourceProxy. 
> We don't keep a reference to the stream and the SourceProxy doesn't allow 
> taking.
>
> But if I'm not missing something since SourceProxy keeps a reference of 
> the original stream, there is a sink nobody is using there.
>
> If I create a sink separately and connect my-source to it, now 
> my-source would be connected to two sinks.
>
> My questions are:
>
> 1. Is there another method for creating only sinks or sources?
> 2. Should the extra/unused source/sink I mentioned above cause concern?
>
> PS: I'm not talking about a scenario where we're creating thousands 
> streams and connecting them all like there's no tomorrow.
>
>
> -- 
> Kind Regards,
> Atamert Ölçgen
>
> ◻◼◻
> ◻◻◼
> ◼◼◼
>
> www.muhuk.com
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.