core.logic for constraint logic programming
I'm looking into rewriting Storm's resource scheduler using core.logic. I want to be able to say constraints like: 1. Topology A's slots should be = 10 and as close to 10 as possible (minimize the delta between assigned slots and 10) 2. All topologies should use less than 200 CPU's and less than 600 GB of memory 3. Topology B should run at most 2 workers on each host 4. Each worker for topology C should run at most one task for component X and one task for component Y 5. Should minimize the amount of reassignment to running topologies in order to satisfy constraints 6. Should only be allowed to reassign workers for an individual topology whose individual constraints are satisfied once every 10 minutes And so on. I have two questions: 1. How good is core.logic at culling the search space using arithmetic logic? For example, if it knows that x + y = 5, x=0, and y=0, it should never bother with areas of the search space where x or y are =5. 2. Can core.logic do things like search the space for which my evaulation criteria are minimized? Or is what we're trying to do a better fit for different techniques like linear programming? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: core.logic for constraint logic programming
Cool, thanks for the quick response. We'll be looking into this pretty soon. I ultimately want the logic engine itself being exposed to users so they can add their own company-specific constraints to resource scheduling – which will be totally badass. If you're interested in tracking this, I opened up an issue on Storm: https://github.com/nathanmarz/storm/issues/383 On Oct 24, 2:07 pm, David Nolen dnolen.li...@gmail.com wrote: On Wed, Oct 24, 2012 at 4:56 PM, nathanmarz nathan.m...@gmail.com wrote: I'm looking into rewriting Storm's resource scheduler using core.logic. I want to be able to say constraints like: 1. Topology A's slots should be = 10 and as close to 10 as possible (minimize the delta between assigned slots and 10) The minimization bit is not in core.logic yet. But basically anything in Mozart/OZ's Finite Domain Constraint Programming feature set is on the table for core.logic :) 2. All topologies should use less than 200 CPU's and less than 600 GB of memory 3. Topology B should run at most 2 workers on each host 4. Each worker for topology C should run at most one task for component X and one task for component Y 5. Should minimize the amount of reassignment to running topologies in order to satisfy constraints 6. Should only be allowed to reassign workers for an individual topology whose individual constraints are satisfied once every 10 minutes Yep definitely sounds like the kind of thing I'd like core.logic to be used for. And so on. I have two questions: 1. How good is core.logic at culling the search space using arithmetic logic? For example, if it knows that x + y = 5, x=0, and y=0, it should never bother with areas of the search space where x or y are =5. While there's always room to improve it already does this well. 2. Can core.logic do things like search the space for which my evaulation criteria are minimized? Or is what we're trying to do a better fit for different techniques like linear programming? It can, but I need to sit down and figure out how to make that work. If I recall the Mozart/OZ work is pretty clear about their approach and I think it can easily be adapted. I'll try to find some time and get back to you on that. David -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: core.logic for constraint logic programming
Wow, thanks for the great information everyone. David – I don't know how we'll make it pluggable, I was thinking users could provide functions that return a set of constraints. And there would probably be a cost function that users could override as well. On Oct 24, 3:26 pm, Jamie Brandon ja...@scattered-thoughts.net wrote: Ok, clearly I've not been keeping up, sorry :) On 24 October 2012 18:17, David Nolen dnolen.li...@gmail.com wrote: On Wed, Oct 24, 2012 at 6:07 PM, Jamie Brandon ja...@scattered-thoughts.net wrote: It sounds like something that would benefit from good constraint propagation. If I remember correctly, core.logic only support propagating equality/inequality constraints which can be pretty slow for exploring large domains. Something like gecode (http://www.gecode.org) might be a better fit if you want to use large integer domains. Where core.logic lvars can only be assigned or fresh, gecode explicitly represents the domain of each variable and can efficiently propagate constraints across them. In your example, x and y would both have domain [0,5]. If you added the constraint x =3, after propagation you would have x: [3,5] y:[0,2] Not true anymore - we have interval propagation and we actually leverage it unlike the Scheme cKanren implementation. There's still plenty of room for improvement. Gecode also has support for custom search strategies which would allow writing heuristics for minimisation. I'm not sure if gecode would ever be as fast as a linear programming solution but it's certainly more flexible. Yes this what I was talking about when I referred to the Mozart/Oz distribute facility. This is pretty simple to add to core.logic as far as I can tell as I've stated earlier. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Clojure + jarjar to avoid dependency conflicts
I've been playing around with the jarjar tool ( http://code.google.com/p/jarjar/ ) in order to package my jar to avoid dependency conflicts with other libs. It doesn't seem to work though with Clojure-created classfiles, even when using aot compilation. At runtime, when doing a require/use it still tries to load the old classname and not the new classname substituted by jarjar. Has anyone had success with jarjar/clojure? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
Luc, what you're saying sounds to me like this is the way it is so
deal with it. Can you give me some concrete code snippets showing why
it's better to box ints as Longs? Do you really think the following is
at all intuitive?
user= (class (Integer/parseInt 1))
java.lang.Long
user= (class (Integer/valueOf 1))
java.lang.Integer
-Nathan
On Oct 20, 10:27 pm, Luc Prefontaine lprefonta...@softaddicts.ca
wrote:
The weirdness here is that you seem to confuse the Java context and the
Clojure
context. They are not the same. Clojure has to satisfy to performance and
consistency
criterias. It's a language of it's own, not a Java offspring.
user= (class (Integer/parseInt 1))
java.lang.Long
user=
Integer/parseInt returns a primitive type. Not a boxed Integer object.
If used as a key in a map or anything else in Clojure, it will get promoted
to a long value as per the math
promotion rules (long/double representation). Obviously needed if it is to be
used later in a computation
otherwise it would break math operations consistency by allowing mixed
int/long operands.
If passed as an interop parameter it will retain it's int type.
user= (class (Integer/valueOf 1))
java.lang.Integer
Integer/valueOf returns an Integer object, not a primitive type.
It's an object, not a primitive type, Clojure will not change it.
If used as a key in a Clojure map or any Clojure data structure, it will
retain its object status.
Just cast your keys accordingly if you want Integer objects as keys.
In your short example, 1 as a key will not do it, it gets promoted to
primitive long.
You may not recall but in Java, int used not to be compatible with Integer
objects.
It's only since java 5 that you can assign an Integer object to a primitive
int.
That's the compiler tricking things to allow you to do that. In the JVM
there's still
not represented in the same way.
The above Integer member functions and their behavior have nothing to do with
Clojure.
They result from bad decisions made years ago when designing Java and the JVM
and you are blaming
Clojure for not handling them according to some patch implemented afterward
in the Java compiler.
You ran in the ClassCast exception by yourself. Clojure did not push you into
it.
When using Java interop you have to obey to Java rules and bend accordingly.
It's not Clojure that needs to bend, it's you to adapt to the interop
restrictions/conventions.
If Java expects an Integer object somewhere make sure you are providing it.
Luc P.
On Thu, 20 Oct 2011 21:11:41 -0700 (PDT)
nathanmarz nathan.m...@gmail.com wrote:
Now I'm confused. So when I do this:
(def i (Integer/parseInt 1))
Is i a primitive int, a primitive long, or a Long object?
I was under the impression that it was a primitive int based on
Justin's test code, but when I run (primitive-type i) in the REPL it
tells me :object.
If i is a primitive int, then the only change I'm proposing is that
if Clojure needs to box that value later on, that it box it as an
Integer instead of a Long. This change in behavior would not affect
primitive number performance since it's at a point when Clojure is
already boxing.
If i is a primitive long (which is what I thought was happening
originally), I propose that Clojure box the value as an Integer unless
you wrap the form in a (long ...) form. In the latter case Clojure
would do what it's doing currently so you can still get the
performance if you need it. The difference is that you're being
explicit about the type changing so there's no possible confusion in
that regard.
Finally, if i is a Long object, I propose that it instead be boxed
as an Integer object.
Note that I am not saying:
1. That Clojure always box primitives into an object form
2. That Clojure implement 32 bit arithmetic
In all these cases, you can still get maximum performance without
Clojure changing ints to longs. Please correct me if there's something
I'm missing here.
Stu's argument from above is that Clojure boxes ints to Longs instead
of Integer to avoid weirdness with hashcode/equality in collections.
This is a reasonable point, but consider this code example:
user= (def m1 {(Integer/valueOf 1) 2})
#'user/m1
user= (def m2 {(Integer/parseInt 1) 2})
#'user/m2
user= (map class (keys m1))
(java.lang.Integer)
user= (map class (keys m2))
(java.lang.Long)
Clojure doesn't prevent you from putting Integer objects in
collections. So there are cases where you still need to do type
coercion yourself. Given that Clojure can't hide this problem
completely from you, I think it's better that it treat int and
Integer consistently by boxing ints as Integers. Then there's no
weirdness like I ran into with getting ClassCastExceptions because the
type changed.
-Nathan
On Oct 20, 6:19 pm, David Nolen dnolen.li...@gmail.com wrote:
On Thu, Oct 20, 2011 at 4:11 PM, nathanmarz
Re: Clojure 1.3 treatment of integers and longs
Yea let's chat on IRC. I'll ping you when I see you online. -Nathan On Oct 21, 4:24 am, Stuart Halloway stuart.hallo...@gmail.com wrote: Luc, what you're saying sounds to me like this is the way it is so deal with it. Can you give me some concrete code snippets showing why it's better to box ints as Longs? Do you really think the following is at all intuitive? user= (class (Integer/parseInt 1)) java.lang.Long user= (class (Integer/valueOf 1)) java.lang.Integer -Nathan If you box Ints and Longs separately then collection keys stop working, as I said at the start of this thread. There is no such thing as intuitive. The behavior above follows from a clear rule that can be followed locally in all cases. What you propose, in addition to breaking collections, requires more contextual reasoning to understand code in a bunch of scenarios. Could we meet on IRC and discuss this later today? Feel like we are going in circles. Stu -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
Yes, I understand the behavior perfectly well. The primitive int gets
converted to a Long immediately, as this code demonstrates:
user= (class (Integer/parseInt 5))
java.lang.Long
The int isn't being boxed into a Long -- the type is being changed.
I'm aware that I can fix things by converting the type back to an
Integer manually, but that's not the point. Changing the types is
unusual behavior that leads to hard to track down bugs like I ran into
with the ClassCastException. My proposal still stands.
-Nathan
On Oct 19, 5:29 pm, Kevin Downey redc...@gmail.com wrote:
On Wed, Oct 19, 2011 at 5:14 PM, nathanmarz nathan.m...@gmail.com wrote:
Here's a code example illustrating the problem I'm having:
https://gist.github.com/1300034I've simplified it to the bare minimum
necessary to illustrate the problem.
Agree 100% that ints and longs are broken in Java. The hashcode/
equality stuff is messed up. Clojure can try really hard to hide this,
but it can't hide it completely since Java libraries can always return
you Integer objects. The additional complexity added from changing the
Existing Integer objects are unchanged. If the method call in your
example did return an Integer object then you would never notice
anything.
types on you isn't worth it IMO.
Here's my proposal for what I think would be better behavior:
1. Clojure boxes ints into Integers rather than convert them into
longs
clojure does not convert ints into longs, you are putting a primitive
into a collection, which requires boxing, clojure 1.3 boxes ints as
Longs. If you put (Integer. …) around your call you will be fine.
2. If you wrap the form in (long ...), Clojure skips the boxing and
does what it does now. Since this is done explicitly, there's no
confusion about types.
-Nathan
On Oct 19, 7:38 am, Stuart Halloway stuart.hallo...@gmail.com wrote:
Thanks. I read through that and it didn't quite answer my question. To
me it seems more logical that:
1. Clojure defaults to longs when you create a new number (with a
literal 0, 1, etc)
2. You can create ints by doing (int 0)
3. Clojure never changes the types of things you're using
I find Clojure's behavior of changing the types of primitive ints to
longs highly unusual, and it is causing a lot of pain in upgrading
Storm to 1.3.
Integers and longs are going to be painful no matter what because they are
broken in Java, e.g.
Object[] objects = new Object[] {-1, -1L};
System.out.println(objects[0].hashCode());
System.out.println(objects[1].hashCode());
Clojure avoids this pit by standardizing on longs, which leaves you with
the need to specifically request ints when you need them for interop. You
can use (int n) hints to select the correct interop method invocation, or
box an int if you want to hold on to a value guaranteed to be int.
Can you post a code example that shows a problem you are having?
Stu
Stuart Halloway
Clojure/corehttp://clojure.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
Thanks, that clarifies the behavior. Regardless though, at some point the int is becoming a Long which is a change of type. I'm arguing that Clojure should box primitive ints as Longs. Stu, I wouldn't say Clojure's behavior makes it just work. For example, if I obtained by number using Integer/valueOf, then Clojure will not change the Integer to a Long and will not prevent me from putting it in a collection. It's confusing that Integer/valueOf will stay an Integer in Clojure-land, and Integer/parseInt will become a Long in Clojure-land. The use case I'm interested in here is just this one point of Java interop: what Clojure does with primitive ints that it gets from a Java object (as far as I can tell, this is the only way to get a primitive int in Clojure 1.3). I think it's better that Clojure be consistent in its treatment of Integer objects and primitive ints by not changing the types on you. -Nathan On Oct 20, 10:19 am, Justin Kramer jkkra...@gmail.com wrote: Oops, I elided a little too much. Need a method with an Object signature to distinguish Integer from int: (definterface IPrimitiveTester (getType [^int x]) (getType [^long x]) ;; etc (getType [^Object x])) (deftype PrimitiveTester [] IPrimitiveTester (getType [this ^int x] :int) (getType [this ^long x] :long) ;; etc (getType [this ^Object x] :object)) (defmacro primitive-type [x] `(.getType (PrimitiveTester.) ~x)) (comment user= (primitive-type (Integer. 5)) :object user= (primitive-type (Integer/parseInt 5)) :int ) On Thursday, October 20, 2011 1:13:03 PM UTC-4, Justin Kramer wrote: Here's a quick proof using an interface-based primitive detector: (definterface IPrimitiveTester (getType [^int x]) (getType [^long x]) ;; other types elided ) (deftype PrimitiveTester [] IPrimitiveTester (getType [this ^int x] :int) (getType [this ^long x] :long) ;; other types elided ) (defmacro primitive-type [x] `(.getType (PrimitiveTester.) ~x)) (comment user= (primitive-type 5) ;unboxed :long user= (primitive-type (Integer/parseInt 5)) ;unboxed :int user= (class (Integer/parseInt 5)) ;boxed java.lang.Long ) Justin -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
Oops, I meant Clojure should box primitive ints as Integers. :-) On Oct 20, 12:15 pm, nathanmarz nathan.m...@gmail.com wrote: Thanks, that clarifies the behavior. Regardless though, at some point the int is becoming a Long which is a change of type. I'm arguing that Clojure should box primitive ints as Longs. Stu, I wouldn't say Clojure's behavior makes it just work. For example, if I obtained by number using Integer/valueOf, then Clojure will not change the Integer to a Long and will not prevent me from putting it in a collection. It's confusing that Integer/valueOf will stay an Integer in Clojure-land, and Integer/parseInt will become a Long in Clojure-land. The use case I'm interested in here is just this one point of Java interop: what Clojure does with primitive ints that it gets from a Java object (as far as I can tell, this is the only way to get a primitive int in Clojure 1.3). I think it's better that Clojure be consistent in its treatment of Integer objects and primitive ints by not changing the types on you. -Nathan On Oct 20, 10:19 am, Justin Kramer jkkra...@gmail.com wrote: Oops, I elided a little too much. Need a method with an Object signature to distinguish Integer from int: (definterface IPrimitiveTester (getType [^int x]) (getType [^long x]) ;; etc (getType [^Object x])) (deftype PrimitiveTester [] IPrimitiveTester (getType [this ^int x] :int) (getType [this ^long x] :long) ;; etc (getType [this ^Object x] :object)) (defmacro primitive-type [x] `(.getType (PrimitiveTester.) ~x)) (comment user= (primitive-type (Integer. 5)) :object user= (primitive-type (Integer/parseInt 5)) :int ) On Thursday, October 20, 2011 1:13:03 PM UTC-4, Justin Kramer wrote: Here's a quick proof using an interface-based primitive detector: (definterface IPrimitiveTester (getType [^int x]) (getType [^long x]) ;; other types elided ) (deftype PrimitiveTester [] IPrimitiveTester (getType [this ^int x] :int) (getType [this ^long x] :long) ;; other types elided ) (defmacro primitive-type [x] `(.getType (PrimitiveTester.) ~x)) (comment user= (primitive-type 5) ;unboxed :long user= (primitive-type (Integer/parseInt 5)) ;unboxed :int user= (class (Integer/parseInt 5)) ;boxed java.lang.Long ) Justin -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
But Clojure is already inconsistent. ints and Integers in interop are treated differently. The only way to make Clojure consistent is to either: 1. Box ints as Integers 2. Always convert Integers to Longs. I'm not sure on the feasibility of #2. I'm not trying to be obtuse, but I really don't see the benefit of boxing primitive ints as Longs given how Integer objects are treated. Right now, if you obtain an Integer object via interop and want it to be compatible with Clojure's regular numerics, you still have to manually convert that Integer object into a Long. What I'm proposing is that you treat primitive ints obtained via interop the exact same way, which avoids the weird type issues that I ran into. -Nathan On Oct 20, 12:26 pm, David Nolen dnolen.li...@gmail.com wrote: Such a change would be make Clojure's numeric design inconsistent. You keep saying that Clojure is changing the types - that's probably not the right way to look at it. It's a semantic change, Clojure now only has 64bit primitives - the same way that JavaScript only has doubles. Prior to the 1.3 change, the semantics gave you a free lunch around primitive ints in the interop scenario. Now you have be explicit just as you do with pretty much any kind of Java interop. David On Thu, Oct 20, 2011 at 3:16 PM, nathanmarz nathan.m...@gmail.com wrote: Oops, I meant Clojure should box primitive ints as Integers. :-) On Oct 20, 12:15 pm, nathanmarz nathan.m...@gmail.com wrote: Thanks, that clarifies the behavior. Regardless though, at some point the int is becoming a Long which is a change of type. I'm arguing that Clojure should box primitive ints as Longs. Stu, I wouldn't say Clojure's behavior makes it just work. For example, if I obtained by number using Integer/valueOf, then Clojure will not change the Integer to a Long and will not prevent me from putting it in a collection. It's confusing that Integer/valueOf will stay an Integer in Clojure-land, and Integer/parseInt will become a Long in Clojure-land. The use case I'm interested in here is just this one point of Java interop: what Clojure does with primitive ints that it gets from a Java object (as far as I can tell, this is the only way to get a primitive int in Clojure 1.3). I think it's better that Clojure be consistent in its treatment of Integer objects and primitive ints by not changing the types on you. -Nathan On Oct 20, 10:19 am, Justin Kramer jkkra...@gmail.com wrote: Oops, I elided a little too much. Need a method with an Object signature to distinguish Integer from int: (definterface IPrimitiveTester (getType [^int x]) (getType [^long x]) ;; etc (getType [^Object x])) (deftype PrimitiveTester [] IPrimitiveTester (getType [this ^int x] :int) (getType [this ^long x] :long) ;; etc (getType [this ^Object x] :object)) (defmacro primitive-type [x] `(.getType (PrimitiveTester.) ~x)) (comment user= (primitive-type (Integer. 5)) :object user= (primitive-type (Integer/parseInt 5)) :int ) On Thursday, October 20, 2011 1:13:03 PM UTC-4, Justin Kramer wrote: Here's a quick proof using an interface-based primitive detector: (definterface IPrimitiveTester (getType [^int x]) (getType [^long x]) ;; other types elided ) (deftype PrimitiveTester [] IPrimitiveTester (getType [this ^int x] :int) (getType [this ^long x] :long) ;; other types elided ) (defmacro primitive-type [x] `(.getType (PrimitiveTester.) ~x)) (comment user= (primitive-type 5) ;unboxed :long user= (primitive-type (Integer/parseInt 5)) ;unboxed :int user= (class (Integer/parseInt 5)) ;boxed java.lang.Long ) Justin -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
I'm not sure we're arguing about the same thing. I think that Clojure only supporting 64 bit primitive arithmetic is fine, and I'm not proposing that it support 32 bit primitive arithmetic. The sole point of contention is what Clojure does when it has to box a primitive int. I think this is orthogonal to primitive args/return, but correct me if I'm wrong. Right now, it boxes ints as a Long, which I think is changing the type. My proposal is that it box ints as Integer objects. Would changing the behavior in this way cause a fundamental performance limitation in Clojure? -Nathan On Oct 20, 12:50 pm, David Nolen dnolen.li...@gmail.com wrote: On Thu, Oct 20, 2011 at 3:45 PM, nathanmarz nathan.m...@gmail.com wrote: But Clojure is already inconsistent. ints and Integers in interop are treated differently. The only way to make Clojure consistent is to either: Clojure is consistent. Whether or not that makes *interop* easier or harder is orthogonal. You do know that Clojure now supports primitive args and return, right? How is what you proposing going to be reconciled with that? David -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
Now I'm confused. So when I do this:
(def i (Integer/parseInt 1))
Is i a primitive int, a primitive long, or a Long object?
I was under the impression that it was a primitive int based on
Justin's test code, but when I run (primitive-type i) in the REPL it
tells me :object.
If i is a primitive int, then the only change I'm proposing is that
if Clojure needs to box that value later on, that it box it as an
Integer instead of a Long. This change in behavior would not affect
primitive number performance since it's at a point when Clojure is
already boxing.
If i is a primitive long (which is what I thought was happening
originally), I propose that Clojure box the value as an Integer unless
you wrap the form in a (long ...) form. In the latter case Clojure
would do what it's doing currently so you can still get the
performance if you need it. The difference is that you're being
explicit about the type changing so there's no possible confusion in
that regard.
Finally, if i is a Long object, I propose that it instead be boxed
as an Integer object.
Note that I am not saying:
1. That Clojure always box primitives into an object form
2. That Clojure implement 32 bit arithmetic
In all these cases, you can still get maximum performance without
Clojure changing ints to longs. Please correct me if there's something
I'm missing here.
Stu's argument from above is that Clojure boxes ints to Longs instead
of Integer to avoid weirdness with hashcode/equality in collections.
This is a reasonable point, but consider this code example:
user= (def m1 {(Integer/valueOf 1) 2})
#'user/m1
user= (def m2 {(Integer/parseInt 1) 2})
#'user/m2
user= (map class (keys m1))
(java.lang.Integer)
user= (map class (keys m2))
(java.lang.Long)
Clojure doesn't prevent you from putting Integer objects in
collections. So there are cases where you still need to do type
coercion yourself. Given that Clojure can't hide this problem
completely from you, I think it's better that it treat int and
Integer consistently by boxing ints as Integers. Then there's no
weirdness like I ran into with getting ClassCastExceptions because the
type changed.
-Nathan
On Oct 20, 6:19 pm, David Nolen dnolen.li...@gmail.com wrote:
On Thu, Oct 20, 2011 at 4:11 PM, nathanmarz nathan.m...@gmail.com wrote:
I'm not sure we're arguing about the same thing. I think that Clojure
only supporting 64 bit primitive arithmetic is fine, and I'm not
proposing that it support 32 bit primitive arithmetic. The sole point
of contention is what Clojure does when it has to box a primitive int.
I think this is orthogonal to primitive args/return, but correct me if
I'm wrong.
If 32bit ints are allowed to exist then the various numeric operators must
handle them. If the numeric operators handle them then primitive arg and
return should probably be supported. But that would exponentially increase
the number of interfaces required for primitive arg return support.
David
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
Thanks Alan, that makes sense. This code example illustrates that
Clojure values can already be primitive ints:
user= (let [i 1] (primitive-type i))
:long
user= (let [i (Integer/parseInt 1)] (primitive-type i))
:int
So it appears that Clojure's behavior is case #2 from my last comment.
All I'm proposing is that when Clojure needs to box a primitive int,
that Clojure box it as an Integer rather than a Long. Then this code
example:
(let [m {:a (Integer/parseInt 1)}]
(map class (vals m)))
will behave the same as this one:
(let [m {:a (Integer/valueOf 1)}]
(map class (vals m)))
-Nathan
On Oct 20, 9:35 pm, Alan Malloy a...@malloys.org wrote:
It is a Long object. Vars hold objects, so it has to be boxed.
However, if instead of def'ing it you immediately called some java
method that will accept either a primitive int or a primitive long, my
understanding is that Clojure would arrange for the int version to be
called, because no boxing would happen.
On Oct 20, 9:11 pm, nathanmarz nathan.m...@gmail.com wrote:
Now I'm confused. So when I do this:
(def i (Integer/parseInt 1))
Is i a primitive int, a primitive long, or a Long object?
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
Here's a code example illustrating the problem I'm having:
https://gist.github.com/1300034 I've simplified it to the bare minimum
necessary to illustrate the problem.
Agree 100% that ints and longs are broken in Java. The hashcode/
equality stuff is messed up. Clojure can try really hard to hide this,
but it can't hide it completely since Java libraries can always return
you Integer objects. The additional complexity added from changing the
types on you isn't worth it IMO.
Here's my proposal for what I think would be better behavior:
1. Clojure boxes ints into Integers rather than convert them into
longs
2. If you wrap the form in (long ...), Clojure skips the boxing and
does what it does now. Since this is done explicitly, there's no
confusion about types.
-Nathan
On Oct 19, 7:38 am, Stuart Halloway stuart.hallo...@gmail.com wrote:
Thanks. I read through that and it didn't quite answer my question. To
me it seems more logical that:
1. Clojure defaults to longs when you create a new number (with a
literal 0, 1, etc)
2. You can create ints by doing (int 0)
3. Clojure never changes the types of things you're using
I find Clojure's behavior of changing the types of primitive ints to
longs highly unusual, and it is causing a lot of pain in upgrading
Storm to 1.3.
Integers and longs are going to be painful no matter what because they are
broken in Java, e.g.
Object[] objects = new Object[] {-1, -1L};
System.out.println(objects[0].hashCode());
System.out.println(objects[1].hashCode());
Clojure avoids this pit by standardizing on longs, which leaves you with the
need to specifically request ints when you need them for interop. You can use
(int n) hints to select the correct interop method invocation, or box an int
if you want to hold on to a value guaranteed to be int.
Can you post a code example that shows a problem you are having?
Stu
Stuart Halloway
Clojure/corehttp://clojure.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Clojure 1.3 treatment of integers and longs
Hey all, I recently started upgrading Storm to Clojure 1.3, and I ran into various issues due to Clojure's treatment of integers and longs. In particular, I have a situation like the following: 1. A Java object returns me an int. Let's call this value v. 2. I put v into a map, and pass that map into a Java object 3. I get ClassCastExceptions when that Java object tries to read that Integer and instead gets a Long back The error arises due to Clojure's auto-coercion of primitive ints to longs. Auto-coercing ints to longs is prone to errors like I ran into, especially when interoperating with Java code. It becomes especially confusing when considering that Integer objects do not get coerced to Long objects. Also, if Clojure is trying to treat everything as longs, I don't understand why there's an unchecked-divide-int function and not an unchecked-divide-long function. What's the rationale behind all this? Why not support both ints and longs? I'm sure this has been discussed before, so feel free to point me to earlier discussions on this. -Nathan -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure 1.3 treatment of integers and longs
Thanks. I read through that and it didn't quite answer my question. To me it seems more logical that: 1. Clojure defaults to longs when you create a new number (with a literal 0, 1, etc) 2. You can create ints by doing (int 0) 3. Clojure never changes the types of things you're using I find Clojure's behavior of changing the types of primitive ints to longs highly unusual, and it is causing a lot of pain in upgrading Storm to 1.3. I don't mean to imply that the design choices made were wrong; I know that Rich et al put a lot of thought and work into these changes. But I would like to understand why changing the types of ints to longs is necessary instead of supporting primitive ints as well. -Nathan On Oct 18, 2:25 pm, David Nolen dnolen.li...@gmail.com wrote: 233 messages long thread from June 2010,http://groups.google.com/group/clojure/browse_thread/thread/c8c850595... David On Tue, Oct 18, 2011 at 5:00 PM, nathanmarz nathan.m...@gmail.com wrote: Hey all, I recently started upgrading Storm to Clojure 1.3, and I ran into various issues due to Clojure's treatment of integers and longs. In particular, I have a situation like the following: 1. A Java object returns me an int. Let's call this value v. 2. I put v into a map, and pass that map into a Java object 3. I get ClassCastExceptions when that Java object tries to read that Integer and instead gets a Long back The error arises due to Clojure's auto-coercion of primitive ints to longs. Auto-coercing ints to longs is prone to errors like I ran into, especially when interoperating with Java code. It becomes especially confusing when considering that Integer objects do not get coerced to Long objects. Also, if Clojure is trying to treat everything as longs, I don't understand why there's an unchecked-divide-int function and not an unchecked-divide-long function. What's the rationale behind all this? Why not support both ints and longs? I'm sure this has been discussed before, so feel free to point me to earlier discussions on this. -Nathan -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Clojure DSL for Storm
I've fleshed out and documented the Clojure DSL for Storm. There were quite a few people interested in this, and I figured the Clojure community at large would want to know about it. Here are the docs: https://github.com/nathanmarz/storm/wiki/Clojure-DSL And here is an example that uses it: https://github.com/nathanmarz/storm-starter/blob/master/src/clj/storm/starter/clj/word_count.clj The DSL is quite nice. If you don't know what Storm is, the DSL won't make a lot of sense, so you should read through the introductory documentation to learn the concepts of Storm and the abstractions it exposes. -Nathan -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Storm
Yea, I have it on my todo list to document this (which seems to be getting longer by the minute). Hopefully I'll do that within the next few weeks. In the meantime, I pasted some example code showing usage of the Clojure DSL, if that helps at all in the meantime: https://gist.github.com/1228302 -Nathan On Sep 20, 10:41 pm, Baishampayan Ghose b.gh...@gmail.com wrote: Hi Nathan, On Tue, Sep 20, 2011 at 8:35 PM, nathanmarz nathan.m...@gmail.com wrote: Yesterday I open-sourced Storm at Strange Loop. Storm is a distributed and fault-tolerant realtime computation system hosted at https://github.com/nathanmarz/storm I would like to congratulate you on the release. Storm seems to be very interesting and apparently it has certain advantages that no other existing system seem to offer. ones) on top of Storm. That said, Storm has a Clojure DSL for programming topologies which is what I personally use for developing topologies. It would be great if the Clojure DSL is documented on the wiki (or did I miss it?). I am really keen on taking it for a test drive and I am sure others in the Clojure community would like to do so as well. Regards, BG -- Baishampayan Ghose b.ghose at gmail.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Storm
Yesterday I open-sourced Storm at Strange Loop. Storm is a distributed and fault-tolerant realtime computation system hosted at https://github.com/nathanmarz/storm Just want to preempt a few questions that I know people are wondering about the project: Is Storm mostly written in Java? If you look at the languages graph on Github, it says that Storm is 64% Java. However, this is inaccurate because those numbers include the Java code generated by the Thrift compiler. If you exclude the generated code, you'll find that Storm is over 50% Clojure in terms of line count. In terms of functionality though, Storm is around 98% Clojure. The Java code I wrote is mostly interfaces and small classes that a user of Storm would encounter in the public API (Java is, ahem, verbose). Why isn't Storm written completely in Clojure? I want Storm to be as accessible to as wide an audience as possible. A user's language preference or constraints shouldn't prevent them from being able to use Storm to solve their realtime computation problems. This is why I chose to define Storm's main interfaces in Java, and this is also why Storm supports using any language (including non-JVM ones) on top of Storm. That said, Storm has a Clojure DSL for programming topologies which is what I personally use for developing topologies. Clojure was a magnificent language to use to build Storm. Storm is a complex, intricate system, and Clojure helped a great deal in managing the complexity of the implementation. If you have any questions, I'd be happy to answer them. -Nathan -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
ANN: ElephantDB, a distributed database written in Clojure
We (BackType) recently released our database ElephantDB as open source. I thought that the Clojure community at large would find this project interesting as it's written in Clojure. ElephantDB is a specialized database for exporting key/value data from Hadoop and serving it in a read-only fashion. It has seemless integration with Cascalog through the elephantdb-cascalog project. ElephantDB makes use of Hadoop, Thrift, and BerkeleyDB Java Edition. Writing this database in Clojure was a great experience. The place where Clojure really shined was for writing the unit tests. Clojure's dynamic capabilities made it really easy to do things like mock out remote servers or mock out pieces that weren't deterministic so that they could be tested. There were a few key macros I wrote for testing purposes that made it a lot easier to write and understand the tests. If you'd like to know more about the project, check out our introductory blog post and the GitHub repos. http://tech.backtype.com/introducing-elephantdb-a-distributed-database https://github.com/nathanmarz/elephantdb https://github.com/nathanmarz/elephantdb-cascalog -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: ANN: ElephantDB, a distributed database written in Clojure
Clojure's not even close to being a bottleneck in this database. The performance is limited by the underlying storage engine which is currently Berkeley DB Java Edition. On Feb 24, 4:21 pm, rogerdpack rogerpack2...@gmail.com wrote: How was clojure speed-wise for you? Was it blazingly fast? Just good enough? -r -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Generating type hints dynamically
I'm having problems creating functions with the type hints generated dynamically. As a contrived example (my actual use case is somewhat complicated), let's say I want to make a macro that takes in as input a class symbol and returns a type hinted function that calls a method on the argument: (defmacro hinted-fn [class-sym] `(fn [~(symbol (str ^ class-sym)) arg#] (.get_val arg#))) This doesn't work, as it ends up creating a function of two arguments. Is there any way to make this work? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Generating type hints dynamically
That worked great, thanks. On Sep 28, 5:42 pm, ataggart alex.tagg...@gmail.com wrote: Type hints are just metadata. Instead of emitting the type-hint from the macro, you set the :tag metadata on the relevant symbol. On Sep 28, 5:20 pm, nathanmarz nathan.m...@gmail.com wrote: I'm having problems creating functions with the type hints generated dynamically. As a contrived example (my actual use case is somewhat complicated), let's say I want to make a macro that takes in as input a class symbol and returns a type hinted function that calls a method on the argument: (defmacro hinted-fn [class-sym] `(fn [~(symbol (str ^ class-sym)) arg#] (.get_val arg#))) This doesn't work, as it ends up creating a function of two arguments. Is there any way to make this work? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Clojure-based query language for Hadoop
I've recently released Cascalog, a Clojure DSL for querying Hadoop, as open-source. Here's the canonical word count query in Cascalog: (?- (stdout) [?word ?count] (sentence ?s) (split ?s : ?word) (c/ count ?count)) Introductory blog post: http://nathanmarz.com/blog/introducing-cascalog/ Project page: http://github.com/nathanmarz/cascalog -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
