ANN: Exoref, Clojure Redis-based reference types.
Regards everyone, to celebrate my first year with Clojure, I'm releasing my first Clojure library, with the hope it will be useful for others. Github: https://github.com/lantiga/exoref Clojars: https://clojars.org/exoref Exoref aims at providing a Redis-based implementation of Clojure reference types, to allow scaling up Clojure applications on multiple machines (or workers on a PaaS) or writing distributed applications featuring multiple components. This first release targets Clojure Atoms, but more reference types are feasible by leveraging on Redis transactions. Comments and suggestions are more than welcome. Cheers! Luca Antiga (@lantiga) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
I think you should do (binding [*print-dup* true] (pr-str value).. instead of just (pr-str value) in the encode-value function. (line 20 in redis_memo.clj) On Jan 4, 2:55 pm, Steve Purcell st...@sanityinc.com wrote: Read the code I posted in this thread and put up on github after you expressed interest. That's part of what it does, using the reader/printer representation. Alternatives would include standard Java binary serialisation or 3rd party libraries (Hessian/Burlap?). -Steve On 4 Jan 2010, at 12:15, Gabi wrote: What if I wanted to use Redis just persist binary (serialized) clojure objects ? What's the easiest (and fastest) way to serialize/de-serialize vectors or lists in Clojure ? (so the can stored as blobs in Redis) On Jan 4, 12:59 pm, Gabi bugspy...@gmail.com wrote: Maybe, though I would avoid distributed transactions as much as possible. They are complex and slow creatures. On Jan 4, 12:51 pm, Shantanu Kumar kumar.shant...@gmail.com wrote: On Jan 2, 5:12 am, Gabi bugspy...@gmail.com wrote: I am interested in the idea: Completely stateless set of Clojure nodes (on many machines), operating on a central state stored in some datastore. If transactions could be managed somehow, I think it would be very compelling model for many applications. Do you mean distributed transactions? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Indeed, thanks - I realized that earlier today myself! -Steve On 5 Jan 2010, at 09:24, Gabi wrote: I think you should do (binding [*print-dup* true] (pr-str value).. instead of just (pr-str value) in the encode-value function. (line 20 in redis_memo.clj) On Jan 4, 2:55 pm, Steve Purcell st...@sanityinc.com wrote: Read the code I posted in this thread and put up on github after you expressed interest. That's part of what it does, using the reader/printer representation. Alternatives would include standard Java binary serialisation or 3rd party libraries (Hessian/Burlap?). -Steve On 4 Jan 2010, at 12:15, Gabi wrote: What if I wanted to use Redis just persist binary (serialized) clojure objects ? What's the easiest (and fastest) way to serialize/de-serialize vectors or lists in Clojure ? (so the can stored as blobs in Redis) On Jan 4, 12:59 pm, Gabi bugspy...@gmail.com wrote: Maybe, though I would avoid distributed transactions as much as possible. They are complex and slow creatures. On Jan 4, 12:51 pm, Shantanu Kumar kumar.shant...@gmail.com wrote: On Jan 2, 5:12 am, Gabi bugspy...@gmail.com wrote: I am interested in the idea: Completely stateless set of Clojure nodes (on many machines), operating on a central state stored in some datastore. If transactions could be managed somehow, I think it would be very compelling model for many applications. Do you mean distributed transactions? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Or rather, more importantly, I realised I should bind *print-level* to nil... As for *print-dup*, I guess there's a danger that stored values might become unreadable if any implementation structure classes were to get renamed in a future clojure release. -Steve On 5 Jan 2010, at 09:24, Gabi wrote: I think you should do (binding [*print-dup* true] (pr-str value).. instead of just (pr-str value) in the encode-value function. (line 20 in redis_memo.clj) On Jan 4, 2:55 pm, Steve Purcell st...@sanityinc.com wrote: Read the code I posted in this thread and put up on github after you expressed interest. That's part of what it does, using the reader/printer representation. Alternatives would include standard Java binary serialisation or 3rd party libraries (Hessian/Burlap?). -Steve On 4 Jan 2010, at 12:15, Gabi wrote: What if I wanted to use Redis just persist binary (serialized) clojure objects ? What's the easiest (and fastest) way to serialize/de-serialize vectors or lists in Clojure ? (so the can stored as blobs in Redis) On Jan 4, 12:59 pm, Gabi bugspy...@gmail.com wrote: Maybe, though I would avoid distributed transactions as much as possible. They are complex and slow creatures. On Jan 4, 12:51 pm, Shantanu Kumar kumar.shant...@gmail.com wrote: On Jan 2, 5:12 am, Gabi bugspy...@gmail.com wrote: I am interested in the idea: Completely stateless set of Clojure nodes (on many machines), operating on a central state stored in some datastore. If transactions could be managed somehow, I think it would be very compelling model for many applications. Do you mean distributed transactions? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
How about congomongo (http://github.com/somnium/congomongo) ? Have anybody used it ? Seems good choice for storing state in central location.. On Jan 4, 2:40 am, Tom Hicks hickstoh...@gmail.com wrote: Have you looked at Neo4J? I have no experience with it but someone in the forum just announced a Clojure wrapper for it: http://groups.google.com/group/clojure/browse_thread/thread/9628c6227... cheers, -t On Jan 1, 2:07 pm, Julian Morrison julian.morri...@gmail.com wrote: I've just recently been poking around these NoSQLs investigating their features, so... Redis has limited data structures - flat un-nested lists and sets, and plain strings. It doesn't have sets exactly - just keys and values. Nothing nested at all, unless you serialize to strings. No indexes, although you can hack up your own. To be honest, Redis isn't that impressive versus what's in Clojure already. It's an in-memory DB (so it's not much different from ref +dosync) and it intermittently spits a snapshot to disk. If you can live with an in-process DB, you could copy (and exceed) its features including snapshot saving in a page of pure Clojure code, and beat it on speed too. Contrast MongoDB: slower because it bothers to save things, but still around twice as fast as MySQL and much faster than CouchDB (cite: the benchmarks page). Arbitrarily nested collections, indexes, atomic updates (in place operations like inc and append, or atomic compare- and-set), JSON syntax, typed data, replication (built in) and sharding (via a broker process). (MongoDB downsides: it grows files in a very greedy way to try and minimize data fragmentation, and it needs a 64bit machine to store more than about 2Gb.) On Dec 30 2009, 11:52 am, Gabi bugspy...@gmail.com wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
On Jan 2, 5:12 am, Gabi bugspy...@gmail.com wrote: I am interested in the idea: Completely stateless set of Clojure nodes (on many machines), operating on a central state stored in some datastore. If transactions could be managed somehow, I think it would be very compelling model for many applications. Do you mean distributed transactions? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Maybe, though I would avoid distributed transactions as much as possible. They are complex and slow creatures. On Jan 4, 12:51 pm, Shantanu Kumar kumar.shant...@gmail.com wrote: On Jan 2, 5:12 am, Gabi bugspy...@gmail.com wrote: I am interested in the idea: Completely stateless set of Clojure nodes (on many machines), operating on a central state stored in some datastore. If transactions could be managed somehow, I think it would be very compelling model for many applications. Do you mean distributed transactions? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
What if I wanted to use Redis just persist binary (serialized) clojure objects ? What's the easiest (and fastest) way to serialize/de-serialize vectors or lists in Clojure ? (so the can stored as blobs in Redis) On Jan 4, 12:59 pm, Gabi bugspy...@gmail.com wrote: Maybe, though I would avoid distributed transactions as much as possible. They are complex and slow creatures. On Jan 4, 12:51 pm, Shantanu Kumar kumar.shant...@gmail.com wrote: On Jan 2, 5:12 am, Gabi bugspy...@gmail.com wrote: I am interested in the idea: Completely stateless set of Clojure nodes (on many machines), operating on a central state stored in some datastore. If transactions could be managed somehow, I think it would be very compelling model for many applications. Do you mean distributed transactions? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Read the code I posted in this thread and put up on github after you expressed interest. That's part of what it does, using the reader/printer representation. Alternatives would include standard Java binary serialisation or 3rd party libraries (Hessian/Burlap?). -Steve On 4 Jan 2010, at 12:15, Gabi wrote: What if I wanted to use Redis just persist binary (serialized) clojure objects ? What's the easiest (and fastest) way to serialize/de-serialize vectors or lists in Clojure ? (so the can stored as blobs in Redis) On Jan 4, 12:59 pm, Gabi bugspy...@gmail.com wrote: Maybe, though I would avoid distributed transactions as much as possible. They are complex and slow creatures. On Jan 4, 12:51 pm, Shantanu Kumar kumar.shant...@gmail.com wrote: On Jan 2, 5:12 am, Gabi bugspy...@gmail.com wrote: I am interested in the idea: Completely stateless set of Clojure nodes (on many machines), operating on a central state stored in some datastore. If transactions could be managed somehow, I think it would be very compelling model for many applications. Do you mean distributed transactions? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
There's actually been some activity from different people regarding Clojure + Neo4j, see: http://wiki.neo4j.org/content/Clojure /anders On Jan 4, 1:40 am, Tom Hicks hickstoh...@gmail.com wrote: Have you looked at Neo4J? I have no experience with it but someone in the forum just announced a Clojure wrapper for it: http://groups.google.com/group/clojure/browse_thread/thread/9628c6227... cheers, -t On Jan 1, 2:07 pm, Julian Morrison julian.morri...@gmail.com wrote: I've just recently been poking around these NoSQLs investigating their features, so... Redis has limited data structures - flat un-nested lists and sets, and plain strings. It doesn't have sets exactly - just keys and values. Nothing nested at all, unless you serialize to strings. No indexes, although you can hack up your own. To be honest, Redis isn't that impressive versus what's in Clojure already. It's an in-memory DB (so it's not much different from ref +dosync) and it intermittently spits a snapshot to disk. If you can live with an in-process DB, you could copy (and exceed) its features including snapshot saving in a page of pure Clojure code, and beat it on speed too. Contrast MongoDB: slower because it bothers to save things, but still around twice as fast as MySQL and much faster than CouchDB (cite: the benchmarks page). Arbitrarily nested collections, indexes, atomic updates (in place operations like inc and append, or atomic compare- and-set), JSON syntax, typed data, replication (built in) and sharding (via a broker process). (MongoDB downsides: it grows files in a very greedy way to try and minimize data fragmentation, and it needs a 64bit machine to store more than about 2Gb.) On Dec 30 2009, 11:52 am, Gabi bugspy...@gmail.com wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Have you looked at Neo4J? I have no experience with it but someone in the forum just announced a Clojure wrapper for it: http://groups.google.com/group/clojure/browse_thread/thread/9628c622784ff45a# cheers, -t On Jan 1, 2:07 pm, Julian Morrison julian.morri...@gmail.com wrote: I've just recently been poking around these NoSQLs investigating their features, so... Redis has limited data structures - flat un-nested lists and sets, and plain strings. It doesn't have sets exactly - just keys and values. Nothing nested at all, unless you serialize to strings. No indexes, although you can hack up your own. To be honest, Redis isn't that impressive versus what's in Clojure already. It's an in-memory DB (so it's not much different from ref +dosync) and it intermittently spits a snapshot to disk. If you can live with an in-process DB, you could copy (and exceed) its features including snapshot saving in a page of pure Clojure code, and beat it on speed too. Contrast MongoDB: slower because it bothers to save things, but still around twice as fast as MySQL and much faster than CouchDB (cite: the benchmarks page). Arbitrarily nested collections, indexes, atomic updates (in place operations like inc and append, or atomic compare- and-set), JSON syntax, typed data, replication (built in) and sharding (via a broker process). (MongoDB downsides: it grows files in a very greedy way to try and minimize data fragmentation, and it needs a 64bit machine to store more than about 2Gb.) On Dec 30 2009, 11:52 am, Gabi bugspy...@gmail.com wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
There you go: http://github.com/purcell/redis-memo I doubt the memoize functions provided therein will be directly useful to you, but you may find a few lines of helpful code there. Best of luck with your experiment. -Steve On 31 Dec 2009, at 16:20, Gabi wrote: Yes. I think it is of much interest. What if I stored a shared data structure in redis (only because its the fastest), using your memoize variant, and process (maybe even updated it) it in parallel from different Clojure nodes. Some kind of primitive map/reduce mechanism I think. On Dec 31, 12:29 pm, Steve Purcell st...@sanityinc.com wrote: Not sure if it's any help, but here's a variant of memoize I wrote, which stores arbitrary readable/printable objects to redis: http://gist.github.com/266689 (If there's any interest, I'll wrap it up in a github project and push it to clojars.) Redis isn't a hierarchical store, so its array/set operations would only benefit the most shallow of data structures. -Steve On 30 Dec 2009, at 11:52, Gabi wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Jackrabbit is heavy. It might be powerful but I am sure it is much slower than Redis or MongoDB. On Dec 31 2009, 6:59 pm, jem jere.mcdev...@gmail.com wrote: Something else to look at might be the Apache Jackrabbit project athttp://jackrabbit.apache.org/. I've been looking at tools along these lines as well, and recently looked at Redis for the same reasons. Right now, though, I'm focusing my attention on Jackrabbit which is an implementation of the JSR170 repository specifications. It supports hierarchical data and lists. I was looking for some way to save dynamic objects (basically runtime defined maps) and this looks like it might work. JSR170 defines a repository as: A content repository consists of one or more workspaces, each of which contains a tree of items. An item is either a node or a property. Each node may have zero or more child nodes and zero or more child properties. There is a single root node per workspace, which has no parent. All other nodes have one parent. Properties have one parent (a node) and cannot have children; they are the leaves of the tree. All of the actual content in the repository is stored within the values of the properties. The available property types cover what appear to be all the bases, including references to other nodes to prevent cycles when serializing. It can support annotated Java objects or use the Node building capability directly and allows querying similar to XPath. Since it is a Java project, using it from Clojure should be trivial to use. On Dec 31, 5:29 am, Steve Purcell st...@sanityinc.com wrote: Not sure if it's any help, but here's a variant of memoize I wrote, which stores arbitrary readable/printable objects to redis: http://gist.github.com/266689 (If there's any interest, I'll wrap it up in a github project and push it to clojars.) Redis isn't a hierarchical store, so its array/set operations would only benefit the most shallow of data structures. -Steve On 30 Dec 2009, at 11:52, Gabi wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Oops, typo - I meant, doesn't have hashes. On Jan 1, 9:07 pm, Julian Morrison julian.morri...@gmail.com wrote: It doesn't have sets exactly - just keys and values. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
I am interested in the idea: Completely stateless set of Clojure nodes (on many machines), operating on a central state stored in some datastore. If transactions could be managed somehow, I think it would be very compelling model for many applications. On Jan 1, 11:07 pm, Julian Morrison julian.morri...@gmail.com wrote: I've just recently been poking around these NoSQLs investigating their features, so... Redis has limited data structures - flat un-nested lists and sets, and plain strings. It doesn't have sets exactly - just keys and values. Nothing nested at all, unless you serialize to strings. No indexes, although you can hack up your own. To be honest, Redis isn't that impressive versus what's in Clojure already. It's an in-memory DB (so it's not much different from ref +dosync) and it intermittently spits a snapshot to disk. If you can live with an in-process DB, you could copy (and exceed) its features including snapshot saving in a page of pure Clojure code, and beat it on speed too. Contrast MongoDB: slower because it bothers to save things, but still around twice as fast as MySQL and much faster than CouchDB (cite: the benchmarks page). Arbitrarily nested collections, indexes, atomic updates (in place operations like inc and append, or atomic compare- and-set), JSON syntax, typed data, replication (built in) and sharding (via a broker process). (MongoDB downsides: it grows files in a very greedy way to try and minimize data fragmentation, and it needs a 64bit machine to store more than about 2Gb.) On Dec 30 2009, 11:52 am, Gabi bugspy...@gmail.com wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Not sure if it's any help, but here's a variant of memoize I wrote, which stores arbitrary readable/printable objects to redis: http://gist.github.com/266689 (If there's any interest, I'll wrap it up in a github project and push it to clojars.) Redis isn't a hierarchical store, so its array/set operations would only benefit the most shallow of data structures. -Steve On 30 Dec 2009, at 11:52, Gabi wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
On Dec 30, 10:40 am, Robert Campbell rrc...@gmail.com wrote: I think anything which lowers the impedance mismatch between Clojure data structures and a persistent store is worth investigating. I'd love to find an ACID, transactional store which accepts native structures. Have you looked at Cupboard (http://github.com/gcv/cupboard)? It tries to make it easy to store native Clojure data structures, and it is as ACID and transactional as the underlying Berkeley DB JE implementation. If you have looked at it, I'd appreciate some feedback as to where it fell short of your expectations. :) I plan to start adding distributed storage support in the next few months. The latest JE release has master-slave replication, which I think will work well for Cupboard. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Yes. I think it is of much interest. What if I stored a shared data structure in redis (only because its the fastest), using your memoize variant, and process (maybe even updated it) it in parallel from different Clojure nodes. Some kind of primitive map/reduce mechanism I think. On Dec 31, 12:29 pm, Steve Purcell st...@sanityinc.com wrote: Not sure if it's any help, but here's a variant of memoize I wrote, which stores arbitrary readable/printable objects to redis: http://gist.github.com/266689 (If there's any interest, I'll wrap it up in a github project and push it to clojars.) Redis isn't a hierarchical store, so its array/set operations would only benefit the most shallow of data structures. -Steve On 30 Dec 2009, at 11:52, Gabi wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
Something else to look at might be the Apache Jackrabbit project at http://jackrabbit.apache.org/. I've been looking at tools along these lines as well, and recently looked at Redis for the same reasons. Right now, though, I'm focusing my attention on Jackrabbit which is an implementation of the JSR170 repository specifications. It supports hierarchical data and lists. I was looking for some way to save dynamic objects (basically runtime defined maps) and this looks like it might work. JSR170 defines a repository as: A content repository consists of one or more workspaces, each of which contains a tree of items. An item is either a node or a property. Each node may have zero or more child nodes and zero or more child properties. There is a single root node per workspace, which has no parent. All other nodes have one parent. Properties have one parent (a node) and cannot have children; they are the leaves of the tree. All of the actual content in the repository is stored within the values of the properties. The available property types cover what appear to be all the bases, including references to other nodes to prevent cycles when serializing. It can support annotated Java objects or use the Node building capability directly and allows querying similar to XPath. Since it is a Java project, using it from Clojure should be trivial to use. On Dec 31, 5:29 am, Steve Purcell st...@sanityinc.com wrote: Not sure if it's any help, but here's a variant of memoize I wrote, which stores arbitrary readable/printable objects to redis: http://gist.github.com/266689 (If there's any interest, I'll wrap it up in a github project and push it to clojars.) Redis isn't a hierarchical store, so its array/set operations would only benefit the most shallow of data structures. -Steve On 30 Dec 2009, at 11:52, Gabi wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Clojure + Redis
On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
I think anything which lowers the impedance mismatch between Clojure data structures and a persistent store is worth investigating. I'd love to find an ACID, transactional store which accepts native structures. Right now I'm using CouchDB, and while JSON is close enough, it still requires a mapping between the Clojure and Couch worlds. Not supporting set, and not allowing fields with dashes (JavaScript thinks it's a minus) are some of the annoyances. On Wed, Dec 30, 2009 at 12:52 PM, Gabi bugspy...@gmail.com wrote: On first look, Redis and Clojure seems to be a perfect match. They both handle sets and maps efficiently. If one could find an easy way to store and retrieve Clojure data structures to Redis (even a small subset- just a list or a set), a distributed clojure app could be very easy (and effective?) thing to do - The stateless Clojure nodes would share and operate on the same central data structure which is stored in Redis). What do you thing ? Is it worth investigating further? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure + Redis
There are other NoSQL datastores written in java, like Voldemort. Perhaps if you investigate them, you will find one that will be much easier to integrate with clojure. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
