Re: [core.logic] Incorporating a database as a source of facts
Have you guys taken a look at Cascalog? ( https://github.com/nathanmarz/cascalog) It's a datalog implementation in Clojure that compiles down to MapReduce jobs. Cascalog's host, Cascading, allows you to pull information from a wide range of datasources (mySQL, HDFS, ElasticSearch, etc) and might fit your use case. On Sun, Nov 27, 2011 at 11:40 PM, Tassilo Horn tass...@member.fsf.orgwrote: David Nolen dnolen.li...@gmail.com writes: Hi David, As Mark said you can avoid the graph issue with tabling. core.logic has tabling. If you look at the tabling section here - https://github.com/clojure/core.logic, you should see something related to your problem. Thanks, I'll have a look. If you do then you should probably just write you own goals that can source data from your graph. This can easily be done by returning a Choice from your custom goal. (defn custom-goal [x y z] ... (choice a (fn [] ...)) choice is like a lazy sequence, you have your first value, and then a thunk to produce the remainder of the sequence. You can do whatever you want in the body of the custom-goal to make sure the optimal resultset is operated on. Great, that looks exactly what I need. Bye, Tassilo -- (What the world needs (I think) is not (a Lisp (with fewer parentheses)) but (an English (with more.))) Brian Hayes, http://tinyurl.com/3y9l2kf -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- Sam Ritchie, Twitter Inc 703.662.1337 @sritchie09 (Too brief? Here's why! http://emailcharter.org) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
On Wed, Nov 30, 2011 at 4:38 PM, Sam Ritchie sritchi...@gmail.com wrote: Have you guys taken a look at Cascalog? ( https://github.com/nathanmarz/cascalog) It's a datalog implementation in Clojure that compiles down to MapReduce jobs. Cascalog's host, Cascading, allows you to pull information from a wide range of datasources (mySQL, HDFS, ElasticSearch, etc) and might fit your use case. +1. Though I'm still curious if core.logic offers any benefits for the simpler use cases :) David -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
As Mark said you can avoid the graph issue with tabling. core.logic has tabling. If you look at the tabling section here - https://github.com/clojure/core.logic, you should see something related to your problem. While defrel/facts are neat - they are really intended for people who don't already have a bunch of data already in memory. If you do then you should probably just write you own goals that can source data from your graph. This can easily be done by returning a Choice from your custom goal. (defn custom-goal [x y z] ... (choice a (fn [] ...)) choice is like a lazy sequence, you have your first value, and then a thunk to produce the remainder of the sequence. You can do whatever you want in the body of the custom-goal to make sure the optimal resultset is operated on. For example, you might have indexed your data on the possible values x, or y, or z. You can check to see if in this call to your custom-goal whether x, y, or z are ground, that is, they are either not logic variables, or they are logic variables that are bound to values. If they are you can use those values to looked at the indexed version of your data. I plan to document this more. If you examine the defrel/fact architecture you can see exactly how this done. David On Thu, Nov 24, 2011 at 3:49 AM, Tassilo Horn tass...@member.fsf.orgwrote: Mark markaddle...@gmail.com writes: Hi Mark, Let's take a specific example: Suppose I have a Person table with id, name and gender columns. Then, I have a Parent-Child table that contains two id columns, both of which are foreign keys back into the Person table. I'd could use the logic system to define a set of rules about family relationships. I could express a query in logic terms which would then be translated into a series of SQL calls to obtain the results (forget any optimization concerns for the moment). I'm pretty much experimenting in the same direction, but not with databases but with graphs. Currently, my approach is to translate the graph's schema (type graph) to a set of relations. For example, if the schema defines a node type Person, I generate a (defrel +Person x) which succeeds if x is a Person vertex. If the schema defines an edge type Knows between Person, I generate a relation (defrel +Knows e a o) which succeeds if e is a Knows edge starting at a node a and ending at a node e. (There are also relations for the attributes of nodes and edges.) Those relations are then populated with facts about the graph. Then I can use core.logic to query the graph in terms of a fact base. For example, here's a query on a graph representing a route map that finds for any County the City being its capital. (with-fresh is only a convenience macro that expands into a `fresh' declaring fresh logic vars for all the ?variables.) (run* [q] (with-fresh (+Locality ?loc) (+ContainsLocality _ ?county ?loc) (+HasCapital _ ?county ?capital) (!= ?capital ?loc) (+name ?capital ?cname) (+name ?loc ?lname) (== q [?cname ?lname]))) Basically, that works pretty well, but there are some issues, which are probably because I take the wrong approach. First, it's very memory intensive. I have the graph in memory anyway, and basically I duplicate it into the relations. That doesn't feel correct. Second, with my simply translation approach, when I'd change the graph, the relations would be out of sync... Basically, I'd like to be able to define the relations that I have right now, at least interface-wise. But instead of duplicating the graph in facts, I'd like to give an own implementation directly on the graph. I mean, for all these relations I can easily tell when they succeed. For examle, for (+ContainsLocality e a b), if b is given, I can tell the valid bindings for e and a in O(degree(b)). Third, it quickly becomes very slow when I define some recursive relations like that: --8---cut here---start-8--- (defn connected Succeeds, if the junctions j1 and j2 are connected by connections (Ways, Streets, AirRoutes). [j1 j2] (with-fresh (conde ((+Connection _ j1 j2)) ((+Connection _ j2 j1)) ((connected j1 ?middle) (connected ?middle j2) (defn connected-locs Succeeds, if the localities l1 and l2 are connected. Localities are connected, if they contain crossroads that are connected by connections (Ways, Streets, AirRoutes). [l1 l2] (with-fresh (conde ((+Airport! l1) (+Airport! l2) (connected l1 l2)) ((+ContainsCrossroad _ l1 ?c1) (connected ?c1 ?c2) (+ContainsCrossroad _ l2 ?c2) ;; What locality is connected with what other locality? (run 250 [q] (with-fresh (+Locality ?l1) (+Locality ?l2) (!= ?l1 ?l2)(connected-locs ?l1 ?l2) (== q [?l1 ?l2]))) --8---cut
Re: [core.logic] Incorporating a database as a source of facts
By examining to-stream https://github.com/clojure/core.logic/blob/master/src/main/clojure/clojure/core/logic.clj#L1413and extend-rel https://github.com/clojure/core.logic/blob/master/src/main/clojure/clojure/core/logic.clj#L1527you should be able to see how to turn any Clojure sequence into a stream of facts that you can plug into a core.logic program. I admit this isn't exposed because I haven't thought about an ideal general interface. I will get around to this, but it's not a high priority at the moment - deep in other implementation issues. However I would love to see discussion on this front and will certainly help move things along if someone is willing to do the design work. David On Thu, Nov 24, 2011 at 11:15 AM, David Nolen dnolen.li...@gmail.comwrote: As Mark said you can avoid the graph issue with tabling. core.logic has tabling. If you look at the tabling section here - https://github.com/clojure/core.logic, you should see something related to your problem. While defrel/facts are neat - they are really intended for people who don't already have a bunch of data already in memory. If you do then you should probably just write you own goals that can source data from your graph. This can easily be done by returning a Choice from your custom goal. (defn custom-goal [x y z] ... (choice a (fn [] ...)) choice is like a lazy sequence, you have your first value, and then a thunk to produce the remainder of the sequence. You can do whatever you want in the body of the custom-goal to make sure the optimal resultset is operated on. For example, you might have indexed your data on the possible values x, or y, or z. You can check to see if in this call to your custom-goal whether x, y, or z are ground, that is, they are either not logic variables, or they are logic variables that are bound to values. If they are you can use those values to looked at the indexed version of your data. I plan to document this more. If you examine the defrel/fact architecture you can see exactly how this done. David On Thu, Nov 24, 2011 at 3:49 AM, Tassilo Horn tass...@member.fsf.orgwrote: Mark markaddle...@gmail.com writes: Hi Mark, Let's take a specific example: Suppose I have a Person table with id, name and gender columns. Then, I have a Parent-Child table that contains two id columns, both of which are foreign keys back into the Person table. I'd could use the logic system to define a set of rules about family relationships. I could express a query in logic terms which would then be translated into a series of SQL calls to obtain the results (forget any optimization concerns for the moment). I'm pretty much experimenting in the same direction, but not with databases but with graphs. Currently, my approach is to translate the graph's schema (type graph) to a set of relations. For example, if the schema defines a node type Person, I generate a (defrel +Person x) which succeeds if x is a Person vertex. If the schema defines an edge type Knows between Person, I generate a relation (defrel +Knows e a o) which succeeds if e is a Knows edge starting at a node a and ending at a node e. (There are also relations for the attributes of nodes and edges.) Those relations are then populated with facts about the graph. Then I can use core.logic to query the graph in terms of a fact base. For example, here's a query on a graph representing a route map that finds for any County the City being its capital. (with-fresh is only a convenience macro that expands into a `fresh' declaring fresh logic vars for all the ?variables.) (run* [q] (with-fresh (+Locality ?loc) (+ContainsLocality _ ?county ?loc) (+HasCapital _ ?county ?capital) (!= ?capital ?loc) (+name ?capital ?cname) (+name ?loc ?lname) (== q [?cname ?lname]))) Basically, that works pretty well, but there are some issues, which are probably because I take the wrong approach. First, it's very memory intensive. I have the graph in memory anyway, and basically I duplicate it into the relations. That doesn't feel correct. Second, with my simply translation approach, when I'd change the graph, the relations would be out of sync... Basically, I'd like to be able to define the relations that I have right now, at least interface-wise. But instead of duplicating the graph in facts, I'd like to give an own implementation directly on the graph. I mean, for all these relations I can easily tell when they succeed. For examle, for (+ContainsLocality e a b), if b is given, I can tell the valid bindings for e and a in O(degree(b)). Third, it quickly becomes very slow when I define some recursive relations like that: --8---cut here---start-8--- (defn connected Succeeds, if the junctions j1 and j2 are connected by connections (Ways, Streets, AirRoutes). [j1 j2] (with-fresh
Re: [core.logic] Incorporating a database as a source of facts
David Nolen dnolen.li...@gmail.com writes: Hi David, As Mark said you can avoid the graph issue with tabling. core.logic has tabling. If you look at the tabling section here - https://github.com/clojure/core.logic, you should see something related to your problem. Thanks, I'll have a look. If you do then you should probably just write you own goals that can source data from your graph. This can easily be done by returning a Choice from your custom goal. (defn custom-goal [x y z] ... (choice a (fn [] ...)) choice is like a lazy sequence, you have your first value, and then a thunk to produce the remainder of the sequence. You can do whatever you want in the body of the custom-goal to make sure the optimal resultset is operated on. Great, that looks exactly what I need. Bye, Tassilo -- (What the world needs (I think) is not (a Lisp (with fewer parentheses)) but (an English (with more.))) Brian Hayes, http://tinyurl.com/3y9l2kf -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
Mark markaddle...@gmail.com writes: Hi Mark, Let's take a specific example: Suppose I have a Person table with id, name and gender columns. Then, I have a Parent-Child table that contains two id columns, both of which are foreign keys back into the Person table. I'd could use the logic system to define a set of rules about family relationships. I could express a query in logic terms which would then be translated into a series of SQL calls to obtain the results (forget any optimization concerns for the moment). I'm pretty much experimenting in the same direction, but not with databases but with graphs. Currently, my approach is to translate the graph's schema (type graph) to a set of relations. For example, if the schema defines a node type Person, I generate a (defrel +Person x) which succeeds if x is a Person vertex. If the schema defines an edge type Knows between Person, I generate a relation (defrel +Knows e a o) which succeeds if e is a Knows edge starting at a node a and ending at a node e. (There are also relations for the attributes of nodes and edges.) Those relations are then populated with facts about the graph. Then I can use core.logic to query the graph in terms of a fact base. For example, here's a query on a graph representing a route map that finds for any County the City being its capital. (with-fresh is only a convenience macro that expands into a `fresh' declaring fresh logic vars for all the ?variables.) (run* [q] (with-fresh (+Locality ?loc) (+ContainsLocality _ ?county ?loc) (+HasCapital _ ?county ?capital) (!= ?capital ?loc) (+name ?capital ?cname) (+name ?loc ?lname) (== q [?cname ?lname]))) Basically, that works pretty well, but there are some issues, which are probably because I take the wrong approach. First, it's very memory intensive. I have the graph in memory anyway, and basically I duplicate it into the relations. That doesn't feel correct. Second, with my simply translation approach, when I'd change the graph, the relations would be out of sync... Basically, I'd like to be able to define the relations that I have right now, at least interface-wise. But instead of duplicating the graph in facts, I'd like to give an own implementation directly on the graph. I mean, for all these relations I can easily tell when they succeed. For examle, for (+ContainsLocality e a b), if b is given, I can tell the valid bindings for e and a in O(degree(b)). Third, it quickly becomes very slow when I define some recursive relations like that: --8---cut here---start-8--- (defn connected Succeeds, if the junctions j1 and j2 are connected by connections (Ways, Streets, AirRoutes). [j1 j2] (with-fresh (conde ((+Connection _ j1 j2)) ((+Connection _ j2 j1)) ((connected j1 ?middle) (connected ?middle j2) (defn connected-locs Succeeds, if the localities l1 and l2 are connected. Localities are connected, if they contain crossroads that are connected by connections (Ways, Streets, AirRoutes). [l1 l2] (with-fresh (conde ((+Airport! l1) (+Airport! l2) (connected l1 l2)) ((+ContainsCrossroad _ l1 ?c1) (connected ?c1 ?c2) (+ContainsCrossroad _ l2 ?c2) ;; What locality is connected with what other locality? (run 250 [q] (with-fresh (+Locality ?l1) (+Locality ?l2) (!= ?l1 ?l2)(connected-locs ?l1 ?l2) (== q [?l1 ?l2]))) --8---cut here---end---8--- Running that last query takes several minutes on a graph that contains less than thousand nodes and edges, whereas our usual graphs contain millions. Even worse, the 250 results are in fact only 3 when removing duplicates. That's probably because I get a result tuple [A B] for any possibly street connection between A and B, even if it contains cycles, etc. Is there something to restrict the search by expressing that any route between A and B makes no sense if the same node is visited twice? Bye, Tassilo -- (What the world needs (I think) is not (a Lisp (with fewer parentheses)) but (an English (with more.))) Brian Hayes, http://tinyurl.com/3y9l2kf -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
I can't speak to the duplicates issue, though I know it's common in logic-based solutions. In specific cases, I suspect the problem could be solved (or ameliorated) by tabling but I'm just getting into logic programming so I'm not too sure about that. Overall, your use case sounds very similar to mine. I've thought that the marriage of an inference engine and a graph database would yield a very powerful tool. I'm looking forward to an abstraction layer that lets me bind any data store and, even more powerfully, multiple data stores to the logic engine. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
Mark markaddle...@gmail.com writes: I can't speak to the duplicates issue, though I know it's common in logic-based solutions. In specific cases, I suspect the problem could be solved (or ameliorated) by tabling but I'm just getting into logic programming so I'm not too sure about that. Well, that's highly likely a programming error on my side. There is an infinite number of ways to get from A to B in A - X - Y - B I can go clearly go directly A, X, Y, B, but also A, X, Y, X, Y, B, and so forth. Nothing in my `connected' relation hinders it for moving back and forth as often as it wants. Overall, your use case sounds very similar to mine. I've thought that the marriage of an inference engine and a graph database would yield a very powerful tool. Ditto. ;-) I'm looking forward to an abstraction layer that lets me bind any data store and, even more powerfully, multiple data stores to the logic engine. Yeah, same for me. So instead of defining relations with defrel and adding facts, which is basically duplicating the information I already have, I'd like to provide a direct implementation that core.logic can use if that's possible. Bye, Tassilo -- (What the world needs (I think) is not (a Lisp (with fewer parentheses)) but (an English (with more.))) Brian Hayes, http://tinyurl.com/3y9l2kf -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
Sounds interesting. I can't offer much guidance on how to build such a system but I'm certainly interested in improvements to core.logic's interface to make implementing such things simpler. What exactly do you expect core.logic to do in this case? David On Wed, Nov 23, 2011 at 5:52 PM, Mark markaddle...@gmail.com wrote: I'd like to incorporate a relational database as a source of facts into the core.logic fact system. Several years ago, I worked with a Java backwards reasoning system called Mandarax ( http://mandarax.sourceforge.net/) which defined an interface between the logic engine and a fact store. The various implementations would take the constraints supplied by the engine, translate them to datastore-specific query and return a set of facts that matched (see org.mandarax.kernel.KnowledgeOwner and ClauseSet). I'd really like something like that for core.logic. I can't figure out where to plug into. If someone can point me to the particular protocol(s) to implement, I think I can map the constraints into SQL, -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
Let's take a specific example: Suppose I have a Person table with id, name and gender columns. Then, I have a Parent-Child table that contains two id columns, both of which are foreign keys back into the Person table. I'd could use the logic system to define a set of rules about family relationships. I could express a query in logic terms which would then be translated into a series of SQL calls to obtain the results (forget any optimization concerns for the moment). Conceptually, this is really no different than what can be done using the facts function today. The difference is that my facts are stored in a database rather than in memory. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
It occurs to me I may not have fully answered your question. I'd like
core.logic to supply a set on constraints which my code would convert into
a SELECT statement and then return a seq of something (vectors? maps?
facts?) that the engine would then use to reason, potentially driving
additional queries.
I'm not familiar with core.logic internals but I suspect that the
constrains could be expressed as a vector of relation names and maps.
Something like [Parent-Child {:parent Mark :child nil}]. My SQL
implementation would translate that into SELECT name FROM Parent,
Parent-Child WHERE Parent-Child.id=Parent.id AND Parent.name='Mark'.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
Seems possible but I can't say much more than that. core.logic operates on
tuples, so you could extract the resultset form the DB and pass them along
to core.logic to constrain further without much difficulty.
David
On Wed, Nov 23, 2011 at 6:24 PM, Mark markaddle...@gmail.com wrote:
It occurs to me I may not have fully answered your question. I'd like
core.logic to supply a set on constraints which my code would convert into
a SELECT statement and then return a seq of something (vectors? maps?
facts?) that the engine would then use to reason, potentially driving
additional queries.
I'm not familiar with core.logic internals but I suspect that the
constrains could be expressed as a vector of relation names and maps.
Something like [Parent-Child {:parent Mark :child nil}]. My SQL
implementation would translate that into SELECT name FROM Parent,
Parent-Child WHERE Parent-Child.id=Parent.id AND Parent.name='Mark'.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: [core.logic] Incorporating a database as a source of facts
Cool! I'll keep an eye out for the update. Thanks -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
