Re: Remote database access again.
Hi Henrik, So I've tried the remote pilog stuff out now (better late than never). Good :) I've attached a simple project (needs to be extracted to /opt/picolisp/projects/). Very cool stuff, indeed. You start by running: /p projects/remote-test/remote.l -go 1 /p projects/remote-test/remote.l -go 2 /p projects/remote-test/main.l Please let me first post two general notes: 1. The 'p' script was deprecated some time ago, and doesn't come with the PicoLisp releases any longer. The recommended startup command is 'pil', with an optional '+' at the end of the command line for debug mode. Then you don't need to load dbg.l and lib/debug.l. lib/misc.l is automatically included, so that only @lib/http.l must be explicitly loaded in the first lines of remote.l and main.l. 2. Since earlier this year, it is necessary to call 'tell' after an explicit 'sync' (see doc/refS.html#sync). So line 18 in remote.l holds the expression (while (rd) (sync) (tell) (out Sock (pr (eval @)) ) ) ) I have three questions: 1.) The result of insertTestData on line 50 in main.l is that only it manages to insert Jane and John in the remotes, it's as if they need to be wakened up by the first query that fails, any idea of why this happens and how to prevent it? This is a synchronization problem. As I have pointed out before, the parent process has the duty of coordinating the communication between the child processes, and should do as little other work as possible. Everything, from DB manipulations to GUI interfacing, should be done in child processes. For that reason, the 'go' function traditionally calls (rollback) immediately before starting the main event (usually (wait)), so that every spawned child process gets a virgin environment, without possibly cached object's inherited from the parent. However, in 'go' in remote.l, you have ... (pool (pack *DbDir *IdxNum)) (rollback) (mapc show (collect 'uname '+User)) (task (port (+ *IdxNum 4040)) ... The problematic line here is the 'collect'. It causes the parent to pollute its cache with objects from the DB. Especially, this contains the *DB root object. As a result, each child starts with this pre-cached and partially filled root object. I would write 'go' as (pool (pack *DbDir *IdxNum)) (mapc show (collect 'uname '+User)) (task (port (+ *IdxNum 4040)) (let? Sock (accept @) (unless (fork) (in Sock (while (rd) (sync) (tell) (out Sock (pr (eval @)) ) ) ) (bye) ) (close Sock) ) ) (forked) (rollback) ) i.e. move down the (rollback), and take care that the parents doesn't do anything else than calling (wait) after that. 2.) This is OT but I noticed that +Key (which the reference says is unique) is not respected, I am able to insert several John for instance. Right. '+Key' maintains a unique index. If you do two inserts into that tree, with the same key but two different values, the second one will remain. Consistency of objects referred to by that tree and their values are handled at higher levels. For example, the GUI will check that, and won't allow the user to give a value to an object which is already indexed by some other object. Is (request) the only way to get around this or? This has probably been discussed on a number of earlier occasions but unfortunately my memory is weak. Yes. 'request' and 'new' are different in this regard. 'request' basically first searches for the key combination, and uses the existing object if found, otherwise calls 'new'. If you call 'new' two times with the same key value, you'll get two objects and an inconsistent index. (dbCheck) will indicate an error in that case. 3.) How much work would it be to implement put! and lose! that somehow infers where an object comes from in order to perform those operations? It seems to me there are two steps to solving that problem Not sure about the concrete case. The interprocess communication protocoll must include additional information from the originating machine (some key per machine) which can be stored along with the object. I don't see that we need some special function like 'put!' for that. that should not be impossible to overcome, first inferring which server the object belongs to with the help of the file offset, then somehow going back to the original, ie {Y} - {X} and then executing the put or lose remotely with the help of {X}. Yes, this sounds feasible to me, though the above idea of simply passing information from the sender might be easier. Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Remote database access again.
I'll see what I can do, thanks for the help and happy holidays! On Mon, Dec 26, 2011 at 10:57 PM, Alexander Burger a...@software-lab.de wrote: Hi Henrik, So I've tried the remote pilog stuff out now (better late than never). Good :) I've attached a simple project (needs to be extracted to /opt/picolisp/projects/). Very cool stuff, indeed. You start by running: /p projects/remote-test/remote.l -go 1 /p projects/remote-test/remote.l -go 2 /p projects/remote-test/main.l Please let me first post two general notes: 1. The 'p' script was deprecated some time ago, and doesn't come with the PicoLisp releases any longer. The recommended startup command is 'pil', with an optional '+' at the end of the command line for debug mode. Then you don't need to load dbg.l and lib/debug.l. lib/misc.l is automatically included, so that only @lib/http.l must be explicitly loaded in the first lines of remote.l and main.l. 2. Since earlier this year, it is necessary to call 'tell' after an explicit 'sync' (see doc/refS.html#sync). So line 18 in remote.l holds the expression (while (rd) (sync) (tell) (out Sock (pr (eval @)) ) ) ) I have three questions: 1.) The result of insertTestData on line 50 in main.l is that only it manages to insert Jane and John in the remotes, it's as if they need to be wakened up by the first query that fails, any idea of why this happens and how to prevent it? This is a synchronization problem. As I have pointed out before, the parent process has the duty of coordinating the communication between the child processes, and should do as little other work as possible. Everything, from DB manipulations to GUI interfacing, should be done in child processes. For that reason, the 'go' function traditionally calls (rollback) immediately before starting the main event (usually (wait)), so that every spawned child process gets a virgin environment, without possibly cached object's inherited from the parent. However, in 'go' in remote.l, you have ... (pool (pack *DbDir *IdxNum)) (rollback) (mapc show (collect 'uname '+User)) (task (port (+ *IdxNum 4040)) ... The problematic line here is the 'collect'. It causes the parent to pollute its cache with objects from the DB. Especially, this contains the *DB root object. As a result, each child starts with this pre-cached and partially filled root object. I would write 'go' as (pool (pack *DbDir *IdxNum)) (mapc show (collect 'uname '+User)) (task (port (+ *IdxNum 4040)) (let? Sock (accept @) (unless (fork) (in Sock (while (rd) (sync) (tell) (out Sock (pr (eval @)) ) ) ) (bye) ) (close Sock) ) ) (forked) (rollback) ) i.e. move down the (rollback), and take care that the parents doesn't do anything else than calling (wait) after that. 2.) This is OT but I noticed that +Key (which the reference says is unique) is not respected, I am able to insert several John for instance. Right. '+Key' maintains a unique index. If you do two inserts into that tree, with the same key but two different values, the second one will remain. Consistency of objects referred to by that tree and their values are handled at higher levels. For example, the GUI will check that, and won't allow the user to give a value to an object which is already indexed by some other object. Is (request) the only way to get around this or? This has probably been discussed on a number of earlier occasions but unfortunately my memory is weak. Yes. 'request' and 'new' are different in this regard. 'request' basically first searches for the key combination, and uses the existing object if found, otherwise calls 'new'. If you call 'new' two times with the same key value, you'll get two objects and an inconsistent index. (dbCheck) will indicate an error in that case. 3.) How much work would it be to implement put! and lose! that somehow infers where an object comes from in order to perform those operations? It seems to me there are two steps to solving that problem Not sure about the concrete case. The interprocess communication protocoll must include additional information from the originating machine (some key per machine) which can be stored along with the object. I don't see that we need some special function like 'put!' for that. that should not be impossible to overcome, first inferring which server the object belongs to with the help of the file offset, then somehow going back to the original, ie {Y} - {X} and then executing the put or lose remotely with the help of {X}. Yes, this sounds feasible to me, though the above idea of simply passing information from the sender might be easier. Cheers, - Alex -- UNSUBSCRIBE:
Re: Remote database access again.
Hi Alex! So I've tried the remote pilog stuff out now (better late than never). I've attached a simple project (needs to be extracted to /opt/picolisp/projects/). Very cool stuff, indeed. You start by running: /p projects/remote-test/remote.l -go 1 /p projects/remote-test/remote.l -go 2 /p projects/remote-test/main.l In 3 different terminals and make sure insertTestData and findByEmail in main.l are both uncommented first. I have three questions: 1.) The result of insertTestData on line 50 in main.l is that only it manages to insert Jane and John in the remotes, it's as if they need to be wakened up by the first query that fails, any idea of why this happens and how to prevent it? 2.) This is OT but I noticed that +Key (which the reference says is unique) is not respected, I am able to insert several John for instance. Is (request) the only way to get around this or? This has probably been discussed on a number of earlier occasions but unfortunately my memory is weak. 3.) How much work would it be to implement put! and lose! that somehow infers where an object comes from in order to perform those operations? It seems to me there are two steps to solving that problem that should not be impossible to overcome, first inferring which server the object belongs to with the help of the file offset, then somehow going back to the original, ie {Y} - {X} and then executing the put or lose remotely with the help of {X}. remote-test.tar.bz2 Description: BZip2 compressed data
Re: Remote database access again.
Thanks Alex, I'll try it out this weekend! On Fri, Jan 8, 2010 at 2:30 PM, Alexander Burger a...@software-lab.dewrote: Hi Henrik, I've just reviewed http://www.mail-archive.com/picolisp@software-lab.de/msg00097.html again in order to move a word index to a separate database file that I want to access through the remote/ext mechanism. I know some of the below has been That post was a little problematical, as it - as we discussed here (or was it in IRC?) recently - contained some errors, most notably the usage of the undefined variable 'P'. Anyway, after that, I corrected and extended the reference documentation, so that I believe that it is now the most accurate and up-to-date explanation of the matter. (doc '*Ext) (doc 'remote/2) Especially 'remote/2' contains an extensive example. (tested here :) The critical part is what is the 'rsrc' function in that example. It must provide a list of cons pairs, each with an out and an in function. For now, I would recommend just to use that function, but for a production application more robustness is needed, to handle closing or failed connections. I've always had a hard time translating examples using ? into something I can use, how would I first do in this case: (solve (quote @Str Str (select (@Feeds) ((title +Feed @Str)) (part @Str @Feeds title))) @Feeds ) If we pretend that the @Feeds were located in a remote DB and the Str comes from a form field in the local process? The above (local) 'solve' looks all right. However, just as a side note, if only a single index (here 'title') is searched, you don't need the full machinery of 'select', and a simple 'db' should do: (solve (quote @Str Str (db title +Feed @Str @Feeds) ) @Feeds ) Assuming the above 'rsrc' and '*Ext' from the reference, the corresponding remote access would be (solve (quote @Str Str @Rsrc (rsrc) (remote (@Feeds . @Rsrc) (db title +Feed @Str @Feeds) ) ) @Feeds ) That is, you just embed the normal Pilog query into a 'remote' call. With 'select' it would be (solve (quote @Str Str @Rsrc (rsrc) (remote (@Feeds . @Rsrc) (select (@Feeds) ((title +Feed @Str)) (part @Str @Feeds title) ) ) ) @Feeds ) (collect 'word '+WordCount W W 'article) What would the above have to look like if the +WordCounts are located remotely? You mean, without using Pilog? Then you can also send the executable s-expr directly. I'll try it here with the 'app' demo: : (setq Sock (connect localhost 4040)) - 17 : (out Sock (pr (list 'collect ''nr ''+Item 2 5 ''nr))) - (collect 'nr '+Item 2 5 'nr) : (in Sock (rd)) - (2 3 4 5) really applicable, a simple custom handler is better. But how do I build it? The following is from the article linked to above: (task (port 4000) # Set up the object server in the background (when (setq Sock (accept @)) (unless (fork) # Child process (task P) (close P) As this is not correct (see above), better use the example in the reference of '*Ext'. It is also a little bit simpler. (in Sock (let Data (rd) (setArticleWords (car Data) (cdr Data While this would work, it would tie the server to a single special purpose. I would go with the above standard server, and just send the whole command (like the 'collect' example above). On the local machine, let's assume that the variable 'Art' holds an article that was received from the remote machine with one of the above methods ('remote' query, or direct call). Then you can do (out Sock (pr 'setArticleWords (lit Art) Data String)) The 'lit' is necessary to have the article quoted (and not evaluated) on the remote side. Also, you need to read the result after each command, with (in Sock (rd)) because the server is set up in such a way that it always sends a result. If I want to avoid that, I sometimes also set up a server task that doesn't return anything. This has the advantage that the process sending the command doesn't have to wait for the answer, so the throughput is higher. On IRC we were discussing remote put et. al. and you wrote down some examples, care to do it again? The above 'setArticleWords' might be replaced by a simple (out Sock (pr 'put! (lit Art) Data String)) Cheers, - Alex -- UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe