Re: Remote database access again.

2011-12-26 Thread Alexander Burger
Hi Henrik,

 So I've tried the remote pilog stuff out now (better late than never).

Good :)

 I've attached a simple project (needs to be extracted to
 /opt/picolisp/projects/).
 
 Very cool stuff, indeed.
 
 You start by running:
 /p projects/remote-test/remote.l -go 1
 /p projects/remote-test/remote.l -go 2
 /p projects/remote-test/main.l

Please let me first post two general notes:

1. The 'p' script was deprecated some time ago, and doesn't come with
   the PicoLisp releases any longer.

   The recommended startup command is 'pil', with an optional '+' at the
   end of the command line for debug mode. Then you don't need to load
   dbg.l and lib/debug.l.

   lib/misc.l is automatically included, so that only @lib/http.l
   must be explicitly loaded in the first lines of remote.l and
   main.l.

2. Since earlier this year, it is necessary to call 'tell' after an
   explicit 'sync' (see doc/refS.html#sync). So line 18 in remote.l
   holds the expression

   (while (rd)
  (sync)
  (tell)
  (out Sock
 (pr (eval @)) ) ) )


 I have three questions:
 
 1.) The result of insertTestData on line 50 in main.l is that only it
 manages to insert Jane and John in the remotes, it's as if they need
 to be wakened up by the first query that fails, any idea of why this
 happens and how to prevent it?

This is a synchronization problem. As I have pointed out before, the
parent process has the duty of coordinating the communication between
the child processes, and should do as little other work as possible.
Everything, from DB manipulations to GUI interfacing, should be done in
child processes.

For that reason, the 'go' function traditionally calls (rollback)
immediately before starting the main event (usually (wait)), so that
every spawned child process gets a virgin environment, without possibly
cached object's inherited from the parent.

However, in 'go' in remote.l, you have

   ...
   (pool (pack *DbDir *IdxNum))
   (rollback)
   (mapc show (collect 'uname '+User))
   (task (port (+ *IdxNum 4040))
   ...

The problematic line here is the 'collect'. It causes the parent to
pollute its cache with objects from the DB. Especially, this contains
the *DB root object. As a result, each child starts with this pre-cached
and partially filled root object.

I would write 'go' as

   (pool (pack *DbDir *IdxNum))
   (mapc show (collect 'uname '+User))
   (task (port (+ *IdxNum 4040))
  (let? Sock (accept @)
 (unless (fork)
(in Sock
   (while (rd)
  (sync)
  (tell)
  (out Sock
 (pr (eval @)) ) ) )
(bye) )
 (close Sock) ) )
   (forked)
   (rollback) )

i.e. move down the (rollback), and take care that the parents doesn't do
anything else than calling (wait) after that.


 2.) This is OT but I noticed that +Key (which the reference says is
 unique) is not respected, I am able to insert several John for
 instance.

Right. '+Key' maintains a unique index. If you do two inserts into that
tree, with the same key but two different values, the second one will
remain.

Consistency of objects referred to by that tree and their values are
handled at higher levels. For example, the GUI will check that, and
won't allow the user to give a value to an object which is already
indexed by some other object.

 Is (request) the only way to get around this or? This has
 probably been discussed on a number of earlier occasions but
 unfortunately my memory is weak.

Yes. 'request' and 'new' are different in this regard. 'request'
basically first searches for the key combination, and uses the existing
object if found, otherwise calls 'new'.

If you call 'new' two times with the same key value, you'll get two
objects and an inconsistent index. (dbCheck) will indicate an error in
that case.


 3.) How much work would it be to implement put! and lose! that
 somehow infers where an object comes from in order to perform those
 operations? It seems to me there are two steps to solving that problem

Not sure about the concrete case. The interprocess communication
protocoll must include additional information from the originating
machine (some key per machine) which can be stored along with the
object. I don't see that we need some special function like 'put!' for
that.

 that should not be impossible to overcome, first inferring which
 server the object belongs to with the help of the file offset, then
 somehow going back to the original, ie {Y} - {X} and then executing
 the put or lose remotely with the help of {X}.

Yes, this sounds feasible to me, though the above idea of simply passing
information from the sender might be easier.

Cheers,
- Alex
-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe


Re: Remote database access again.

2011-12-26 Thread Henrik Sarvell
I'll see what I can do, thanks for the help and happy holidays!


On Mon, Dec 26, 2011 at 10:57 PM, Alexander Burger a...@software-lab.de wrote:
 Hi Henrik,

 So I've tried the remote pilog stuff out now (better late than never).

 Good :)

 I've attached a simple project (needs to be extracted to
 /opt/picolisp/projects/).

 Very cool stuff, indeed.

 You start by running:
 /p projects/remote-test/remote.l -go 1
 /p projects/remote-test/remote.l -go 2
 /p projects/remote-test/main.l

 Please let me first post two general notes:

 1. The 'p' script was deprecated some time ago, and doesn't come with
   the PicoLisp releases any longer.

   The recommended startup command is 'pil', with an optional '+' at the
   end of the command line for debug mode. Then you don't need to load
   dbg.l and lib/debug.l.

   lib/misc.l is automatically included, so that only @lib/http.l
   must be explicitly loaded in the first lines of remote.l and
   main.l.

 2. Since earlier this year, it is necessary to call 'tell' after an
   explicit 'sync' (see doc/refS.html#sync). So line 18 in remote.l
   holds the expression

               (while (rd)
                  (sync)
                  (tell)
                  (out Sock
                     (pr (eval @)) ) ) )


 I have three questions:

 1.) The result of insertTestData on line 50 in main.l is that only it
 manages to insert Jane and John in the remotes, it's as if they need
 to be wakened up by the first query that fails, any idea of why this
 happens and how to prevent it?

 This is a synchronization problem. As I have pointed out before, the
 parent process has the duty of coordinating the communication between
 the child processes, and should do as little other work as possible.
 Everything, from DB manipulations to GUI interfacing, should be done in
 child processes.

 For that reason, the 'go' function traditionally calls (rollback)
 immediately before starting the main event (usually (wait)), so that
 every spawned child process gets a virgin environment, without possibly
 cached object's inherited from the parent.

 However, in 'go' in remote.l, you have

   ...
   (pool (pack *DbDir *IdxNum))
   (rollback)
   (mapc show (collect 'uname '+User))
   (task (port (+ *IdxNum 4040))
   ...

 The problematic line here is the 'collect'. It causes the parent to
 pollute its cache with objects from the DB. Especially, this contains
 the *DB root object. As a result, each child starts with this pre-cached
 and partially filled root object.

 I would write 'go' as

   (pool (pack *DbDir *IdxNum))
   (mapc show (collect 'uname '+User))
   (task (port (+ *IdxNum 4040))
      (let? Sock (accept @)
         (unless (fork)
            (in Sock
               (while (rd)
                  (sync)
                  (tell)
                  (out Sock
                     (pr (eval @)) ) ) )
            (bye) )
         (close Sock) ) )
   (forked)
   (rollback) )

 i.e. move down the (rollback), and take care that the parents doesn't do
 anything else than calling (wait) after that.


 2.) This is OT but I noticed that +Key (which the reference says is
 unique) is not respected, I am able to insert several John for
 instance.

 Right. '+Key' maintains a unique index. If you do two inserts into that
 tree, with the same key but two different values, the second one will
 remain.

 Consistency of objects referred to by that tree and their values are
 handled at higher levels. For example, the GUI will check that, and
 won't allow the user to give a value to an object which is already
 indexed by some other object.

 Is (request) the only way to get around this or? This has
 probably been discussed on a number of earlier occasions but
 unfortunately my memory is weak.

 Yes. 'request' and 'new' are different in this regard. 'request'
 basically first searches for the key combination, and uses the existing
 object if found, otherwise calls 'new'.

 If you call 'new' two times with the same key value, you'll get two
 objects and an inconsistent index. (dbCheck) will indicate an error in
 that case.


 3.) How much work would it be to implement put! and lose! that
 somehow infers where an object comes from in order to perform those
 operations? It seems to me there are two steps to solving that problem

 Not sure about the concrete case. The interprocess communication
 protocoll must include additional information from the originating
 machine (some key per machine) which can be stored along with the
 object. I don't see that we need some special function like 'put!' for
 that.

 that should not be impossible to overcome, first inferring which
 server the object belongs to with the help of the file offset, then
 somehow going back to the original, ie {Y} - {X} and then executing
 the put or lose remotely with the help of {X}.

 Yes, this sounds feasible to me, though the above idea of simply passing
 information from the sender might be easier.

 Cheers,
 - Alex
 --
 UNSUBSCRIBE: 

Re: Remote database access again.

2011-12-25 Thread Henrik Sarvell
Hi Alex!

So I've tried the remote pilog stuff out now (better late than never).
I've attached a simple project (needs to be extracted to
/opt/picolisp/projects/).

Very cool stuff, indeed.

You start by running:
/p projects/remote-test/remote.l -go 1
/p projects/remote-test/remote.l -go 2
/p projects/remote-test/main.l

In 3 different terminals and make sure insertTestData and findByEmail
in main.l are both uncommented first.

I have three questions:

1.) The result of insertTestData on line 50 in main.l is that only it
manages to insert Jane and John in the remotes, it's as if they need
to be wakened up by the first query that fails, any idea of why this
happens and how to prevent it?

2.) This is OT but I noticed that +Key (which the reference says is
unique) is not respected, I am able to insert several John for
instance. Is (request) the only way to get around this or? This has
probably been discussed on a number of earlier occasions but
unfortunately my memory is weak.

3.) How much work would it be to implement put! and lose! that
somehow infers where an object comes from in order to perform those
operations? It seems to me there are two steps to solving that problem
that should not be impossible to overcome, first inferring which
server the object belongs to with the help of the file offset, then
somehow going back to the original, ie {Y} - {X} and then executing
the put or lose remotely with the help of {X}.


remote-test.tar.bz2
Description: BZip2 compressed data


Re: Remote database access again.

2010-01-08 Thread Henrik Sarvell
Thanks Alex, I'll try it out this weekend!



On Fri, Jan 8, 2010 at 2:30 PM, Alexander Burger a...@software-lab.dewrote:

 Hi Henrik,

  I've just reviewed
  http://www.mail-archive.com/picolisp@software-lab.de/msg00097.html again
 in
  order to move a word index to a separate database file that I want to
 access
  through the remote/ext mechanism. I know some of the below has been

 That post was a little problematical, as it - as we discussed here (or
 was it in IRC?) recently - contained some errors, most notably the usage
 of the undefined variable 'P'.

 Anyway, after that, I corrected and extended the reference
 documentation, so that I believe that it is now the most accurate and
 up-to-date explanation of the matter.

   (doc '*Ext)
   (doc 'remote/2)

 Especially 'remote/2' contains an extensive example. (tested here :)

 The critical part is what is the 'rsrc' function in that example. It
 must provide a list of cons pairs, each with an out and an in
 function. For now, I would recommend just to use that function, but for
 a production application more robustness is needed, to handle closing or
 failed connections.


  I've always had a hard time translating examples using ? into something I
  can use, how would I first do in this case:
 
  (solve
(quote @Str Str
   (select (@Feeds)
  ((title +Feed @Str))
  (part @Str @Feeds title)))
@Feeds )
 
  If we pretend that the @Feeds were located in a remote DB and the Str
 comes
  from a form field in the local process?

 The above (local) 'solve' looks all right. However, just as a side note,
 if only a single index (here 'title') is searched, you don't need the
 full machinery of 'select', and a simple 'db' should do:

   (solve
  (quote
 @Str Str
 (db title +Feed @Str @Feeds) )
  @Feeds )

 Assuming the above 'rsrc' and '*Ext' from the reference, the
 corresponding remote access would be

   (solve
  (quote
 @Str Str
 @Rsrc (rsrc)
 (remote (@Feeds . @Rsrc)
(db title +Feed @Str @Feeds) ) )
  @Feeds )

 That is, you just embed the normal Pilog query into a 'remote' call.

 With 'select' it would be

   (solve
  (quote
 @Str Str
 @Rsrc (rsrc)
 (remote (@Feeds . @Rsrc)
 (select (@Feeds)
   ((title +Feed @Str))
   (part @Str @Feeds title) ) ) )
  @Feeds )


  (collect 'word '+WordCount W W 'article)
 
  What would the above have to look like if the +WordCounts are located
  remotely?

 You mean, without using Pilog? Then you can also send the executable
 s-expr directly. I'll try it here with the 'app' demo:

   : (setq Sock (connect localhost 4040))
   - 17
   : (out Sock (pr (list 'collect ''nr ''+Item 2 5 ''nr)))
   - (collect 'nr '+Item 2 5 'nr)
   : (in Sock (rd))
   - (2 3 4 5)


  really applicable, a simple custom handler is better. But how do I build
 it?
  The following is from the article linked to above:
 
  (task (port 4000)  # Set up the object server in the background
   (when (setq Sock (accept @))
  (unless (fork)  # Child process
 (task P)
 (close P)

 As this is not correct (see above), better use the example in the
 reference of '*Ext'. It is also a little bit simpler.

 (in Sock
(let Data (rd)
   (setArticleWords (car Data) (cdr Data

 While this would work, it would tie the server to a single special
 purpose. I would go with the above standard server, and just send the
 whole command (like the 'collect' example above).

 On the local machine, let's assume that the variable 'Art' holds an
 article that was received from the remote machine with one of the above
 methods ('remote' query, or direct call). Then you can do

   (out Sock (pr 'setArticleWords (lit Art) Data String))

 The 'lit' is necessary to have the article quoted (and not evaluated) on
 the remote side.

 Also, you need to read the result after each command, with

   (in Sock (rd))

 because the server is set up in such a way that it always sends a
 result. If I want to avoid that, I sometimes also set up a server task
 that doesn't return anything. This has the advantage that the process
 sending the command doesn't have to wait for the answer, so the
 throughput is higher.


  On IRC we were discussing remote put et. al. and you wrote down some
  examples, care to do it again?

 The above 'setArticleWords' might be replaced by a simple

   (out Sock (pr 'put! (lit Art) Data String))

 Cheers,
 - Alex
 --
 UNSUBSCRIBE: mailto:picol...@software-lab.de?subject=unsubscribe