Re: Parallel command execution
Hi Jorge, Henrik,
On Tue, May 29, 2012 at 11:26:42AM +0700, Henrik Sarvell wrote:
More complicated (but prettier) would be a later - wait combo (if
possible in your situation):
Yes, I would also say that 'later' / 'wait' is the way to go. But as I
understood Jorge, he wants to do something with (a subset of) all
database objects of a given (single) database.
In the general case, I would select those objects with a 'pilog'
expression, and then process them with 'later'. If -- as in Jorge's
question -- all objects are desired, I'd use 'iter' to directly iterate
a tree:
(de processCustomers (N . Prg)
(let Lst (need N)
(iter (tree 'nr '+CuSu)
'((This)
(let Pos (wait NIL (memq NIL Lst))
(set Pos T)
(later Pos
(run Prg)
NIL ) ) ) ) ) )
I've tested it in the demo 'app'
$ ./pil app/main.l lib/too.l -main +
...
: (processCustomers 5 (msg This) (wait 500))
{2-1}
{2-5}
...
The 'Lst' variable holds a list of the desired length, initialized to
all 'NIL' elements.
Then all customer objects are iterated, using the 'nr' number index.
'memq' finds the next free slot, and points 'Pos' to it. 'set' marks
that slot with 'T', then 'later' starts to do the work in 'Prg'. Finally
the 'later' body returns 'NIL', to mark that slot as free again.
(Note that for a general solution (e.g. in a library), it is recommended
to use transient symbols for all involved parameters and variables)
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets
On further examination I have decided that websockets are overkill for my purpose, it seems that XHR v2 is a better fit for file uploads. Here is an example: https://github.com/Integralist/XHR2-Multiple-File-Upload--with-PHP- That code makes me believe that I will be able to accomplish a progressbar without having to change my current PL web server code at all. To be continued after I actually try it... On Mon, May 28, 2012 at 1:41 PM, Henrik Sarvell hsarv...@gmail.com wrote: Thanks Jose, awesome! Will check this out more thoroughly tonight. What do you mean with xor descrambling, the ut8 decode function? It has a comment labelling it as slow. On Mon, May 28, 2012 at 12:54 AM, José Romero j...@2.71828.com.ar wrote: On Sun, 27 May 2012 18:53:31 +0700 Henrik Sarvell hsarv...@gmail.com wrote: Google couldn't show me any prior discussions of websockets and picolisp. Is it too early to start thinking about this maybe, seems like the spec/ref is still changing a little bit too fast/much atm? http://en.wikipedia.org/wiki/WebSocket http://tools.ietf.org/html/rfc6455 C lib: http://git.warmcat.com/cgi-bin/cgit/libwebsockets/ It came up today when I started to think about how crude file uploads via http is with no way of knowing the progress, but they can be used for much much more than fancy uploading interfaces of course. Having looked a bit at the ref ( http://tools.ietf.org/html/rfc6455 ) it seems to me that it looks quite straight forward. The question is, what is easier/more efficient, implementing this from scratch or with the help of the C lib and native? That is the question that maybe someone more experienced in C/in general than me can answer? I guess i should have posted this here before: https://bitbucket.org/cyborgar/web.l/src/249fe9f5c7d8/web/sockets.l That implements the RFC almost completely (there's no much implementation of the error code thingamajig because actually browsers don't even implement ping/pong correctly yet!). The implementation is in pure lisp, but it could be accelerated with a C library (actually the only thing to accelerate is the xor descrambling, could be done in an 'in-like environment). The webtest.l in the root of the repo provides a (ugly) chat server example using my web.l framework. Sorry if the code is not very elegant but I didn't devote that much time to that example. Here's the documentation of that module: https://bitbucket.org/cyborgar/web.l/wiki/Sockets Also, the spec is pretty much set in stone, at least as far as the IETF is concerned, RFC 6455 is a standards track document. Cheers, José -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets
Hi Henrik, That code makes me believe that I will be able to accomplish a progressbar without having to change my current PL web server code at all. If it is only a progress bar what you are after, note that something similar is already existent in the standard PicoLisp release. It (ab)uses the Drag/Drop button for that. When you drag a document onto that button, the button's label changes to a percent display which counts up till 100% while transferring the file. You might consider using that directly, or modifying it to pop up a progress dialog instead of the simple percent text. Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Ubuntu mess
Hi all, gradually I'm getting rather disappointed by the Ubuntu maintainers. On Mon, Apr 30, 2012 at 12:15:25PM +0200, Alexander Burger wrote: bad news: The 64-bit version of PicoLisp doesn't (yet) run properly on Ubuntu 12.04 LTS. I've noticed the day before yesterday that some internal calls to C libraries (e.g. dlopen()) crashed with segfaults. To be precise: In the ... They are either not willing or not able to upgrade Ubuntu 12.04 LTS (Precise Pangolin) to a newer PicoLisp version. I've posted a bug report, asking for the upgrade, in the middle of March: https://bugs.launchpad.net/ubuntu/+source/picolisp/+bug/956731 Meanwhile, 12.04 is out, and nothing happened until now :( This is annoying, because 12.04 is a long term support version. And other releases like Mint also depend on it. The next version (12.10, quantal) has an up-to-date PicoLisp, but this isn't really helpful. Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Ubuntu mess
On May 29, 2012 at 2:04 PM Alexander Burger a...@software-lab.de wrote: I've posted a bug report, asking for the upgrade, in the middle of March: https://bugs.launchpad.net/ubuntu/+source/picolisp/+bug/956731 The report mentions that the bug is filed against Ubuntu in general and not 12.04 in particular. Is this fixed? (I am not familiar with Launchpad and can not tell.) best regards, Jakob -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Ubuntu mess
I am not sure, but I suspect that the easiest way to get this fixed in Ubuntu 12.04 is create a point release for 3.0.8.10, for instance replacing 3.0.8.10-1 with 3.0.8.10-2 or something, with a minimal fix which only fixes the dynamic loader issue. Now, I don't know the differences between 3.1.0.4 and 3.0.8.10 but in general maintainers are very conservative about changes, so as to not introduce unintended consequences. (But as I said in my previous email, the problem may not be even that, but rather that the bug was filed against the wrong package.) best regards, Jakob -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Parallel command execution
On May 29, 2012, at 12:30 AM, José Romero wrote: On Mon, 28 May 2012 23:51:21 +0200 Jorge Acereda jacer...@gmail.com wrote: Hi, I need to invoke external commands for each entry in my database and I want to run those in parallel, but restricting the number of simultaneous jobs to a certain number to avoid bringing the machine to its knees (sort of 'make -jN'). How would you approach such problem? First a bit more of information is needed, what do you need to parallelize? The processing steps within each entry, or the processing of the entries themselves? Are there any dependencies that should be taken into account? I'll try to explain better what I'm trying to do. As a learning exercise, I'm implementing a build tool (sort of 'scons'). A target can have explicit dependencies (specified when the target is inserted in the database) or calculated dependencies (in the case of a C target, obtained via 'cc -M', 'mkdep' or something similar). I still don't know how fast will be the stage that determines the build order, so at this stage i'm only worried about being able to run the mkdeps stage and the build stage in parallel. Regards, Jorge Thanks, Jorge Cheers, José -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Parallel command execution
On May 29, 2012, at 9:15 AM, Alexander Burger wrote: (de processCustomers (N . Prg) (let Lst (need N) (iter (tree 'nr '+CuSu) '((This) (let Pos (wait NIL (memq NIL Lst)) (set Pos T) (later Pos (run Prg) NIL ) ) ) ) ) ) Thanks, that looks good. My only objection would be the additional fork() implied by 'later'. I'll try to get it running with that approach and do some benchmarking. Later, I'll try to compare that to a patched version that uses vfork() instead of fork() and see if that makes a difference. Another option to avoid the fork() would be to have a pool of pre-forked instances reading from a jobs queue or something like that (perhaps taking advantage of an additional database to implement the queue?), but my skills are still lacking on how to implement that. I'll worry about that if the fork() time is noticeable. Thanks, Jorge -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Ubuntu mess
Hi Jakob, On Tue, May 29, 2012 at 02:49:39PM +0200, Jakob Eriksson wrote: I've posted a bug report, asking for the upgrade, in the middle of March: https://bugs.launchpad.net/ubuntu/+source/picolisp/+bug/956731 The report mentions that the bug is filed against Ubuntu in general and not 12.04 in particular. Is this fixed? (I am not familiar with Launchpad and can not tell.) Wow! It seems you managed to get things starting to move! :) Just now I've got a message from Launchpad Bug Tracker: Status changed to 'Confirmed' because the bug affects multiple users. It seems this is because you also registered as an affected user ;-) Thanks! Let's see ... Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Ubuntu mess
On May 29, 2012 at 4:34 PM Alexander Burger a...@software-lab.de wrote: So let's wait a little if the upgrade will happen now as the bug is finally confirmed. No, I am pretty sure that is not affecting it at all. I think you as the bug reporter need to do what the janitor code said: To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/956731/+editstatus https://bugs.launchpad.net/ubuntu/+bug/956731/+editstatus and add the package name in the text box next to the word Package. best regards, Jakob -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
db new! performance
I'm evaluating the use of picolisp for analyzing large datasets. Is it surprising that inserting a million rows into a simple db would take 5+ minutes on modern hardware? I killed it after that after about 500K were inserted. I checked by ctrl+c and then inspecting N. It seems to progressively get slower after about 100K records. (pool foo.db) (class +Invoice +Entity) (rel nr (+Key +Number)) (zero N) (do 100 (new! '(+Invoice) 'nr (inc 'N))) I have just testing out the concept. My input data will be a flat file of invoice data (12 million rows+) Thanks Joe
Re: db new! performance
Use new and chunk it up: (dbSync) (for A As (at (0 . 1000) (commit 'upd) (prune) (dbSync)) (new (db: +Article) '(+Article) key1 value1 key2 value2 ... )) (commit 'upd) With new! you are locking and writing every row so should only be used in cases where you know you are only inserting one (or maybe very few). Above we create them in memory and write 1000 of them at a time. If you have 12 million you should probably use an even higher number than 1000. /Henrik On Wed, May 30, 2012 at 10:36 AM, Joe Bogner joebog...@gmail.com wrote: I'm evaluating the use of picolisp for analyzing large datasets. Is it surprising that inserting a million rows into a simple db would take 5+ minutes on modern hardware? I killed it after that after about 500K were inserted. I checked by ctrl+c and then inspecting N. It seems to progressively get slower after about 100K records. (pool foo.db) (class +Invoice +Entity) (rel nr (+Key +Number)) (zero N) (do 100 (new! '(+Invoice) 'nr (inc 'N))) I have just testing out the concept. My input data will be a flat file of invoice data (12 million rows+) Thanks Joe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
