Re: efficiently building a large list
Here's what I think, note I can't exude the same certainty as Alex, you have to wait for his comment to get the whole picture. See this cop paste of my terminal: : (de Func () (setq L 5)) - Func : (Func) - 5 : (println L) 5 - 5 Note how the L variable is set within the function and still holds its value outside of the function. This line: (set (car Idx) (+ (val (car Idx)) Amt)) ) ) ) creates variables in much the same way as my example above. I don't know what (car Idx) contains in your case but you need to be careful here, in my earlier example from the vizreader code we got in effect variables inside symbols, ie: : (setq Var Hello) - Hello : (println Var) Hello - Hello These things are transient, they cease to exist outside of the file currently being executed, plus they won't replace important stuff in the PL environment. In your case it would not have mattered though, but I'm just saying, be careful if you set using non-symbols as names. In your case, however, you could have used () to clear things out if you had used transients as containers instead: : (setq Var 123) - 123 : (setq S 123) - 123 : () - () : (println Var) Var - Var : (println S) 123 - 123 However, be careful with , because as you can see from the reference: : (setq S abc) # Read abc - abc : (== S abc) # Read again, get the same symbol - T : () # Close scope - NIL : (== S abc) # Read again, get another symbol - NIL ALL transients disappear. To sum it up, in our cases the idx is simply pointing to variables defined elsewhere, clearing out the idx won't clear out what it was pointing at... On Sun, Jun 3, 2012 at 5:10 AM, Joe Bogner joebog...@gmail.com wrote: To be more clear, this is the call pattern that I'm referring to: : (for X (idx 'A) (set (car (idx 'A X)) 0)) (Sum) 4.492 sec - 495029119 : (for X (idx 'A) (set (car (idx 'A X)) 0)) (Sum) 4.451 sec - 495029119 : (for X (idx 'A) (set (car (idx 'A X)) 0)) (Sum) 4.519 sec - 495029119 : As you can see, clearing it before calling Sum gives the correct results. On Sat, Jun 2, 2012 at 6:08 PM, Joe Bogner joebog...@gmail.com wrote: Hi Henrik - Thanks for sharing. I used your approach and it ran quickly after I built the index using balance. (bench (setq SL (by '((X) (get X 'CustNum)) sort L))) T) (bench (setq SLC (mapcar '((This) (: CustNum)) SL)) T) (off A) (bench (balance 'A SLC T)) I'm stumped one piece. If I run the below code multiple times then my total increases : (Sum) 4.466 sec - 495029119 : (Sum) 4.497 sec - 990058238 : (Sum) 4.507 sec - 1485087357 (de Sum () (zero Amount) (bench (for This SL (let (Key (: CustNum) Amt (: Amount) Idx (idx 'A Key)) (setq Amt (if Amt Amt 0)) (inc 'Amount Amt) #check figure to make sure it sums up # the val of the cell is by default a customer number, set it to be 0 if it's non-numeric (ifn (num? (val (car Idx))) (set (car Idx) 0)) (set (car Idx) (+ (val (car Idx)) Amt)) ) ) ) (sum '((X) (car X)) (idx 'A)) ) I don't know exactly how to phrase the question. I'm storing the total in the val of the cell (I think). I would have thought it was in the val of the cell stored in the index. However, if I (off A) (bench (balance 'A SLA T)) , it still duplicates. If I run this first, it clears it out: (for X (idx 'A) (set (car (idx 'A X)) 0)) Where is the value being stored such that I need to set each value of the cell to 0 regardless of rebuilding the index? Here's a simple example that I used to understand the concept: : (setq Z abc) - abc : (val Z) - abc : (set Z 0) - 0 : (val Z) - 0 : (set Z (+ (val Z) 1)) - 1 : (val Z) - 1 : Z - abc Like your example, I think I'm storing the number in the val of the symbol (cell). I apologize for the long winded question Thanks Joe On Fri, Jun 1, 2012 at 1:38 AM, Henrik Sarvell hsarv...@gmail.com wrote: I noticed you were talking about idx. The below code is from vizreader and was part of a system that counted and stored all the non-common words in every article: # We extract all words from the article without special characters and count them (dm words (L) (let Words NIL (for W L (and (setq W (lowc (pack W))) (not (common? This W)) (if (idx 'Words W T) (inc (car @)) (set W 1 (idx 'Words))) It is using idx and summing up the occurrences of each word and turned out to be the fastest way of solving that problem anyway, maybe it's helpful to you. On Fri, Jun 1, 2012 at 10:33 AM, Joe Bogner joebog...@gmail.com wrote: Thanks Tomas, I've started using nil now. This is what I came up with to aggregate the data. It actually runs reasonably well. I'm sharing because I always enjoy reading other people's picoLisp code so I figure others may as well. My source file has
Re: Parallel command execution
HI Jorge, OK, if I understood 'task' correctly this runs entirely on the 'main' process (the one that will be accessing the database and queueing the commands). Yes. Looks like if the main process is busy in a very CPU-intensive task (no IO at all) it could be missing some chances to launch additional processes, but I guess I can workaround that inserting a (wait 0) at some point in the expensive calculation to give the task a chance to run, right? Yes. Besides during 'wait', the background tasks also run when 'dbSync', 'key' or 'listen' are executed. From these, 'key' is perhaps useful here, calling (key 0) to check for a key press without delay. Another possibility is to write an explicit 'job' (e.g. with 'curry') or a coroutine, and call that both in the background task and in the CPU intensive code. Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: efficiently building a large list
Hi Joe,
the core of the question is here what you actually stored in the index.
Perhaps it helps if I try to explain the 'idx' structure.
Each node in an 'idx' tree consists either of one (if it is a leaf node)
or two (if it is an internal node) cells. The first cell holds the
payload data in the CAR, and a pointer to the second cell (if any) in
the CDR. The second cell holds pointers to the left and right subtrees.
The 'idx' function treats the payload data as a key. It is up to the
application what to do with that, like storing data there directly, or
storing a (typically transient) symbol as a _key_, while the value of
that symbol holds a _value_.
(de Sum ()
(zero Amount)
(bench
(for This SL
(let (Key (: CustNum) Amt (: Amount) Idx (idx 'A Key))
In general, 'idx' returns the first cell of a node (which effectively
means that it returns the whole subtree).
The question is what the cell returned by (idx 'A Key) in your case
contains. The line
(ifn (num? (val (car Idx))) (set (car Idx) 0))
makes me believe that the cell's CAR holds a 'var' (symbol or cell),
whose value is checked for a number: (num? (val (car Idx)))
The same here
(set (car Idx) (+ (val (car Idx)) Amt)) ) ) )
The value of the 'var' in the CAR is taken, and the incremented value
stored there. Note that you can write that a bit simpler
(inc (car Idx) Amt)
So the above assumes that the payload is a 'var' (symbol or cell). The
statement
(sum '((X) (car X)) (idx 'A)) )
retrieves all payload items (symbols or cells), takes their values
('car' in this case, so it seems to be cells), and sums them up.
This looks all right. Note that the function wrapper is not needed,
as ((X) (foo X)) is a tautology
(sum car (idx 'A))
(is tautology the right term?)
I don't know exactly how to phrase the question. I'm storing the total in
the val of the cell (I think). I would have thought it was in the val of
From the above, you don't store it in the val (i.e. the CAR) of the tree
nodes, but in the cells contained therein. This also looks correct.
Here's a simple example that I used to understand the concept:
: (setq Z abc)
- abc
: (val Z)
- abc
Are you aware that you evaluated twice here? (val 'Z) gives abc, and
then you applied 'val' to it, giving abc again because transient
symbols point to themselves initially.
: (set Z 0)
- 0
Now you have set the value of the symbol abc to zero.
You could test that with
: abc
- 0
: (val Z)
- 0
: (set Z (+ (val Z) 1))
- 1
or simply: (inc Z)
: (val Z)
- 1
: Z
- abc
OK
Concerning the above 'idx' troubles, I suggest that you build a small
'idx' tree, and look at it interactively, perhaps also with (edit 'Idx).
Cheers,
- Alex
--
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: efficiently building a large list
Thank you. Very helpful. I was confused where the value was actually being stored. I was thinking that in my example it was being stored in the cell in the index. Then, I couldn't figure out how it retained it's value after I cleared out the index. Turns out that it actually stored it back on the original symbol that held the customer number string. I'm fairly clear now on it. Definitely an interesting deep dive On Sun, Jun 3, 2012 at 4:23 AM, Alexander Burger a...@software-lab.dewrote: On Sat, Jun 02, 2012 at 06:10:05PM -0400, Joe Bogner wrote: To be more clear, this is the call pattern that I'm referring to: : (for X (idx 'A) (set (car (idx 'A X)) 0)) (Sum) ... As you can see, clearing it before calling Sum gives the correct results. OK, so this makes sense (also the assumption in my last mail about the idx values). You clear the values of the payload items. Note that 'X' in the above loop already contains the payload items (symbols or cells). So looking them up again is not necessary. You can write (for X (idx 'A) (set X 0)) A second note: (idx 'A) collects all items in the _whole_ index tree into a list. This makes sense only for small trees, of course, and is not used often in production code. Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
