Hi Joe + Henrik, On Mon, Feb 10, 2014 at 06:37:34AM -0500, Joe Bogner wrote: > If you end up speeding up please share. I know it's just a mock example so > may not be worth the time. It's nice to have small reproducible examples.
Oops! I just notice that the 'prune' semantics Henrik uses is outdated. I'm not sure which PicoLisp version you use, but 'prune' has changed in last December (with 188.8.131.52) ... Sorry, should have posted a note about this :( If you have a more recent version, you should call 'prune' with a count during import operation, and just (prune) (i.e. (prune NIL)) to disable pruning. Otherwise, pruning is not enabled at all, and your process keeps growing and growing ... With that, the example becomes (for (D Sday (>= Eday D) (inc D)) (for (S 1 (>= 86400 S) (inc S)) ... ) (commit) (prune 10) ) (commit) (prune) Also, you can save quite some time if you pre-allocate memory, to avoid an increase with each garbage collection. I would call (gc 800) in the beginning, to allocate 800 MB, and (gc 0) in the end. This gives: (gc 800 100) (for (D Sday (>= Eday D) (inc D)) (for (S 1 (>= 86400 S) (inc S)) (let Stamp (stamp D S) ## (println Stamp) (new F '(+Transaction) 'amount 100 'createdAt Stamp) ) ) (commit) (prune 10) ) (commit) (prune) (gc 0) I tested with 10% of the data, and got a speed increase by a factor of seventeen (2493 sec vs. 145 sec on a notebood with HD, no SSD). > It's neat to hear that the queries are sub second. Yes, but I still feel uneasy by storing time and date as a string in the database. Not only being inconvenient to use for date calculations, a string like "2013-10-05 00:00:00" takes up 20 bytes both in the entity object and in the index tree, while a date and a time like (735580 53948) takes only 10 bytes. In the experiments above, the calls to (stamp) took 21.4 seconds in total. That's 1/7th of the total import time. In addition, because of the smaller key size, you get more index entries into a disk block, further increasing import speed. ♪♫ Alex -- UNSUBSCRIBE: mailto:email@example.com?subject=Unsubscribe