So by (735580 53948) you mean a +Ref +List? Is it possible to get a range by way of collect with that setup?
I tested with two separate relations, ie one +Ref +Time and one +Ref +Date, the database file ended up the same size. On Mon, Feb 10, 2014 at 9:31 PM, Alexander Burger <a...@software-lab.de>wrote: > Hi Joe + Henrik, > > On Mon, Feb 10, 2014 at 06:37:34AM -0500, Joe Bogner wrote: > > If you end up speeding up please share. I know it's just a mock example > so > > may not be worth the time. It's nice to have small reproducible examples. > > Oops! I just notice that the 'prune' semantics Henrik uses is outdated. > > I'm not sure which PicoLisp version you use, but 'prune' has changed in > last December (with 3.1.4.13) ... Sorry, should have posted a note about > this :( > > If you have a more recent version, you should call 'prune' with a count > during import operation, and just (prune) (i.e. (prune NIL)) to disable > pruning. Otherwise, pruning is not enabled at all, and your process > keeps growing and growing ... > > With that, the example becomes > > (for (D Sday (>= Eday D) (inc D)) > (for (S 1 (>= 86400 S) (inc S)) > ... ) > (commit) > (prune 10) ) > (commit) > (prune) > > > Also, you can save quite some time if you pre-allocate memory, to avoid > an increase with each garbage collection. I would call (gc 800) in the > beginning, to allocate 800 MB, and (gc 0) in the end. This gives: > > (gc 800 100) > (for (D Sday (>= Eday D) (inc D)) > (for (S 1 (>= 86400 S) (inc S)) > (let Stamp (stamp D S) > ## (println Stamp) > (new F '(+Transaction) 'amount 100 'createdAt Stamp) ) ) > (commit) > (prune 10) ) > (commit) > (prune) > (gc 0) > > I tested with 10% of the data, and got a speed increase by a factor of > seventeen (2493 sec vs. 145 sec on a notebood with HD, no SSD). > > > > > It's neat to hear that the queries are sub second. > > Yes, but I still feel uneasy by storing time and date as a string in the > database. Not only being inconvenient to use for date calculations, a > string like > > "2013-10-05 00:00:00" > > takes up 20 bytes both in the entity object and in the index tree, while > a date and a time like > > (735580 53948) > > takes only 10 bytes. In the experiments above, the calls to (stamp) took > 21.4 seconds in total. That's 1/7th of the total import time. In > addition, because of the smaller key size, you get more index entries > into a disk block, further increasing import speed. > > ♪♫ Alex > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe >
