Hi Joe + Henrik,

On Mon, Feb 10, 2014 at 06:37:34AM -0500, Joe Bogner wrote:
> If you end up speeding up please share. I know it's just a mock example so
> may not be worth the time. It's nice to have small reproducible examples.

Oops! I just notice that the 'prune' semantics Henrik uses is outdated.

I'm not sure which PicoLisp version you use, but 'prune' has changed in
last December (with 3.1.4.13) ... Sorry, should have posted a note about
this :(

If you have a more recent version, you should call 'prune' with a count
during import operation, and just (prune) (i.e. (prune NIL)) to disable
pruning. Otherwise, pruning is not enabled at all, and your process
keeps growing and growing ...

With that, the example becomes

   (for (D Sday (>= Eday D) (inc D))
      (for (S 1 (>= 86400 S) (inc S))
         ... )
      (commit)
      (prune 10) )
   (commit)
   (prune)


Also, you can save quite some time if you pre-allocate memory, to avoid
an increase with each garbage collection. I would call (gc 800) in the
beginning, to allocate 800 MB, and (gc 0) in the end. This gives:

   (gc 800 100)
   (for (D Sday (>= Eday D) (inc D))
      (for (S 1 (>= 86400 S) (inc S))
         (let Stamp (stamp D S)
            ## (println Stamp)
            (new F '(+Transaction) 'amount 100 'createdAt Stamp) ) )
      (commit)
      (prune 10) )
   (commit)
   (prune)
   (gc 0)

I tested with 10% of the data, and got a speed increase by a factor of
seventeen (2493 sec vs. 145 sec on a notebood with HD, no SSD).



> It's neat to hear that the queries are sub second.

Yes, but I still feel uneasy by storing time and date as a string in the
database. Not only being inconvenient to use for date calculations, a
string like

   "2013-10-05 00:00:00"

takes up 20 bytes both in the entity object and in the index tree, while
a date and a time like

   (735580 53948)

takes only 10 bytes. In the experiments above, the calls to (stamp) took
21.4 seconds in total. That's 1/7th of the total import time. In
addition, because of the smaller key size, you get more index entries
into a disk block, further increasing import speed.

♪♫ Alex
-- 
UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe

Reply via email to