Hi Joe, I seemed to get the best performance doing a commit + prune
everyday. Tried every 7 days first (see at) but from my experience the
prune needs to happen more often.
On Mon, Feb 10, 2014 at 6:37 PM, Joe Bogner <joebog...@gmail.com> wrote:
> Henrik - Thank you for posting the code. I enjoyed tinkering around with
> it. The inserts took a long time --- I stopped after about 30m and then
> added some timing info. I think it was taking about 20 seconds per day and
> that time will grow if I recall correctly. I am guessing it would take 2-3
> hours to insert the 31M rows (on SSD and a xen environment) and a fair
> amount of disk space. I think I was up to about 2 gig with 50 days.
> I may look further into experimenting with different block sizes:
> If you end up speeding up please share. I know it's just a mock example so
> may not be worth the time. It's nice to have small reproducible examples.
> It's neat to hear that the queries are sub second.
> On Sun, Feb 9, 2014 at 6:24 AM, Henrik Sarvell <hsarv...@gmail.com> wrote:
>> "Yes, a bit perhaps."
>> I tested, it is of no consequence (at least for my applications), given
>> one transaction per second for a full year, fetching a random +Ref +String
>> day takes a fraction of a second on my PC equipped with SSD, here is the
>> Note that it's only the collect at the end that takes a fraction of a
>> section, the insertions do NOT.
>> (class +Transaction +Entity)
>> (rel amount (+Number))
>> (rel createdAt (+Ref +String))
>> (4 +Transaction)
>> (4 (+Transaction createdAt)) )
>> (pool "/opt/picolisp/projects/test/db/db" *Dbs)
>> (setq Sday (date 2013 01 01))
>> (setq Eday (+ Sday 364))
>> (setq F (db: +Transaction))
>> (for (D Sday (>= Eday D) (inc D))
>> (for (S 1 (>= 86400 S) (inc S))
>> (let Stamp (stamp D S)
>> (println Stamp)
>> (new F '(+Transaction) 'amount 100 'createdAt Stamp) ) )
>> (prune) )
>> (prune T)
>> (println (collect 'createdAt '+Transaction "2013-10-05 00:00:00"
>> "2013-10-05 23:59:59"))
>> On Sat, Feb 8, 2014 at 5:44 PM, Alexander Burger <a...@software-lab.de>wrote:
>>> Hi Henrik,
>>> On Fri, Feb 07, 2014 at 08:29:07PM +0700, Henrik Sarvell wrote:
>>> > Given a very large amount of external objects, representing for
>>> > transactions, what would be the quickest way of handling the creation
>>> > be with regards to future lookups by way of start stamp and end stamp?
>>> > It seems to me that using two relations might be optimal, one +Ref
>>> > and an extra +Ref +Time. Then a lookup could first use the +Date
>>> > to filter out all transactions that weren't created during the
>>> > days followed by (optionally) a filter by +Time.
>>> You could use two separate relations, but then I would definitely
>>> combine them with '+Aux'
>>> (rel d (+Aux +Ref +Date) (t)) # Date
>>> (rel t (+Time)) # Time
>>> In this way a single B-Tree access is sufficient to find any time range