Re: A dbs+ problem

[Henrik Sarvell]

>
> For example, when I have a new class of entities where I estimate that
> the these are relatively large objects, I add them to 'D'. Creating a
> new file instead of that won't have any performance advantage, because
> access to objects is done with 'lseek' and thus the size of the files
> doesn't matter.
>
> Opening too many files might even have disadvantages, due to the number
> of open file descriptors, and may have hardly-to-predict dis/advantages
> due to worse/better disk cache efficiency.
>

So you recommend 10 files in total, at the begining, no need for more, it
might even be suboptimal with more, yet at the end your example has a lot
more files than 10.

I'm confused?




On Sun, Sep 8, 2013 at 3:36 PM, Alexander Burger <a...@software-lab.de>wrote:

> On Sun, Sep 08, 2013 at 02:49:31PM +0700, Henrik Sarvell wrote:
> > At the same time projects that rely on the framework need to be able to
> do
> > the same thing without being screwed if the framework needs a new "slot".
>
> As I wrote, allocating a new "slot" not a problem. In fact, this is
> normally happening quite frequently in a typical application.
>
> Still: Is it normally not necessary to extend the number of files each
> time new class or index is added.
>
> Usually, I start up with a skeleton like:
>
>    # Database sizes
>    (dbs
>       (3 +Role +User)                  # 512 Prevalent objects
>       (0 )                             # A:64 Tiny objects
>       (1 (+User pw))                   # B:128 Small objects
>       (2 )                             # C:256 Normal objects
>       (4 )                             # D:1024 Large objects
>       (6 )                             # E:4096 Huge objects
>       (2 (+Role nm) (+User nm))        # F:256 Small indexes
>       (4 )                             # G:1024 Normal indexes
>       (6 ) )                           # H:4096 Large indexes
>
> and then, whenever necessary, add new entries to _these_ files.
>
> For example, when I have a new class of entities where I estimate that
> the these are relatively large objects, I add them to 'D'. Creating a
> new file instead of that won't have any performance advantage, because
> access to objects is done with 'lseek' and thus the size of the files
> doesn't matter.
>
> Opening too many files might even have disadvantages, due to the number
> of open file descriptors, and may have hardly-to-predict dis/advantages
> due to worse/better disk cache efficiency.
>
> For a concrete result of that process, see the 'dbs' call at the end of
> this mail [*]. Here we probably already _have_ too many files ;-)
>
>
> Also, keep in mind that this whole issue is not sooo extremely critical,
> as long as you don't put _everything_ into a single file. If objects are
> smaller than the block size, a little space is wasted, and if they are
> larger, they will occupy more than one block, resulting in slightly
> longer load times (which is not critical either, because objects are
> cached anyway).
>
> A little more critical than entities are the indexes. The block size
> should not be too small here, but if you go with all indexes on block
> size 4, you are on the safe side (just wasting _a_little_ space for very
> small indexes). In my experience, index block sizes of more than 6 show
> a decrease in performance, probably because of longer loading times.
>
>
> > If *Dbs is only used to point to locations where new objects are created
> > how does the database logic work for objects that already exist? How does
> > it know to look for X in file Y if it doesn't use the pointers in *Dbs?
>
> The value of '*Dbs' itself is used _only_ in 'pool' (to create or open
> the DB). So this list in *Dbs doesn't hold any pointers, just the
> indicators for the block sizes in the corresponding file. And even these
> are used only once, when the DB is created the first time. After that,
> they are just used as indicators to _open_ the existing files (where the
> block sizes are already kept in the file headers).
>
>
> Objects are located solely by their "name", which encodes the file
> number and the file-offset to the symbol's first block. Once an object
> is created, it won't move (its first block, that is)
>
> ♪♫ Alex
>
>
> *) Example from an existing application (just for illustration, no need
>    to understand the details).
>
> # Database sizes
> (dbs
>    (4)
>    (1 +Note +Leg (+User pw))
>    (1 +Role +Jahr +Art +ArtKat +Status +BSped +Abteilung +Inhalt +Land)
>    (1 +Kfz +Log +Currency +Team +Steuer +Anlass +Branche +Carrier +Inco
> +Svs)
>    (1 +Memo +Besuch +Anr +AbwChk +Abw +Arb)
>    (2 +User +Mitarb +Sup +Ort +Dln +Ves +ZollAmt +Lag)
>    (3 +Kleb +PrForm)
>    (4 +Firma +Entn +Artikel +Schaden +AbhA)
>    (4 +TxtKat +MacText +BrfText +prs +PGrp)
>    (4 +Aust)
>    (4 +Messe)
>    (5 +Beleg +PosText)
>    (5 +Sendung)
>    (5 +Ansp)
>    (6 +Person)
>    (3
>       (+User nm)
>       (+Role nm)
>       (+Kfz kz)
>       (+Currency key)
>       (+Team key)
>       (+Jahr key)
>       (+Steuer txt)
>       (+ZollAmt nm)
>       (+Anlass txt)
>       (+Anr nm)
>       (+PGrp nm prs)
>       (+Ves nm ets)
>       (+PrForm nm)
>       (+Land de en es) )
>    (4 (+Leg pos ansp dat mes))
>    (4
>       (+Kfz mit)
>       (+Mitarb sb plz ort tel fax mob mail)
>       (+AbwChk nr txt)
>       (+Abw ter)
>       (+Entn snd)
>       (+Arb nr mes snd)
>       (+Lag nr mes snd)
>       (+Schaden pos btg mit dat kw) )
>    (4 (+Note nm act kw) (+Log mit) (+Mitarb name) (+Memo dat mit))
>    (3 (+Entn nr) (+Sup mc) (+Branche key) (+Carrier key) (+Inco key))
>    (4 (+Sup nm plz ort tel fax mob))
>    (4 (+Entn nm txt dat tim kfz))
>    (4 (+Artikel nm))
>    (4 (+Artikel sup mc kat))
>    (4 (+Ort nm lnd))
>    (4 (+Ort cont exp imp))
>    (4 (+Branche txt) (+Carrier nm) (+Inco nm))
>    (4 (+Besuch dat txt))
>    (4 (+Person nr nr2))
>    (4
>       (+Kunde nm)
>       (+Partner nm)
>       (+Btg nm)
>       (+Dfg nm)
>       (+Org nm)
>       (+Dtv nm)
>       (+LkwU nm)
>       (+Sonst nm) )
>    (4
>       (+Kunde plz ort lnd tel fax mail)
>       (+Partner plz ort lnd tel fax mail)
>       (+Btg plz ort lnd tel fax mail)
>       (+Dfg plz ort lnd tel fax mail)
>       (+Org plz ort lnd tel fax mail)
>       (+Dtv plz ort lnd tel fax mail)
>       (+LkwU plz ort lnd tel fax mail)
>       (+Sonst plz ort lnd tel fax mail) )
>    (4 (+Ansp nm))
>    (4 (+Ansp bra tel fax mob mail gtxt gt))
>    (4 (+Messe plan imp exp kw))
>    (4 (+Mes nm) (+Arc nm))
>    (4 (+Mes ort von bis bra ueb vo part) (+Arc ort von bis bra ueb vo
> part))
>    (4 (+Sendung pos ansp mit car inc dat awb lkw))
>    (4 (+Sendung aust stb ksp avo hal std ves kw))
>    (4 (+Beleg ansp nr dat mit) (+Rech of) (+Guts of))
>    (4
>       (+Aust lst)
>       (+PosText mc)
>       (+Art txt)
>       (+ArtKat txt)
>       (+Status txt)
>       (+BSped txt)
>       (+Abteilung txt)
>       (+Inhalt txt)
>       (+TxtKat bez)
>       (+MacText txt)
>       (+BrfText mc) )
>    (4 (+Person zd sd bse bse1 bse2 bse3 bse4))
>    (0 +Doc +Calc +V)
>    (6 +Tarif +Manifest +Instr)
>    (4 (+Calc nr trf) (+Tarif nm)) )
> --
> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
>