Re: Is there any way to keep same size between real data and TDB

[Andy Seaborne]

On 10/12/2018 04:24, Lee, Seokju | Daniel | TPDD wrote:
Hi Andy,

Thanks for the reply.

The in-memory dataset described above is fully transactional
Interesting that I didn't know that it is different from using TDB even I used 
to use only for test purpose because I thought it is the same as TDB.

TDB in -memory if TDB with a location of "--mem--" (assembler) or 
TDBFactory.createDataset().

The in-memory dataset is DatasetFactory.createTxnMem().


I have another question that is how you can keep this persistent because 
in-memory means if application is crash with whatever the reason is, it would 
be losed.
Am I right?

Yes.

Personally, I reload the data for each test or test suite or run Fuseki 
(in-process or separately).

(Just for your understanding, to use ramdisk is just for dev and stg 
environment for functional test not production. For prodecution we will use SSD)

For staging? I'd use the same setup as prod.

    Andy


Thanks
Daniel




-----Original Message-----
From: Andy Seaborne <a...@apache.org>
Sent: Friday, December 7, 2018 8:09 PM
To: users@jena.apache.org
Subject: Re: Is there any way to keep same size between real data and TDB



On 07/12/2018 01:03, Lee, Seokju | Daniel | TPDD wrote:
Greetings,

I am using Apache Jena 3.7.0 now and encounter the following issue so I would 
like to know how to solve it.

Background:

    *   We created our own sparql endpoint for using apache jena.
    *   Sometimes we need to clear data store and restore from new ttl file.
    *   For performance, we are using ram disk for TDB instead of SSD with our 
own reason.
    *   We thought that we have big enough memory for TDB

Issue

    *   Our application was just down because of full of ram disk
    *   Back then, we repeated to restore from new ttl files
    *   The number of TTL is 1.5 millions and file size is around 250MB
    *   Ramdisk size is 4 GB (first time when we restore it, ram disk used 
under 1 GB)

Have you considered using an in-memory Jena graph?

<#dataset> rdf:type ja:MemoryDataset;
     ja:data "data.trig";
.


Investigating

    *   I think nodes.dat is real data and looks like SPO.dat, POS.dat, OSP.dat 
looked didn't remove old data that I removed in my application.

Question

    *   Is there any way to keep same size between real data and TDB?

ja:MemoryDataset

    *   We are using for removal with "Model.removeAll()" and "TDB.sync()"

TDB.sync() is not necessary when using transactions.  And not using 
transactions is not a good idea for a SPARQL endpoint. TDB.sync is legacy and 
for older single-threaded applications.

The in-memory dataset described above is fully transactional (Serialization 
level isolation), uses the heap for storage so only uses what is needed and 
deleted data gets garbage collected.

(TDB2 has a compaction operation but does mean there are times where there are 
two copies of the database.)

      Andy


Thanks
Daniel