This script looks somewhat suspect. You start a compaction (which is an asynchronous background task) but then immediately start deleting files (which could still be in use by a running compaction)
You really want to be polling the task status APIs to check that a compaction has actually succeeded before you start deleting any old files - https://jena.apache.org/documentation/fuseki2/fuseki-server-protocol.html#tasks What I suspect is happening is that compactions are failing (most likely because you are deleting the old files out from underneath them) and those tasks are failing/getting stuck as a result. Eventually you have too many stuck tasks and new tasks cannot be accepted Also the fact that you've deleted the database files when they might still be in use is going to make queries fail Rob On 31/03/2021, 04:18, "jaa...@kolumbus.fi" <jaa...@kolumbus.fi> wrote: Hello, I've been trying TDB2 with compact. I have 2 TDB2 datasets in my jena-fuseki. Both of them are being uploaded by 50 MBs every 5 minutes. At the same time they are compacted hourly by the attached script. At some point I start getting thse messages: + curl -i -XPOST 'localhost:8061/$/compact/pxmeta_hub_fed_prod' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 59 100 59 0 0 59000 0 --:--:-- --:--:-- --:--:-- 59000 HTTP/1.1 400 Bad Request Date: Tue, 30 Mar 2021 23:54:47 GMT Fuseki-Request-Id: 2706 Content-Type: text/plain;charset=utf-8 Cache-Control: must-revalidate,no-cache,no-store Pragma: no-cache Content-Length: 59 Async task request rejected - exceeds the limit of 4 tasks After this the dataset in question doesn't any more return from queries invoked in UI. What is wrong now ? Br, Jaana Andy Seaborne kirjoitti 9.3.2021 19:58: > Hi Jaana, > > On 09/03/2021 11:40, jaa...@kolumbus.fi wrote: >> hello, >> >> I've met the following problem with jena-fuseki (should I create bug >> ticket ?): >> >> We need to update jena-fuseki dataset every 5 minutes by a 50 Mbytes >> ttl-file. > > How many triples? > And is is new data to replace the old data or in addition to the > existing data? > >> This causes the memory consumption in the machine where jena-fuseki is >> running to increase by gigas. >> >> This was 1st detected with jena-fuseki 3.8 and later with jena-fuseki >> 3.17. >> >> To be exact I executed blankdots/jena-fuseki:fuseki3.17.0 in a docker >> container posting continously that ttl-file into the same dataset >> (pxmeta_hub_fed_prod). > > This is a TDB1 database? > > TDB2 is better at this - the database still grows but there is a way > to compact the database live. > > JENA-1987 exposes the compaction in Fuseki. > https://jena.apache.org/documentation/tdb2/tdb2_admin.html > > The database grows for two reasons: it allocates space in sparse files > in 8M chunks but the space does not count in du until actually used. > The space for deleted data is not fully recycled across transactions > because it may be in-use in a concurrent operation. (TDB1 would be > very difficult to do block ref counting; in TDB2 the solution is > compaction.) > > Andy > >> >> see the output of command "du -h | sort -hr|head -30" below. attached >> the shell-script that I was executing during the time period. >> >> root@3d53dc3fdf8d:/#alias du3="du -h | sort -hr|head -30" >> root@3d53dc3fdf8d:/# du3 >> 9.0G . >> 8.5G ./data/fuseki/databases/pxmeta_hub_fed_prod >> 8.5G ./data/fuseki/databases >> 8.5G ./data/fuseki >> 8.5G ./data > >> root@3d53dc3fdf8d:/# date >> Tue Mar 9 06:02:46 UTC 2021 >> root@3d53dc3fdf8d:/# >> > >> 3.5G . >> 3.0G ./data/fuseki/databases/pxmeta_hub_fed_prod >> 3.0G ./data/fuseki/databases >> 3.0G ./data/fuseki >> 3.0G ./data >> root@3d53dc3fdf8d:/# date >> Tue Mar 9 05:28:09 UTC 2021 >> root@3d53dc3fdf8d:/# >> >> Br, Jaana
