Hi Christian,
Thank you for the suggestions! I'm creating my backups through java
(actually scala) code but I ended up using an approach where I:
1. Run job:eval with a custom job ID and a query that runs a dummy
update followed by a prof:sleep for a long period of time
2. Wait for the job to be running
3. Create the backup
4. Run job:remove to cancel the job
Next time I have a slow few days at work I'll take a closer look at
contributing the .tar.zst backup behavior along with tests and
documentation :)
Best,
Matt
On Thu, Jun 12, 2025 at 6:16 AM Christian Grün <christian.gr...@gmail.com>
wrote:
> Hi Matt,
>
> I figured it might be ill-advised, just as my reason to do it might be… :)
>>
>
> …the reason sounds reasonable ;)
>
> Thanks for your offer to contribute code. This is always appreciated, but
> note in advance that the time consuming part could become me persisting on
> the notorious 20% percent to ensure consistent support of both archive
> types for all backup features, to include tests, to update the
> documentation, etc. ;)
>
> We try to keep the number of code dependencies as small as possible, so
> the best option would be to make the dependencies optional (see e.g. [1]
> for how this is done for HTML parsing), and use .tar.zst compression
> whenever the library is in the classpath.
>
> Other pragmatic approaches:
>
> 1. Use db:create-backup with compress:false() [2]. This will at least
> speed up the time for creating the backup.
> 2. With XQuery locks [3], you can at least ensure that no other process
> with the same lock key will be executed in parallel
> 3. Not sure to recommend this, but it should work: To enforce a write
> lock, you can specify a dummy update operation on a database that won’t
> change your data, and do the actual backup with proc:system:
>
> delete node db:get('factbook')//DUMMY,
> proc:system('zstd', ('-r', db:option('dbpath' || $db-name, '-o', ...)),
> proc:system('upload', 'to' 's3')
>
> Hope this helps,
> Christian
>
> [1]
> https://github.com/BaseXdb/basex/blob/main/basex-core/src/main/java/org/basex/build/html/HtmlParser.java
> [2] https://docs.basex.org/main/Database_Functions#db:create-backup
> [3] https://docs.basex.org/main/Transaction_Management#xquery_locks
>
>
>
> I’m creating database backups manually because I want to compress them
>> with tar and zstd instead of zip. I also upload them to S3. I was
>> previously creating a backup through BaseX, unzipping it, recompressing it,
>> and then uploading it, but I can achieve the same result in a much faster
>> way by creating and uploading the .tar.zst file by hand. While doing so, I
>> was hoping to lock the database.
>>
>> As an alternative, I would be happy to contribute code that lets users
>> select how to compress their backups — the .tar.zst approach requires a
>> couple extra dependencies but the compression ratio is worth it for me.
>>
>> Let me know what you think.
>>
>> Thanks,
>> Matt
>>
>> On Wed, Jun 11, 2025 at 4:44 PM Christian Grün <christian.gr...@gmail.com>
>> wrote:
>>
>>> Hi Matt,
>>>
>>> Locking is a low-level mechanism of the database system. It should be
>>> avoided to try to influence the behavior from outside.
>>>
>>> What would be your reason for (un)locking databases manually?
>>>
>>> Best,
>>> Christian
>>>
>>>
>>> Matt Dziuban <mrdziu...@gmail.com> schrieb am Mi., 11. Juni 2025, 21:55:
>>>
>>>> Hello everyone,
>>>>
>>>> Is there any function or command I can use to manually lock and unlock
>>>> a database? Based on the documentation on File-System Locks
>>>> <https://docs.basex.org/12/Transaction_Management#file-system_locks>
>>>> it sounds like maybe just creating an upd.basex file in the database
>>>> directory would be sufficient?
>>>>
>>>> Thanks in advance,
>>>> Matt
>>>>
>>>
--
Matthew R. Dziuban
mattdziuban.com
703-973-6717
mrdziu...@gmail.com
