Hi Sebastian,
I ran the job as normal, and am attaching the trace for that run. I then
reversed the order of the Option blocks, and it appears that it did honor
the OneFS option.
As for a single Option block, I currently have this:
Options {
# Wild Dir directives are not allowed in Exclude blocks, so placing here
Wild Dir = "/usr/src"
Wild Dir = "/lib/modules"
Exclude = yes
}
Options {
Signature = MD5
One FS = No # change into other filessytems
FS Type = btrfs
FS Type = ext2
FS Type = ext3
FS Type = ext4
FS Type = reiserfs
FS Type = jfs
FS Type = xfs
FS Type = zfs
}
I presume it would work correctly if I combined these into a single
Options, like this:
Options {
# Wild Dir directives are not allowed in Exclude blocks, so placing here
Wild Dir = "/usr/src"
Wild Dir = "/lib/modules"
Exclude = yes
Signature = MD5
One FS = No # change into other filessytems
FS Type = btrfs
FS Type = ext2
FS Type = ext3
FS Type = ext4
FS Type = reiserfs
FS Type = jfs
FS Type = xfs
FS Type = zfs
}
Does this seem right? I inherited this configuration, and don't have a
strong intuition yet about whether or not any given options directives will
interact with each other. Thanks,
Toma
bareos-dir.trace01.txt
<https://drive.google.com/file/d/1y-ax_9lWTf3c41N8E5uFyex8JABICYjQ/view?usp=drive_web>
On Mon, Nov 4, 2024 at 1:01 AM Sebastian Sura <[email protected]>
wrote:
> Hi Toma,
>
> it looks like somehow One FS = NO is not honered here. According to our
> docs:
> https://docs.bareos.org/Configuration/Director.html#config-Dir_Fileset_Include_Options_OneFs
> the warning you receive is expected even if you explicitly add "File =
> /export", but it should still get backed up in that case.
> Could you verify once more that "/export" is not backed up even if you
> list it explicitly ? Is it possible that you forgot to reload your config ?
>
> There is still the issue of why it ignores your OneFS option however.
> Could you run a backup with trace level 500 and post the output here ?
> This should print the config that the file daemon received.
> Its very possible that the cause of the problem is using multiple options
> blocks. They are a bit buggy and we recommend against using them. Could
> you try switching up the order of your option blocks ? For some options
> only the last option block is considered.
>
> Kind Regards
> Sebastian Sura
> Am 30.10.24 um 16:10 schrieb Toma Morris:
>
> Ok, now that the memory issues have been resolved, it has uncovered
> another issue. It appears that the job that I am attempting to restore from
> failed to include the target files, because it did not descend into the
> /export directory with the following error message:
>
> CSSPFILE01-fd JobId 51714: /export is a different filesystem. Will not
> descend from / into it.
>
> The fileset used for this job specifies "One FS = No", and at some point
> in the past this job was descending into /export. / and /export are
> separate ext4 partitions. Here are the most relevant configs for the job
> and fileset:
>
> FileSet {
> Name = agent-fs
> Description = "Backup all regular filesystems, including postgres."
> Include {
> Options {
> #compression=GZIP
> Signature = MD5
> One FS = No # change into other filessytems
> FS Type = btrfs
> FS Type = ext2
> FS Type = ext3
> FS Type = ext4
> FS Type = jfs
> FS Type = xfs
> FS Type = zfs
> }
> Options {
> # Wild Dir directives are not allowed in Exclude blocks, so placing
> here
> Regex Dir = ".*/\.opam/*"
> Regex Dir = ".*/\.gradle/*"
> Regex Dir = ".*/\.vscode/*"
> Wild Dir = "/usr/src"
> Wild Dir = "/lib/modules"
> Exclude = yes
> }
> File = /
> Exclude Dir Containing = .nobackup
> }
> Exclude {
> File = /var/lib/bareos
> File = /var/lib/bareos/storage
> File = /proc
> File = /tmp
> File = /dev
> File = /run
> File = /sys
> File = /var/tmp
> File = /.journal
> File = /.fscka
> File = /var/lib/lxcfs
> File = */.local
> File = /swapfile
> File = /export/archive
> File = /snap
> }
> }
>
> Job {
> Name = file01-bu
> Client = csspfile01-fd
> Type = Backup
> FileSet = agent-fs
> Schedule = MonthlyCycle
> Messages = StandardNew
> Pool = IncrPool1
> Level = Incremental
> Write Bootstrap = "/var/lib/bareos/%c.bsr"
> Full Backup Pool = FullPool0
> Differential Backup Pool = DiffPool0
> Incremental Backup Pool = IncrPool1
> Priority = 10
> RunScript {
> RunsWhen = Before
> RunsOnClient = yes
> Fail Job On Error = yes
> Command = "sudo rear mkrescue"
> }
> Run Script {
> Console = ".bvfs_update"
> RunsWhen = After
> RunsOnClient = No
> }
> }
>
> I have been able to force the backup of /export going forward by adding a
> "File = /export" directive, but my understanding of how this should work
> (based on
> https://docs.bareos.org/Configuration/Director.html#config-Dir_Fileset_Include_Options_OneFs)
> is that this should not be necessary. Happy to provide any other configs,
> traces, etc as needed.
>
> -Toma
>
> On Wednesday, October 30, 2024 at 9:26:25 AM UTC-4 Bruno Friedmann
> (bruno-at-bareos) wrote:
>
>> Thanks for the feedback and the report.
>>
>> On Tuesday 29 October 2024 at 16:18:44 UTC+1 Toma Morris wrote:
>>
>>> Thank you, that pointed me in the right direction. That query succeeded,
>>> but took a very long time. I did "explain analyze" on it, and saw that it
>>> was using much more than than the allotted working memory (using 334Mb,
>>> where work_mem in postgres was set to 4Mb), which caused postgres to use an
>>> external merge rather than quicksort. I set work_mem to 512MB, which
>>> brought the execution time from 224 s to 38 s and allowed the Bareos
>>> restore interface to function.
>>>
>>> On Mon, Oct 28, 2024 at 6:59 AM Bruno Friedmann (bruno-at-bareos) <
>>> [email protected]> wrote:
>>>
>>>> Hi Toma,
>>>>
>>>> It look like the query building the tree didn't finish or vanish.
>>>>
>>>> This may happen when your PostgreSQL cluster doesn't have enough free
>>>> space to store tmp queries/tables.... During a very limited time, the disk
>>>> space become full and the connection is then cleared, PG dropping the
>>>> running worker, free space reappear just after.
>>>>
>>>> You may want to try the following query (issue from your trace) and
>>>> watch carefully the free space available where you PG cluster reside and
>>>> build its temp data
>>>>
>>>> SELECT DISTINCT ON (PathId) 'D', SpecialDir.PathId, SpecialDir.Path,
>>>> JobId, LStat, FileId FROM ( SELECT 2711925 AS PathId, '.' AS Path UNION
>>>> SELECT PPathId AS PathId, '..' AS Path FROM PathHierarchy WHERE PathId =
>>>> 2711925 ) AS SpecialDir LEFT JOIN ( SELECT PathId, JobId, LStat, FileId
>>>> FROM File WHERE File.Name = '' AND File.JobId IN
>>>> (46255,46283,46436,46623,46653,51205,51516,51538,51560,51582,51604) ORDER
>>>> BY PathId ASC, JobId DESC ) AS DirAttribute ON (SpecialDir.PathId =
>>>> DirAttribute.PathId) UNION ( SELECT DISTINCT ON (Path) 'D', PathId, Path,
>>>> JobId, LStat, FileId FROM ( SELECT Path1.PathId AS PathId, Path1.Path AS
>>>> Path, lower(Path1.Path) AS lpath, listfile1.JobId AS JobId, listfile1.LStat
>>>> AS LStat, listfile1.FileId AS FileId FROM ( SELECT listpath1.PathId AS
>>>> PathId FROM ( SELECT DISTINCT PathHierarchy1.PathId AS PathId FROM
>>>> PathHierarchy AS PathHierarchy1 INNER JOIN Path AS Path2 ON
>>>> (PathHierarchy1.PathId = Path2.PathId) INNER JOIN PathVisibility AS
>>>> PathVisibility1 ON (PathHierarchy1.PathId = PathVisibility1.PathId) WHERE
>>>> PathHierarchy1.PPathId = 2711925 AND PathVisibility1.JobId IN
>>>> (46255,46283,46436,46623,46653,51205,51516,51538,51560,51582,51604) ) AS
>>>> listpath1 LEFT JOIN ( SELECT PVD1.PathId AS PathId FROM ( SELECT PV1.PathId
>>>> AS PathId, MAX(JobId) AS MaxJobId FROM PathVisibility AS PV1 WHERE JobId IN
>>>> (46255,46283,46436,46623,46653,51205,51516,51538,51560,51582,51604) GROUP
>>>> BY PathId ) AS PVD1 INNER JOIN File AS F2 ON (F2.PathId = PVD1.PathId AND
>>>> F2.JobId = PVD1.MaxJobId AND F2.FileIndex = 0 AND F2.Name = '') ) AS
>>>> listpath2 ON (listpath1.PathId = listpath2.PathId) WHERE listpath2.PathId
>>>> IS NULL ) AS listpath3 INNER JOIN Path AS Path1 ON (listpath3.PathId =
>>>> Path1.PathId) LEFT JOIN ( SELECT File1.PathId AS PathId, File1.JobId AS
>>>> JobId, File1.LStat AS LStat, File1.FileId AS FileId FROM File AS File1
>>>> WHERE File1.Name = '' AND File1.JobId IN
>>>> (46255,46283,46436,46623,46653,51205,51516,51538,51560,51582,51604) ) AS
>>>> listfile1 ON (listpath3.PathId = listfile1.PathId) ) AS A ORDER BY Path
>>>> ASC, JobId DESC ) ORDER BY Path ASC,JobId DESC LIMIT 1000 OFFSET 0
>>>>
>>>>
>>>> On Friday 25 October 2024 at 10:37:27 UTC+2 Toma Morris wrote:
>>>>
>>>>> Here is the debug trace during attempted restore, resulting in the
>>>>> same error condition. Thank you!
>>>>>
>>>>> -Toma
>>>>>
>>>>> On Thursday, October 24, 2024 at 12:40:24 AM UTC-4 Sebastian Sura
>>>>> wrote:
>>>>>
>>>>>> Hi Toma,
>>>>>>
>>>>>> could you try doing a debug trace on the director when you are trying
>>>>>> the restore ?
>>>>>> You can enable the debug log by writing
>>>>>>
>>>>>> setdebug level=200 trace=1 director
>>>>>>
>>>>>> in the bconsole. Once you have reproduced the issue, you can turn
>>>>>> it off with e.g. setdebug level=10 trace=0 director again.
>>>>>> Having this debug trace would be immensely helpful.
>>>>>>
>>>>>> Kind Regards,
>>>>>>
>>>>>> Sebastian Sura
>>>>>> Am 23.10.24 um 19:47 schrieb Toma Morris:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to restore files from one of my backups. In the Restore
>>>>>> tab of the webui, I select the client and then after a delay I get the
>>>>>> following error message:
>>>>>>
>>>>>> {"id":"#","xhr":{"readyState":4,"responseText":"Decoding failed:
>>>>>> Syntax errorError: API 2 not available on director.\n
>>>>>> Please upgrade to verison 15.2.2 or greater and/or compile with jansson
>>>>>> support.","status":200,"statusText":"OK"}
>>>>>>
>>>>>> Bareos is running on an Ubuntu 20.04 server with hostname cscisbu01,
>>>>>> and I am trying to access a backup from a client with hostname
>>>>>> csspfile01.
>>>>>> I initially got this error while on 23.0.4~pre169.6644119b5-114. I
>>>>>> upgraded
>>>>>> bareos on both the server and client to the most recent version via apt,
>>>>>> which brought me up to 23.0.5~pre146.7e91df1c0-130 with no apparent
>>>>>> change
>>>>>> in behavior.
>>>>>>
>>>>>> I have successfully restored files within the last couple of months
>>>>>> on this server from the same client, and have not changed any
>>>>>> configurations since then.
>>>>>>
>>>>>> I've attached my apache2 error logs, which contain some
>>>>>> possibly-relevant php errors. I didn't see anything relevant in
>>>>>> /var/log/bareos/bareos.log or /var/log/bareos/bareos-audit.log, but can
>>>>>> provide those or anything else if needed.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -Toma
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "bareos-users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion visit
>>>>>> https://groups.google.com/d/msgid/bareos-users/4ea3f42e-9b0e-4c3f-b28d-a2f382b3d81an%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/bareos-users/4ea3f42e-9b0e-4c3f-b28d-a2f382b3d81an%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> --
>>>>>> Sebastian Sura [email protected]
>>>>>> Bareos GmbH & Co. KG Phone: +49 221 630693-0
>>>>>> https://www.bareos.com
>>>>>> Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
>>>>>> Komplementär: Bareos Verwaltungs-GmbH
>>>>>> Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
>>>>>>
>>>>>> --
>>>>
>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "bareos-users" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/bareos-users/t7mBaL-hNKI/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> [email protected].
>>>> To view this discussion visit
>>>> https://groups.google.com/d/msgid/bareos-users/cb1770ec-d5b0-4744-9d56-46eb8a30ccfdn%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/bareos-users/cb1770ec-d5b0-4744-9d56-46eb8a30ccfdn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion visit
> https://groups.google.com/d/msgid/bareos-users/0d04ff3a-043d-4f41-8cee-ae5386e11580n%40googlegroups.com
> <https://groups.google.com/d/msgid/bareos-users/0d04ff3a-043d-4f41-8cee-ae5386e11580n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> --
> Sebastian Sura [email protected]
> Bareos GmbH & Co. KG Phone: +49 221 630693-0
> https://www.bareos.com
> Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
> Komplementär: Bareos Verwaltungs-GmbH
> Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "bareos-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/bareos-users/t7mBaL-hNKI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion visit
> https://groups.google.com/d/msgid/bareos-users/ba245412-8ecf-4a28-b9b8-91a366b4277f%40bareos.com
> <https://groups.google.com/d/msgid/bareos-users/ba245412-8ecf-4a28-b9b8-91a366b4277f%40bareos.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/bareos-users/CAKfWHbq%3DdgYJo_LpGSaVe9U%2BF8jDHS9U5jFbdPR0gtQeKDWC3g%40mail.gmail.com.
