Thanks for looking into it!
Hm, there seems to be no file with fileindex=3 in the database for the copy:
Enter SQL query: select * from file where jobid=94146 and fileindex=3;
No results to list.
All files that are being backed up are just regular files, nothing special.
The copy runs locally on the director's host from its SD to the same SD.
Here's file 2 and 3 of today's consolidated incremental backup which will
be copied tomorrow and most likely lead to the same warning again as it has
for the last few days (for this specific fileset it always warns about
`File=3 attributes=2`):
Enter SQL query: select *, decode_lstat(lstat) from file where jobid=94511
and fileindex=2;
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
| fileid | fileindex | jobid | pathid | deltaseq | markid | fhinfo |
fhnode | lstat |
md5 | name | decode_lstat
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
| 484,575,603 | 2 | 94511 | 910,901 | 0 | 0 | 0 |
0 | P0D CgA4 IGk B A i A sADp5 BAA FgAo BlxAt1 BlxASF BlxVYH A A d |
D/VouduY5TF4KFacjRE7Hw | backup_www_1100CC.tar.gz |
(64771,655416,33188,1,0,34,0,738212473,4096,1441832,1707346805,1707345029,1707431431,0,0,29)
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+----------------------------------------------------------------+------------------------+--------------------------+----------------------------------------------------------------------------------------------+
Enter SQL query: select *, decode_lstat(lstat) from file where jobid=94511
and fileindex=3;
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
| fileid | fileindex | jobid | pathid | deltaseq | markid | fhinfo |
fhnode | lstat |
md5 | name | decode_lstat
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
| 484,575,604 | 3 | 94511 | 910,901 | 0 | 0 | 0 |
0 | P0D CgA5 IGk B A i A BA9gA BAA IHw BlxAt8 BlxASH BlxVYH A A d |
J9h+OdR/6XCqhqpJdJvYtw | backup_databases_1100CC.tar |
(64771,655417,33188,1,0,34,0,17029120,4096,33264,1707346812,1707345031,1707431431,0,0,29)
|
+-------------+-----------+-------+---------+----------+--------+--------+--------+---------------------------------------------------------------+------------------------+-----------------------------+-------------------------------------------------------------------------------------------+
Restore of this job (jobid=94511) works just fine:
23-Feb 13:44 bareos-sd JobId 94559: Releasing device
"FileDevice-ReadOnly-0005" (/backup_1/bareos).
23-Feb 13:44 bareos-dir JobId 94559: Max configured use duration=82,800
sec. exceeded. Marking Volume "AI-Consolidated-1771" as Used.
23-Feb 13:44 bareos-dir JobId 94559: Bareos bareos-dir
23.0.2~pre32.0a0e55739 (31Jan24):
Build OS: Ubuntu 20.04.5 LTS
JobId: 94559
Job: Restore-Files.2024-02-23_13.43.47_38
Restore Client: "bareos-fd" 23.0.2~pre32.0a0e55739 (31Jan24)
Ubuntu 20.04.5 LTS,ubuntu
Start time: 23-Feb-2024 13:43:49
End time: 23-Feb-2024 13:44:20
Elapsed time: 31 secs
Files Expected: 33
Files Restored: 33
Bytes Restored: 9,143,514,060
Rate: 294952.1 KB/s
FD Errors: 0
FD termination status: OK
SD termination status: OK
Bareos binary info: Bareos community build (UNSUPPORTED): Get
professional support from https://www.bareos.com
Job triggered by: User
Termination: Restore OK
I'll also try restoring its copy from tape once it has been copied.
Best regards,
Samuel
On Friday, February 23, 2024 at 1:16:32 PM UTC+1 Sebastian Sura wrote:
> I meant to say File 3 has no stream 1, sorry for the confusion!
> Am 23.02.24 um 13:15 schrieb Sebastian Sura:
>
> Hi Samuel
>
> thanks for the gathering this info. Ill look into the bscan issue as well
> though i think this might be a known issue.
> Regardless, regarding your actual issue: The bscan output shows that File
> 3 is missing its attributes (it has no stream 3).
> This confuses the director as it never got told that file 3 now started
> getting backed up and this is why you get the warning
> message from the director.
>
> The same is happening during the restore: the filedaemon never got told
> that now a new file was started (because it never got the attribute stream)
> so it basically merged both File 2 and File 3 into one file. Even with
> this in mind, i think the restore should have caught that and issues a
> warning.
> Ill look into why this did not happen.
>
> If you add up all the data records (stream=2) with fileid=2 and fileid=3,
> you will get 738212473 bytes, which is exactly what the filedaemon reported
> as size for file 2. Since the log contains the size it expected i imagine
> that you could manually restore file 3 by splitting file 2 into two.
>
> Can you check which file fileid 3 corresponds to ? Is it a special kind of
> file or just a normal one ?
>
> Ill try to see if i can reproduce your issue in the copy system test. Do
> you do a local copy (so copy to the same sd) or a remote one?
>
> Kind Regards
>
> Sebastian Sura
> Am 23.02.24 um 12:56 schrieb 'Samuel' via bareos-users:
>
> Hi Sebastian,
>
> After looking at the logs some more I noticed that all jobs with this
> warning have in common that they're using compression (LZ4) and autoxflate
> on replication.
> Perhaps this combination is still not entirely fixed yet. I'll try
> disabling autoxflate in the coming days.
>
> The entire result of bscan is in the attachment. However here's the last
> part of it where bscan seems to abort (?):
> $ sudo -u bareos bscan -b md5_digest_error_copy_only.bsr --list-records
> TapeDevice2 2>&1 | tee records.txt
> ...
> bscan: stored/bscan.cc:494-0 Record: SessId=3480 SessTim=1707222936
> FileIndex=23 Stream=2 len=65536
> bscan: stored/bscan.cc:494-0 Record: SessId=3480 SessTim=1707222936
> FileIndex=23 Stream=2 len=65536
> bscan: stored/bscan.cc:494-0 Record: SessId=3480 SessTim=1707222936
> FileIndex=23 Stream=2 len=65536
> bscan: stored/bscan.cc:494-0 Record: SessId=3480 SessTim=1707222936
> FileIndex=23 Stream=2 len=22528
> bscan: stored/bscan.cc:494-0 Record: SessId=3480 SessTim=1707222936
> FileIndex=23 Stream=3 len=16
> bscan: stored/bscan.cc:494-0 Record: SessId=3480 SessTim=1707222936
> FileIndex=24 Stream=1 len=100
> bscan: stored/bscan.cc:494-0 Record: SessId=3480 SessTim=1707222936
> FileIndex=25 Stream=1 len=83
> bscan: stored/bscan.cc:494-0 Record: SessId=3480 SessTim=1707222936
> FileIndex=-5 Stream=94145 len=193
> bscan: stored/bscan.cc:681-0 Could not find SessId=3480
> SessTime=1707222936 for EOS record.
> Records would have been added or updated in the catalog:
> 0 Media
> 1 Pool
> 0 Job
> 0 File
> 0 RestoreObject
> 23-Feb 12:05 bscan JobId 0: Releasing device "TapeDevice2"
> (/dev/tape/by-id/scsi-35000e111c71ac0bf-nst).
>
> I also tried restoring the copy which ends in error.
>
> Most files are restored successfully except for one file which happens to
> have FileIndex=2 (corresponding to the `attributes=2` in the warning?):
> Enter SQL query: select path,name from file,path where jobid=94146 and
> fileindex=2 and file.pathid=path.pathid;
> +-----------------------------------+--------------------------+
> | path | name |
> +-----------------------------------+--------------------------+
> | /var/backup/old/2024-02-07_23:30/ | backup_www_1100CC.tar.gz |
> +-----------------------------------+--------------------------+
>
> A full restore of the original non-copy backup jobs works fine, as do
> restores of other copy jobs on tape of filesets that don't use compression.
> The original job that was copied (93682) doesn't exist anymore; I think it
> was an always-incremental consolidated incremental backup which got
> consolidated again the following day.
>
> Best regards,
> Samuel
> --
> You received this message because you are subscribed to the Google Groups
> "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bareos-users/e766e8c6-5b58-424f-816f-f7c5d7252180n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/bareos-users/e766e8c6-5b58-424f-816f-f7c5d7252180n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> --
> Sebastian Sura [email protected]
> Bareos GmbH & Co. KG Phone: +49 221 630693-0
> https://www.bareos.com
> Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
> Komplementär: Bareos Verwaltungs-GmbH
> Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
>
> --
> You received this message because you are subscribed to the Google Groups
> "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bareos-users/111cecfe-d3e1-4eb9-8a0e-f053a2777d0f%40bareos.com
>
> <https://groups.google.com/d/msgid/bareos-users/111cecfe-d3e1-4eb9-8a0e-f053a2777d0f%40bareos.com?utm_medium=email&utm_source=footer>
> .
>
> --
> Sebastian Sura [email protected]
> Bareos GmbH & Co. KG Phone: +49 221 630693-0
> https://www.bareos.com
> Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
> Komplementär: Bareos Verwaltungs-GmbH
> Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz
>
>
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/b8c5c90a-b151-46e0-a4c4-05b912e74850n%40googlegroups.com.
