Good morning,
As said yesterday, thank you so much for your answer. This is very
important for me, really.
I answer below, in green bold for instance... for being able to be
distinguished better my comments...
El 2022-03-08 11:10, Radosław Korzeniewski escribió:
> ATENCION: Este correo se ha enviado desde fuera de la organización. No pinche
> en los enlaces ni abra los adjuntos a no ser que reconozca el remitente y
> sepa que el contenido es seguro.
>
> Hello,
>
> pon., 7 mar 2022 o 12:44 egoitz--- via Bacula-devel
> <bacula-devel@lists.sourceforge.net> napisał(a):
>
>> - I'm working in trying to create an open source version of the delta
>> encoding plugin by using the bacula-fd plugin api. When working on it I have
>> seen Bacula's source is aware of delta and delta file sequentiation. I have
>> seen for instance, even a .bvfs command exists for showing deltas of a file
>> id. But, what I have not found is that Bacula works on that Delta files
>> generation (patch generation, signatures, etc...).
>
> "delta files generation" - whatever it is is solely responsible for the
> plugin.
> Bacula provides a delta sequencing mechanism only.
>
> THIS IS WHAT I WANTED TO CONFIRM :) :) YEP :) :)
>
>> I assume that Bacula in the non-fd part, acts just as a just delta file
>> holder keeping the files and stores the patch sequentiation just that.
>
> It is stored in the catalog, so during restore the Director knows that it has
> to include all incremental files in sequence instead of the latest one.
>
> I SEE... IT WAS IMPORTANT FOR ME TO CLARIFY THIS...
>
>> Bacula keeps records of deltas in database (and file storages) but only fd
>> works with them (with probably a library like librsync in the delta plugin)
>
> No, the delta sequencing - a.k.a. block level incrementals is handled by fd
> and dir where a fd/plugin is responsible for all diff + patches.
>
> I SEE...
>
>> in the sense of applying patches over an original file or even generating
>> deltas when backup. Am I wrong?. Was just for understanding the nice work
>> done and what's already written and free in Bacula's source for this purpose.
>
> Yes, the fd/plugin is responsible for all "the dirty work" and Bacula
> "framework" helps organize it.
>
> VERY CLEAR NOW RADOSLAW... I NEEDED TO CONFIRM IT :) :) BECUASE YOU KNOW...
> PERHAPS THERE IS A NON-AVAILABLE CODE THAT DOES SOMETHING ELSE WITH THAT INFO
> STORED IN CATALOG... AND JUST... FOR HAVING A VISION OF HOW IS BEING USED
> THAT STORED DATA FOR LATER WRITTING IN SOME MANNER THE PENDING CODE :) .
> THANKS MATE :)
>
>> - By the way, I have one question about virtual files. I have not seen very
>> clear (perhaps my problem as don't understand it) how to work with them. I
>> understand the concept, but have not seen a clear example of how for
>> instance in the backup you create a virtual file, how do you see it in bvfs
>> and finally... what you get after restoring. In page 36/146 of Bacula 11 for
>> developers pdf, you say "This will create a virtual file." but really you
>> are entering in the structure :
>>
>> sp->type = FT_REG;
>> sp->statp.st_mode = 0700 | S_IFREG;
>>
>> FT_REG and S_IFREG both are for regular files.... what exactly causes a
>> virtual file to be created?. Perhaps st_size -1?.
>
> In this sense a "virtual" is a file which does not exist on a backup server
> and is created programmatically with a plugin API. This file is seen by a
> Bacula as an ordinary file with a slightly different backup stream id, so fd
> will know what tool to use to restore it.
> You can fill "struct stat" with whatever acceptable values you want, where
> st_size should match as close as possible the real size of the saved file.
> The size == -1 will confuse users.
>
> OK, OK SO A VIRTUAL FILE IS WRITTEN IN THE STRUCT THE SAME MANNER AS A
> REGULAR FILE. YOU MEAN THEN, THERE'S NOTHING SHOULD BE DONE SPECIALLY IN THE
> STRUCTS, FOR BACULA TO KNOW WHETHER IT'S HANDLING A VIRTUAL OR A REGULAR FILE
> (AND OBVIOUSLY FOR ACTING ACCORDINGLY?.
>
> BUT THEN... I CAN'T REALLY UNDERSTAND HOW DOES BACULA DO THE CORRECT THING?.
> I MEAN... IT HAS SOME SORT OF TABLE, WITH THE FILEID, REFERENCING TO OTHER X
> FILEIDS TO BE RESTORED TO DISK OR SIMILAR?.
>
> PLEASE, LET ME SET AN EXAMPLE :
>
> I WANTED TO BACKED THE THREE FILES IN A VIRTUAL FILE, I SHOULD DO SOMETHING
> AS THIS?
>
> static const char *files[] = {
> "/filea",
> "/fileb",
> "/filec"
> };
> static int nb_files = 3;
> STATIC BRC STARTBACKUPFILE(BPCONTEXT *CTX, STRUCT SAVE_PKT *SP)
>
> {
> delta_test *self = get_self(ctx);
> if (!self) {
> return bRC_Error;
> }
> time_t now = time(NULL);
> sp->fname = (char *)"/name-of-the-virtual-for-grouping-the-three-files";
> sp->type = FT_REG;
> sp->statp.st_mode = 0700 | S_IFREG;
> sp->statp.st_ctime = now;
> sp->statp.st_mtime = now;
> sp->statp.st_atime = now;
> sp->statp.st_size = the-size-of-the-three-files-in-disk;
> sp->statp.st_blksize = 4096;
> sp->statp.st_blocks = 1;
> if (self->level == 'I' || self->level == 'D') {
> bRC state = bfuncs->checkChanges(ctx, sp);
> /* Should always be bRC_OK */
> sp->type = (state == bRC_Seen)? FT_NOCHG : FT_REG;
> sp->flags |= (FO_OFFSETS);
> }
> pm_strcpy(self->fname, files[]);
> return bRC_OK;
> }
>
> LATER IN THIS CASE... IT'S NOT NOTHING TO DO WITH DELTAS... JUST LEAVE THE
> CORE CODE TO HANDLE IT?
>
> WHEN HAVING TO USE DELTAS... APART OBVIOUSLY OF SETTING THE FLAGS
> APPROPIATELY YOU DO IN THIS SAME FUNCTION :
>
> PM_STRCPY(SELF->FNAME, FILES[SELF->DELTA % NB_FILES]);
>
> I GUESS IT'S FOR SYING YOU WILL HAVE N FILES EACH TIME FOR ADDING TO VIRTUAL
> FILE AND THAT ARE DELTA FILES.... ISN'T IT?
>
> AM I WRONG?
>
>> Are they relevant for what I'm trying to do?. It seems Bacula handles delta
>> sequentiation so... perhaps for this purpose I shouldn't need "virtual
>> files"?.
>
> No, the virtual files are available in command plugins api only and are used
> mainly for creating backups of different applications, i.e. running
> databases, where a standard file backup is useless, not optimal or simply
> unavailable.
>
> I SEE, I SEE MATE...
>
>> - I'm planning to implement delta encoding by checking the previous day file
>> signature done by librsync. Instead of looking at the filesystem it would be
>> nice if I could take a look at that signature in the last backup done
>> (yesterday backup).
>
> What "signature" are you thinking of?
> There is an "accurate catalog query api" in Bacula but as far I know it is
> not handling checksums (md5, sha, etc).
> You can extend this code if you wish.
>
> I JUST WANTED TO BE SURE, THAT THE SIGNATURE FILE HAS NOT BEEN MODIFIED SINCE
> I LEFT IT ON DISK WHEN GENERATED (OR REGENERATED). I DON'T REALLY MIND USING
> CHECKSUMS OR NOT, IF SOMETHING CAN SAY ME: "RELAX, NOBODY HAS CHANGED THE
> SIGNATURE FILE". I JUST WANTED TO BE SURE THE FILE IS NOT CHANGED SINCE I
> MODIFIED. DOES THE ACURATE API CAN HELP WITH THAT?. IF IT CAN, IT'S MORE THAN
> ENOUGH FOR ME. I'LL TAKE A LOOK AT ACCURATE API :) TO SEE IF IT COULD HELP ME
> :) :)
>
> . I don't really mind what the signature is... just to know if someone has
> "broken" the file with the signature... because then the patch won't allow
> later
>
>> Could it be possible in some manner, that if I see a file passed in
>> EventHandleBackupFile() to check if yesterdays signature exists in the
>> backup of yesterday, and then read the yesterday signature from the own
>> backup?. I mean, instead of having to leave the signature in the being
>> backed server's filesystem.
>
> Do you want to store your librsync signatures in a catalog database? Did you
> count the required size?
>
> A CHECKSUM LIKE A SHA256 WOULD BE ENOUGH FOR ME. I DON'T REALLY NEED THE
> WHOLE SIGNATURE....
>
>> - The last one :) . For restoring, and for the code seen (for instance in
>> insert_missing_delta()) I assume Bacula detects we are restoring a delta
>> compressed file.
>
> Block incremental is far from compressed, IMVHO. BEE Plugins (i.e. vSphere,
> Proxmox, XENServer among others) heavily use delta sequencing for VM Image
> incremental backups. This functionality is not compression which you can
> apply to generated backup stream, i.e. LZO, GZIP or GED.
>
> WELL YES... TOTALLY AGREE.... I HAVE BADLY CALLED DELTA COMPRESSION.... LETS
> SAY JUST... DELTA ENCODING.... PERHAPS I SHOULD HAVE SAID "WE ARE RESTORING A
> DELTA ENCODED FILE."
>
>> Then I assume Bacula restores apart from the own initial file, patches to
>> arrive to the day we want to restore to. Am I wrong?.
>
> I do not understand your question, so I describe how it works - for fd
> command plugin:
> - fd command plugin during backup generates delta sequence for a file and
> saves incremental information - in most cases as an offset to changed block
> and its new content, but it can do anything else, i.e. a QEMU Incremental
> Plugin generates an incremental qcow2 image in this step
> - during restore Bacula will send selected fd command plugin the same
> information and data what it generates during backup, so for XENServer Plugin
> it simply get offset and raw data to perform destination image patching, but
> for QEMU Plugin it receives a qcow2 incremental image which is then used for
> qcow2 image patching.
>
> SORRY FOR EXPLAINING WRONGLY. WHAT I TRIED TO SAY IS : "HOW THE BACKUP WILL
> LEAVE THE SOURCE FILE PLUS PATCHES IN THE DISK AFTER RESTORATION?. FOR
> INSTANCE :
>
> - BACULA CHECKS IF SOMETHING IS DELTA ENCODED IN THE DIRECTORY HE HAS BEEN
> ASKED TO GET RESTORED.
> - IF EXIST, WITH A LOOP IT RESTORES ALL PATCHES UNTIL NO MORE PATCHES TO
> RESTORE EXIST.
> - LATER CONTINUE WITH NOT DELTA ENCODED FILES?.
>
>> Perhaps later in a post-restore job I could run a shell script that tries to
>> find patches pending to be applied to a parent file. I suppose then I could
>> apply and the backup would become finally restored. Does some other more
>> elegant way you could advise me?.
>
> Do not run external scripts for that. It will hurt you more than you think.
> :)
>
> ADVISE LISTENED MATE :) :)
>
> best regards
>
> THANK YOU SO MUCH RADOSLAW!!! BEST REGARDS MY FRIEND :) :) --
> Radosław Korzeniewski
> rados...@korzeniewski.net_______________________________________________
Bacula-devel mailing list
Bacula-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-devel
