Hello Phil, On 2020-12-18 15:38, Phil Stracchino wrote: > On 12/18/20 8:02 AM, Eric Bollengier via Bacula-devel wrote: >> >> This is a strange error, it would mean that you don't use the Batch >> insert mode >> >> in your job, because we don't insert Path records one by one (Maybe in bscan >> >> in some cases). With Batch sessions, you should not have duplicates. > > > That is correct, I run with batch mode disabled because it is not > compatible with Galera clustering, because although Bacula has a batch > size configuration directive, as far as I can tell it is ignored. This > means that attribute despooling will always cause job failure on any Job > of over 128K files if running against a Galera cluster. > > (Galera 3 has a hard limit of 128k files/4GB total data size on > writesets, although it is recommended for performance reasons to keep > batch writes much smaller than that.)
I would recommend to test again, the batch mode is doing the insert of the File records 32 by 32 (with MySQL) in the batch table, and for the Path/File update, it's done with one query each. I hope that the limitation is not on the number of records written by a single query, (in something like INSERT INTO test1 SELECT * from test2), else whatever they advise, I would recommend to ask them to address this gotcha in their roadmap. >> Note that with MySQL, you can have a duplicate issue because the index is >> >> based on the first 255 characters, and if your path is very long it >> might raise >> >> an error like this (but I'm not sure if we have declared the index unique). > > Yes, it is a UNIQUE INDEX over Path(255). However, path length clearly > isn't the issue in this case. > > All jobs ran perfectly last night, and for months previously, which is > what leads me to suspect a race condition between concurrent jobs. The non-batch mode is not intensively tested, however, I believe that we lock the SQL connection to insert Path, so we should not have this error. In this specific case, an INSERT IGNORE (not very standard I believe) can probably do the work (if it returns the correct PathId). You may also look if the bacula DB is properly locked and if you have only one connection open or one per running job. Best Regards, Eric _______________________________________________ Bacula-devel mailing list Bacula-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-devel
