Re: [Bacula-users] Again vchanger and volumes in error...

Josh Fisher via Bacula-users Mon, 18 Mar 2024 10:51:35 -0700


On 3/15/24 12:28, Marco Gaiarin wrote:

Following the hint on:
        
https://gitlab.bacula.org/bacula-community-edition/bacula-community/-/issues/2683

i (re)post here, seeking feedback.


Situation: bacula 9.4 (debian buster), using RDX cassette/disks for backup,
using the wonderful 'vchanger' virtual autochanger script.

Following the vchanger doc:
        https://vchanger.sourceforge.io/
        
https://sourceforge.net/projects/vchanger/files/vchangerHowto.html/download

it is needed to create a virtual changer device for every 'media' in the
'media pool'; so my SD configuration is:

  Autochanger {
    Name = RDXAutochanger
    Description = "RDX Virtual Autochanger on ODPVE2"
    Device = RDXStorage0
    Device = RDXStorage1
    Device = RDXStorage2
    Changer Command = "/usr/bin/vchanger %c %o %S %a %d"
    Changer Device = "/etc/vchanger/ODPVE2RDX.conf"
  }
  Device {
    Name = RDXStorage0
    Description = "RDX 0 File Storage on ODPVE2"
    Drive Index = 0
    Device Type = File
    Media Type = RDX
    RemovableMedia = no
    RandomAccess = yes
    Maximum Concurrent Jobs = 1
    Archive Device = "/var/spool/vchanger/ODPVE2RDX/0"
  }
  Device {
    Name = RDXStorage1
    Description = "RDX 1 File Storage on ODPVE2"
    Drive Index = 1
    Device Type = File
    Media Type = RDX
    RemovableMedia = no
    RandomAccess = yes
    Maximum Concurrent Jobs = 1
    Archive Device = "/var/spool/vchanger/ODPVE2RDX/1"
  }
  Device {
    Name = RDXStorage2
    Description = "RDX 2 File Storage on ODPVE2"
    Drive Index = 2
    Device Type = File
    Media Type = RDX
    RemovableMedia = no
    RandomAccess = yes
    Maximum Concurrent Jobs = 1
    Archive Device = "/var/spool/vchanger/ODPVE2RDX/2"
  }

every 'media' in 'media pool' have some volumes in the pool, more or less
like inserting in an (real) autochanger a set of tapes.

So when i insert a cartdrige, i get:

  root@odpve2:~# bconsole
  Connecting to Director bacula.lnf.it:9101
  1000 OK: 103 lnfbacula-dir Version: 9.4.2 (04 February 2019)
  Enter a period to cancel a command.
  *list media pool=VEN-OD-ODPVE2RDXPool
  Automatically selected Catalog: BaculaLNF
  Using Catalog "BaculaLNF"
  
+---------+---------------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+
  | mediaid | volumename          | volstatus | enabled | volbytes        | 
volfiles | volretention | recycle | slot | inchanger | mediatype | voltype | 
volparts | lastwritten         | expiresin |
  
+---------+---------------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+
  |      25 | ODPVE2RDX_0000_0000 | Used      |       1 |  15,258,511,119 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-04 23:09:08 |   798,842 |
  |      26 | ODPVE2RDX_0000_0001 | Used      |       1 |  17,769,884,030 |     
   4 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-06 23:11:49 |   971,803 |
  |      27 | ODPVE2RDX_0000_0002 | Used      |       1 |  65,296,705,760 |     
  15 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-03 02:09:00 |   636,834 |
  |      28 | ODPVE2RDX_0000_0003 | Used      |       1 |  14,995,402,621 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-02 23:09:13 |   626,047 |
  |      29 | ODPVE2RDX_0000_0004 | Used      |       1 |  16,099,504,717 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-05 23:12:59 |   885,473 |
  |      30 | ODPVE2RDX_0000_0005 | Used      |       1 |  15,067,862,578 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-03 23:11:20 |   712,574 |
  |      31 | ODPVE2RDX_0000_0006 | Used      |       1 |  15,359,960,121 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-08 09:55:38 | 1,096,832 |
  |      32 | ODPVE2RDX_0000_0007 | Used      |       1 | 259,203,030,230 |     
  60 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-01 23:35:29 |   541,223 |
  |      55 | ODPVE2RDX_0001_0000 | Used      |       1 |  16,354,496,268 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-12 23:10:10 | 1,486,504 |
  |      56 | ODPVE2RDX_0001_0001 | Used      |       1 |  15,253,608,839 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-10 23:11:32 | 1,317,386 |
  |      57 | ODPVE2RDX_0001_0002 | Used      |       1 |  65,139,795,652 |     
  15 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-10 02:09:59 | 1,241,693 |
  |      58 | ODPVE2RDX_0001_0003 | Used      |       1 |  16,239,231,609 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-11 23:10:35 | 1,400,129 |
  |      59 | ODPVE2RDX_0001_0004 | Used      |       1 |  15,986,423,895 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-14 23:13:56 | 1,659,530 |
  |      60 | ODPVE2RDX_0001_0005 | Used      |       1 |  16,833,801,705 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-13 23:10:37 | 1,572,931 |
  |      61 | ODPVE2RDX_0001_0006 | Used      |       1 | 261,299,430,776 |     
  60 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-08 23:34:25 | 1,145,959 |
  |      62 | ODPVE2RDX_0001_0007 | Used      |       1 |  15,131,374,566 |     
   3 |    1,728,000 |       1 |    0 |         0 | RDX       |       1 |        
0 | 2024-03-09 23:08:44 | 1,230,818 |
  |     155 | ODPVE2RDX_0002_0000 | Used      |       1 |  14,852,213,967 |     
   3 |    1,728,000 |       1 |    1 |         1 | RDX       |       1 |        
0 | 2024-02-24 23:08:03 |    21,177 |
  |     156 | ODPVE2RDX_0002_0001 | Used      |       1 |  64,919,242,574 |     
  15 |    1,728,000 |       1 |    2 |         1 | RDX       |       1 |        
0 | 2024-02-25 02:08:49 |    32,023 |
  |     157 | ODPVE2RDX_0002_0002 | Used      |       1 |  16,368,880,451 |     
   3 |    1,728,000 |       1 |    3 |         1 | RDX       |       1 |        
0 | 2024-02-28 23:08:56 |   366,830 |
  |     158 | ODPVE2RDX_0002_0003 | Used      |       1 |  15,087,137,850 |     
   3 |    1,728,000 |       1 |    4 |         1 | RDX       |       1 |        
0 | 2024-02-26 23:10:05 |   194,099 |
  |     159 | ODPVE2RDX_0002_0004 | Used      |       1 |  15,186,796,386 |     
   3 |    1,728,000 |       1 |    5 |         1 | RDX       |       1 |        
0 | 2024-02-29 23:12:01 |   453,415 |
  |     160 | ODPVE2RDX_0002_0005 | Used      |       1 |  15,193,122,988 |     
   3 |    1,728,000 |       1 |    6 |         1 | RDX       |       1 |        
0 | 2024-02-25 23:10:58 |   107,752 |
  |     161 | ODPVE2RDX_0002_0006 | Used      |       1 | 258,282,781,991 |     
  60 |    1,728,000 |       1 |    7 |         1 | RDX       |       1 |        
0 | 2024-02-23 23:36:59 |         0 |
  |     162 | ODPVE2RDX_0002_0007 | Used      |       1 |  15,144,882,105 |     
   3 |    1,728,000 |       1 |    8 |         1 | RDX       |       1 |        
0 | 2024-02-27 23:09:10 |   280,444 |
  
+---------+---------------------+-----------+---------+-----------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+

note the 'inchanger' and 'slot' property.


In this way, alla works as expected BUT if ALL volumes in a 'media' are
unavailable (typically because of some incorrect cycling of 'media'; we
cannot kill every collegues that insert the wrong cartdrige, because will
run quickly out of collegues ;-) bacula try to mount 'available' volumes
even on non-mounted 'media', eg, for volumes without the inchanger=1 property.


Some logs:

  07-Mar 23:00 lnfbacula-dir JobId 18015: There are no more Jobs associated with Volume 
"ODPVE2RDX_0000_0006". Marking it purged.
  07-Mar 23:00 lnfbacula-dir JobId 18015: All records pruned from Volume 
"ODPVE2RDX_0000_0006"; marking it "Purged"
  07-Mar 23:00 lnfbacula-dir JobId 18015: Recycled volume "ODPVE2RDX_0000_0006"
  07-Mar 23:00 lnfbacula-dir JobId 18015: Using Device "RDXStorage1" to write.
  07-Mar 23:03 odpve2-sd JobId 18015: Warning: mount.c:215 Open of File device 
"RDXStorage1" (/var/spool/vchanger/ODPVE2RDX/1) Volume "ODPVE2RDX_0000_0006" 
failed: ERR=file_dev.c:190 Could not open(/var/spool/vchanger/ODPVE2RDX/1,OPEN_READ_WRITE,0640): 
ERR=No such file or directory
  07-Mar 23:03 odpve2-sd JobId 18015: Marking Volume "ODPVE2RDX_0000_0006" in 
Error in Catalog.

This Looks like the volume is marked as being in a slot in the baculacatalog, but the RDX cartridge containing that volume is not actuallymounted. This can happen if a cartridge is removed but an 'update slots'command is never run or else failed due to an error.

Are you sure that the 'update slots' command is being launched inbconsole, both when a cartridge is inserted and when removed? If youconfigured vchanger to do so via udev scripts, then the vchanger logwill show any failures in running bconsole and will log "WARNING!'update slots' needed in bconsole" to the vchanger log file. I suggestsetting LogLevel = LOG_DEBUG in the vchanger config to get details aboutwhat vchanger is sending bconsole when a cartridge is inserted/removed.I know the udev scripts shipped with vchanger work for removable driveslike USB or hot-plug SATA backplane, but I'm not so sure that the RDXdevice is the same from a device driver / udev viewpoint. Maybe udevrules included with vchanger don't work for RDX. I don't know.


so, bacula find a volume in pool that can purge, purge it, and try to mount
it even if it is not 'inchanger', cannot access it, and put the volume in
'Error' state.


I've solve this for now with a simple script that lookup for volumes that
are in Error state with expiresin=0, and (if the related 'media' is mounted)
i reenable purging it.


But is strange...

_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again vchanger and volumes in error...

Reply via email to