Re: Optional exclude list results in STRANGE report
Am 12.07.21 um 19:01 schrieb Debra S Baddorf: Using a global exclude list seems to work out fine, so thanks for pointing that out. Enjoy the weekend! Cheers, Kees Yeah - I realized later that you might have meant “leading dot”. I was actually using ./amanda-exclude-file to indicate that it was at the top of the DLE. But no dot IN the name. I prefer to have my exclude-files on the amanda-server, with the actual config: /etc/amanda/daily/excludes contains root_excludes var_excludes you_name_the_DLE_excludes If I have more than one amanda-config, I either symlink or just refer to the files from the other directory. I do it this way because then I have the full config in one place, ready to tar or commit to git. Just another way of doing it.
Re: Optional exclude list results in STRANGE report
> On Jul 10, 2021, at 2:31 AM, Kees Meijs | Nefos wrote: > > Hi all, > > Thank you for all your answers and support. > > On 09-07-2021 20:27, Jose M Calhariz wrote: >> I am using a .amanda-exclude.list per DLE on the top of it and is not >> broken for me. I can share my settings if it help. >> > > The root cause of the errors generated in my case is the FAT32 filesystem not > supporting leading dots in filenames. > > Using a global exclude list seems to work out fine, so thanks for pointing > that out. > > Enjoy the weekend! > > Cheers, > Kees Yeah - I realized later that you might have meant “leading dot”. I was actually using ./amanda-exclude-file to indicate that it was at the top of the DLE. But no dot IN the name. Sorry I helped to add confusion. Deb Baddorf
Re: Optional exclude list results in STRANGE report
Hi all, Thank you for all your answers and support. On 09-07-2021 20:27, Jose M Calhariz wrote: I am using a .amanda-exclude.list per DLE on the top of it and is not broken for me. I can share my settings if it help. The root cause of the errors generated in my case is the FAT32 filesystem not supporting leading dots in filenames. Using a global exclude list seems to work out fine, so thanks for pointing that out. Enjoy the weekend! Cheers, Kees
Re: Optional exclude list results in STRANGE report
Hi, I am using a .amanda-exclude.list per DLE on the top of it and is not broken for me. I can share my settings if it help. Kind regards Jose M Calhariz On Fri, Jul 09, 2021 at 05:13:19PM +, Debra S Baddorf wrote: > On the other hand: I also used .amanda-exclude-list > at the top of any DLE that needed an exclude list. > > However, they broke a few years ago. I don’t know if that was my > version of Linux or my version of amanda. > > So trying a globalized version is a good idea. > > Deb Baddorf > Fermilab > > > On Jul 9, 2021, at 8:28 AM, Kees Meijs | Nefos wrote: > > > > Hi Charles, > > > > Thank you for your prompt answer. > > > > On 09-07-2021 15:10, Charles Curley wrote: > >> I wonder why you put the excludes file in with the stuff being backed > >> up? I usually put my excludes files in with my amanda files, and refer > >> to them with a fully qualified path: > >> > >> define dumptype comp-root-tar-var { > >> comp-root-tar > >> comment "Root partitions with compression, special for /var" > >> exclude list "/etc/amanda/DailySet1/excludes.var" > >> } > > > > Well that's easy: I was under the impression one must have a excludes file > > per DLE. I was wrong. > > > >> I'm no expert on UEFI, but it strikes me as asking for problems to put > >> anything other than UEFI files in /boot/efi. I vaguely recall that that > >> is a FAT file system, in which case the file name may be a problem. > >> > >> > >> Also, it's rather hard to diagnose what is likely a permissions issue > >> without knowing what the relevant permissions are. > >> > >> root@hawk:/etc/amanda/DailySet1# ll excludes.var > >> -rw-r--r-- 1 backup backup 138 Mar 26 2018 excludes.var > >> root@hawk:/etc/amanda/DailySet1# > > > > Yes that's FAT32 and yes the filename is invalid and therefore a problem. > >> -- Does anybody read signatures any more? > > > > Yes, I just did. :-) > > > > Thanks! Will try with a general exclude file in a different location. > > > > Cheers, > > Kees > > > -- -- Ah se eu fosse homem... Clodovil. signature.asc Description: PGP signature
Re: Optional exclude list results in STRANGE report
On the other hand: I also used .amanda-exclude-list at the top of any DLE that needed an exclude list. However, they broke a few years ago. I don’t know if that was my version of Linux or my version of amanda. So trying a globalized version is a good idea. Deb Baddorf Fermilab > On Jul 9, 2021, at 8:28 AM, Kees Meijs | Nefos wrote: > > Hi Charles, > > Thank you for your prompt answer. > > On 09-07-2021 15:10, Charles Curley wrote: >> I wonder why you put the excludes file in with the stuff being backed >> up? I usually put my excludes files in with my amanda files, and refer >> to them with a fully qualified path: >> >> define dumptype comp-root-tar-var { >> comp-root-tar >> comment "Root partitions with compression, special for /var" >> exclude list "/etc/amanda/DailySet1/excludes.var" >> } > > Well that's easy: I was under the impression one must have a excludes file > per DLE. I was wrong. > >> I'm no expert on UEFI, but it strikes me as asking for problems to put >> anything other than UEFI files in /boot/efi. I vaguely recall that that >> is a FAT file system, in which case the file name may be a problem. >> >> >> Also, it's rather hard to diagnose what is likely a permissions issue >> without knowing what the relevant permissions are. >> >> root@hawk:/etc/amanda/DailySet1# ll excludes.var >> -rw-r--r-- 1 backup backup 138 Mar 26 2018 excludes.var >> root@hawk:/etc/amanda/DailySet1# > > Yes that's FAT32 and yes the filename is invalid and therefore a problem. >> -- Does anybody read signatures any more? > > Yes, I just did. :-) > > Thanks! Will try with a general exclude file in a different location. > > Cheers, > Kees
Re: Optional exclude list results in STRANGE report
Hi Charles, Thank you for your prompt answer. On 09-07-2021 15:10, Charles Curley wrote: I wonder why you put the excludes file in with the stuff being backed up? I usually put my excludes files in with my amanda files, and refer to them with a fully qualified path: define dumptype comp-root-tar-var { comp-root-tar comment "Root partitions with compression, special for /var" exclude list "/etc/amanda/DailySet1/excludes.var" } Well that's easy: I was under the impression one must have a excludes file per DLE. I was wrong. I'm no expert on UEFI, but it strikes me as asking for problems to put anything other than UEFI files in /boot/efi. I vaguely recall that that is a FAT file system, in which case the file name may be a problem. Also, it's rather hard to diagnose what is likely a permissions issue without knowing what the relevant permissions are. root@hawk:/etc/amanda/DailySet1# ll excludes.var -rw-r--r-- 1 backup backup 138 Mar 26 2018 excludes.var root@hawk:/etc/amanda/DailySet1# Yes that's FAT32 and yes the filename is invalid and therefore a problem. -- Does anybody read signatures any more? Yes, I just did. :-) Thanks! Will try with a general exclude file in a different location. Cheers, Kees
Re: Optional exclude list results in STRANGE report
On Fri, 9 Jul 2021 12:05:52 +0200 Kees Meijs | Nefos wrote: > Using Amanda 3.5.1-2+b2 (Debian) I've configured: > > > exclude list optional ".amanda-excludes.gtar" I wonder why you put the excludes file in with the stuff being backed up? I usually put my excludes files in with my amanda files, and refer to them with a fully qualified path: define dumptype comp-root-tar-var { comp-root-tar comment "Root partitions with compression, special for /var" exclude list "/etc/amanda/DailySet1/excludes.var" } I'm no expert on UEFI, but it strikes me as asking for problems to put anything other than UEFI files in /boot/efi. I vaguely recall that that is a FAT file system, in which case the file name may be a problem. Also, it's rather hard to diagnose what is likely a permissions issue without knowing what the relevant permissions are. root@hawk:/etc/amanda/DailySet1# ll excludes.var -rw-r--r-- 1 backup backup 138 Mar 26 2018 excludes.var root@hawk:/etc/amanda/DailySet1# -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Optional exclude list results in STRANGE report
Hi list, In danger of asking something that has been asked before (yes, I did try some search engines)... Using Amanda 3.5.1-2+b2 (Debian) I've configured: exclude list optional ".amanda-excludes.gtar" When making back-ups of UEFI capable hosts the back-up reports back as being STRANGE with the following error: ? Can't open exclude file '/boot/efi/.amanda-excludes.gtar': Permission denied I tried adding a property as following: property append "IGNORE" "open exclude file" But alas, still STRANGE. Is this a problem in terms of semantics (maybe a more proper regular expression is needed) or am I trying to do an impossible thing? Hopefully someone is reading this with a similar problem and (again, hopefully) solution. Thanks in advance! Cheers, Kees -- https://nefos.nl/contact <https://nefos.nl/contact> Nefos IT bv Ambachtsweg 25 (industrienummer 4217) 5627 BZ Eindhoven Nederland KvK 66494931 /Bereikbaar op maandag, dinsdag, woensdag en vrijdag tussen 09:00u en 17:00u./
Re: ZWC and exclude list not behaving as expected
Here's some more info: The reason why ZWC is not excluding the files is because of the way exclude pattern is specified. Following are my test results: ZWC 3.1.1 rev 22762 ZMC 3.1.1 rev 22752 1. Correct exclude pattern (Added using ZMC): * 192.168.15.217 C:/Documents and Settings/Administrator/My Documents/My Music C:/Documents and Settings/Administrator/My Documents/My Music { zmc_windows_base exclude *.mp3 *.avi *.pdf * All files with extension .mp3, .avi .pdf got excluded. 2. Incorrect exclude pattern (Modified manually to match what customer had specified) : * 192.168.15.217 C:/Documents and Settings/Administrator/My Documents/My Music C:/Documents and Settings/Administrator/My Documents/My Music { zmc_windows_base exclude .\\*.mp3 .\\*.avi .\\*.pdf * Nothing got excluded. Please ask the customer to modify the exclude pattern. Regards, Prashant -- Open Source Storage Engineer http://www.zmanda.com
Re: ZWC and exclude list not behaving as expected
On Thu, Jul 1, 2010 at 10:59 AM, Dustin J. Mitchell dus...@zmanda.com wrote: 1. Correct exclude pattern (Added using ZMC): * 192.168.15.217 C:/Documents and Settings/Administrator/My Documents/My Music C:/Documents and Settings/Administrator/My Documents/My Music { zmc_windows_base exclude *.mp3 *.avi *.pdf * All files with extension .mp3, .avi .pdf got excluded. I was able to confirm this using ZWC 3.1.beta (3.1.1 is not available on the download site yet.) So, to help out other poor souls who may be having a very bad day, I've added this to the wiki: http://wiki.zmanda.com/index.php/Zmanda_Windows_Client#Adding_dumptype_definitions_on_the_Amanda_server_for_Windows_clients Also note that using single, forward slashes in the path seems to work fine in 3.1.beta. Apparently it did not in the past. Thanks to Dustin and the others who helped solve this syntax problem. Kind Regards, Chris
ZWC and exclude list not behaving as expected
Hi all, I'm here again with another ZWC-exclude list issue. This paste (http://pastebin.mozilla.org/743992) shows a DLE for a ZWC client and a listing of the resulting dump. This configuration is setup with the sole purpose of testing the ZWC and exclude lists. As you can see, the exclude list has a number of multi-media file extensions in regexp form. The target directory is almost entirely mp3 files. In spite of this exclude list, the ZWC still insists on dumping all of the mp3 files. I have verified via tcpdump that the exclude list is passed to the ZWC. However, setting the log level of ZWC to 5 shows a call to zwcIncludeExcludeList(), but logs nothing to indicate that the client actually received the exclude list. 2304:2112:30/6/2010:13:45:602::Entering zwcIncludeExcludeList() 2304:2112:30/6/2010:13:45:602::Leaving zwcIncludeExcludeList() So... am I missing something that I don't know about or is this a bug? Incidentally, if the source for ZWC were available, I'd be glad to take a look and see if I could fix this. Kind Regards, Chris
Re: ZWC and exclude list not behaving as expected
http://pastebin.mozilla.org/744016 Updated to include a bit more information. On Wed, Jun 30, 2010 at 1:58 PM, Chris Nighswonger cnighswon...@foundations.edu wrote: Hi all, I'm here again with another ZWC-exclude list issue. This paste (http://pastebin.mozilla.org/743992) shows a DLE for a ZWC client and a listing of the resulting dump. This configuration is setup with the sole purpose of testing the ZWC and exclude lists. As you can see, the exclude list has a number of multi-media file extensions in regexp form. The target directory is almost entirely mp3 files. In spite of this exclude list, the ZWC still insists on dumping all of the mp3 files. I have verified via tcpdump that the exclude list is passed to the ZWC. However, setting the log level of ZWC to 5 shows a call to zwcIncludeExcludeList(), but logs nothing to indicate that the client actually received the exclude list. 2304:2112:30/6/2010:13:45:602::Entering zwcIncludeExcludeList() 2304:2112:30/6/2010:13:45:602::Leaving zwcIncludeExcludeList() So... am I missing something that I don't know about or is this a bug? Incidentally, if the source for ZWC were available, I'd be glad to take a look and see if I could fix this. Kind Regards, Chris
Re: ZWC and exclude list not behaving as expected
On Wed, Jun 30, 2010 at 7:21 PM, Dustin J. Mitchell dus...@zmanda.com wrote: On Wed, Jun 30, 2010 at 12:58 PM, Chris Nighswonger cnighswon...@foundations.edu wrote: So... am I missing something that I don't know about or is this a bug? Replying on Paddy's behalf: There's a 3.1.1 that's currently in internal testing that might improve the situation. It should be ready in a few days. Send another email if you don't hear anything about it soon? Ok. I did find some further info in the wiki which indicated that the ZWC requires the full path for each entry in the exclude list. I did verify that this does work. However, it makes for quite a chore from the administrative aspect trying to keep up with a full path to every directory where one wants to exclude a certain file type or types. Other backup solutions which use the MS VSS services allow both exclusion and inclusion based on file extension matching patterns alone. It seems that ZWC could be made to behave this way as well. Kind Regards, Chris
Re: ZWC and exclude list not behaving as expected
On Wed, Jun 30, 2010 at 12:58 PM, Chris Nighswonger cnighswon...@foundations.edu wrote: So... am I missing something that I don't know about or is this a bug? Replying on Paddy's behalf: There's a 3.1.1 that's currently in internal testing that might improve the situation. It should be ready in a few days. Send another email if you don't hear anything about it soon? Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: ZWC and exclude list question (another one)
On Wed, Jun 23, 2010 at 8:22 AM, Chris Nighswonger cnighswon...@foundations.edu wrote: So is the only way to do this to have a dumptype setup using 'exclude' and 'exclude append' to catch each of potentially 300 some multimedia file extensions? This would mean 300 some 'exclude'/'exclude append' directives. It seems there must be a better way. Surely you could get 90% of the space-saving effects with only 10 extensions.. Dustin -- Open Source Storage Engineer http://www.zmanda.com
Re: ZWC and exclude list question (another one)
On Thu, Jun 24, 2010 at 12:34 PM, Dustin J. Mitchell dus...@zmanda.com wrote: On Wed, Jun 23, 2010 at 8:22 AM, Chris Nighswonger cnighswon...@foundations.edu wrote: So is the only way to do this to have a dumptype setup using 'exclude' and 'exclude append' to catch each of potentially 300 some multimedia file extensions? This would mean 300 some 'exclude'/'exclude append' directives. It seems there must be a better way. Surely you could get 90% of the space-saving effects with only 10 extensions.. True. However, there could still quite a bit of administrative overhead to exclude files based on extensions if the list were to become very long. It would be nice to be able to reference an exclude file on the amanda server in such a case as this. Or just have ZWC support exclude files client-side. Kind Regards, Chris
ZWC and exclude list question (another one)
I'm wanting to exclude from my user's Windows clients all multimedia files. I plan to use regexps to match file extensions (ie. ./*.mp3 and so forth). It is my understanding that ZWC does not respect exclude lists stored on the client. So is the only way to do this to have a dumptype setup using 'exclude' and 'exclude append' to catch each of potentially 300 some multimedia file extensions? This would mean 300 some 'exclude'/'exclude append' directives. It seems there must be a better way. Kind Regards, Chris
Re: exclude list not working on clients.
Hi. all of the machines in question are an upgrade to CentOS 5.4 and it seems there was an error in updating the packages and hence confusing PATH etc. When I checked via amadmin version it reported a different version as from yum list installed | grep ama. So I went back to my usual way of handling these type of packages and compiled it myself and all clients and server run the same amanda now 2.6.0p2. The reason why I used that I had a local tar, and I knew this would work (as it did before). With this I wanted to make sure that every possible error is gone and all packages match and pathnames are correct. However a couple of things I like to mention: The problem I described below is gone and it seemed that upon saving and exit a c character sneaked into the pathname of the exclude file which is really hard to see. What surprises me that such an error that is fairly severe is not reported in the email that is send to the admin after a finished backup. Other errors are reported, e.g. missing tape etc or when a file has vanished during backup (e.g. temporary files like firefox' parent.lock). But there was no mention that the exclude file was not found ... Is this intended?? Jobst On Fri, Feb 26, 2010 at 09:32:06PM -0500, Jean-Louis Martineau (martin...@zmanda.com) wrote: They are on the client machine. Jean-Louis Jobst Schmalenbach wrote: Thanks for replying, Jean-Louis. I have trouble finding them for the host AND partitions in question. I can only find the sendbackup.*.debug files for the tape host itself but not the client machines (none of them). They should be in /tmp/amanda but they are not (except on the tape host itslef). Bugger. Need to do some research. Jobst -Original Message- From: Jean-Louis Martineau martin...@zmanda.com To: Jobst Schmalenbach jo...@barrett.com.au Cc: amanda-users@amanda.org Subject: Re: exclude list not working on clients. Date: Fri, 26 Feb 2010 10:03:21 -0500 Jobst, What you do looks good, post the sendbackup.*.debug file. Jean-Louis Jobst Schmalenbach wrote: Hi. I have one client host that refuses to listen to the exclude list option in the dumptype. I know it works, for example I backup a 120GB /amanda directory/parition on the tape host that contains the directory holdingdisk, that is excluded in the dumptype: define dumptype amandadir { global exclude list /usr/local/etc/amanda/DailySet1/amanda_exclude.gtar program GNUTAR } and that file contains one line and one line only: [r...@tapehost /usr/local/etc/amanda/DailySet1] #cat amanda_exclude.gtar ./holdingdisk and that [thankfully ;-) ] is not backed up. So here is what I have (well the important stuff) with regards to the config I have some problems with: On the amanda tape server in amanda.conf: define dumptype sambadir { global exclude list /usr/local/etc/amanda/DailySet1/samba_exclude.gtar program GNUTAR } On the amanda tape server in disklist (amongst other partitions and hosts): ... 192.168.X.Y /var standard 192.168.X.Y /src standard 192.168.X.Y /samba sambadir ... on the client 192.168.X.Y I have a file that is refered to above that contains [r...@192.168.x.y /] #cat /usr/local/etc/amanda/DailySet1/samba_exclude.gtar ./Shares/Backup ./Shares/score_testing ./Shares/SharedBinariesWinNT ./Shares/SharedBinariesWin2000 Those directories are full of stuff that does not need to be backed up, but when I look in the gnutar-list directory of that client: [r...@192.168.x.y /var/lib/amanda/gnutar-lists] #grep ./Shares\/Backup 192.168.X.Y_samba_1 2067 12304757 ./Shares/Backup/Software/DevStudio_SourceCode/VC/SAMPLES/SDK/OLE/INOLE2/CHAP07/PATRON 2067 11026541 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/custom/layout/modules 2067 278714 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa5.4.1/expl/productupdates/shared 2067 246195 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa/locale/ja 2067 10896037 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/temp/k9D2wD/SugarPro-Upgrade-4.0.1-to-4.5.0d/include/ytree/TreeView/css 2067 10354877 ./Shares/Backup/ServerBackup/Documents/2000_11_31/Assessment Services/SPQ 2067 12370086 ./Shares/Backup/WebsServers/VPS_april2006/localicons_b4_update/Activate2/small 2067 12763398 ./Shares/Backup/WebsServers/Moja_aug2008/home/httpd/www.activate.com.au/dynamic/HVP_TP/pdf/ask 2067 11043546 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/modules/Studio/wizards 2067 11405291 ./Shares/Backup/Software/DevStudio_SourceCode/MyProjects/Colin Herbert 2067 10322192 ./Shares/Backup/ServerBackup/Documents/1998_12_31/Z_SystemAdministration/motherboards 2067 82041 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa.org.02_04_08/bin/etc 2067
exclude list not working on clients.
Hi. I have one client host that refuses to listen to the exclude list option in the dumptype. I know it works, for example I backup a 120GB /amanda directory/parition on the tape host that contains the directory holdingdisk, that is excluded in the dumptype: define dumptype amandadir { global exclude list /usr/local/etc/amanda/DailySet1/amanda_exclude.gtar program GNUTAR } and that file contains one line and one line only: [r...@tapehost /usr/local/etc/amanda/DailySet1] #cat amanda_exclude.gtar ./holdingdisk and that [thankfully ;-) ] is not backed up. So here is what I have (well the important stuff) with regards to the config I have some problems with: On the amanda tape server in amanda.conf: define dumptype sambadir { global exclude list /usr/local/etc/amanda/DailySet1/samba_exclude.gtar program GNUTAR } On the amanda tape server in disklist (amongst other partitions and hosts): ... 192.168.X.Y /var standard 192.168.X.Y /src standard 192.168.X.Y /samba sambadir ... on the client 192.168.X.Y I have a file that is refered to above that contains [r...@192.168.x.y /] #cat /usr/local/etc/amanda/DailySet1/samba_exclude.gtar ./Shares/Backup ./Shares/score_testing ./Shares/SharedBinariesWinNT ./Shares/SharedBinariesWin2000 Those directories are full of stuff that does not need to be backed up, but when I look in the gnutar-list directory of that client: [r...@192.168.x.y /var/lib/amanda/gnutar-lists] #grep ./Shares\/Backup 192.168.X.Y_samba_1 2067 12304757 ./Shares/Backup/Software/DevStudio_SourceCode/VC/SAMPLES/SDK/OLE/INOLE2/CHAP07/PATRON 2067 11026541 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/custom/layout/modules 2067 278714 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa5.4.1/expl/productupdates/shared 2067 246195 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa/locale/ja 2067 10896037 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/temp/k9D2wD/SugarPro-Upgrade-4.0.1-to-4.5.0d/include/ytree/TreeView/css 2067 10354877 ./Shares/Backup/ServerBackup/Documents/2000_11_31/Assessment Services/SPQ 2067 12370086 ./Shares/Backup/WebsServers/VPS_april2006/localicons_b4_update/Activate2/small 2067 12763398 ./Shares/Backup/WebsServers/Moja_aug2008/home/httpd/www.activate.com.au/dynamic/HVP_TP/pdf/ask 2067 11043546 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/modules/Studio/wizards 2067 11405291 ./Shares/Backup/Software/DevStudio_SourceCode/MyProjects/Colin Herbert 2067 10322192 ./Shares/Backup/ServerBackup/Documents/1998_12_31/Z_SystemAdministration/motherboards 2067 82041 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa.org.02_04_08/bin/etc 2067 10797122 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/patch/SugarPro-Patch-3.5.1b-restore/include 2067 10715411 ./Shares/Backup/ServerBackup/Documents/2007_and_earlier/Consulting/Proposals/2002/Tender Submissions/NESA Aug02 So I am not sure why it does not work, basically it is 50GB of crap I do not need and is only required for historical purposes. What am I doing wrong? Jobst -- Today is the first day of the rest of the mess. | |0| | Jobst Schmalenbach, jo...@barrett.com.au, General Manager | | |0| Barrett Consulting Group P/L The Meditation Room P/L |0|0|0| +61 3 9532 7677, POBox 277, Caulfield South, 3162, Australia
Re: exclude list not working on clients.
Jobst, What you do looks good, post the sendbackup.*.debug file. Jean-Louis Jobst Schmalenbach wrote: Hi. I have one client host that refuses to listen to the exclude list option in the dumptype. I know it works, for example I backup a 120GB /amanda directory/parition on the tape host that contains the directory holdingdisk, that is excluded in the dumptype: define dumptype amandadir { global exclude list /usr/local/etc/amanda/DailySet1/amanda_exclude.gtar program GNUTAR } and that file contains one line and one line only: [r...@tapehost /usr/local/etc/amanda/DailySet1] #cat amanda_exclude.gtar ./holdingdisk and that [thankfully ;-) ] is not backed up. So here is what I have (well the important stuff) with regards to the config I have some problems with: On the amanda tape server in amanda.conf: define dumptype sambadir { global exclude list /usr/local/etc/amanda/DailySet1/samba_exclude.gtar program GNUTAR } On the amanda tape server in disklist (amongst other partitions and hosts): ... 192.168.X.Y /var standard 192.168.X.Y /src standard 192.168.X.Y /samba sambadir ... on the client 192.168.X.Y I have a file that is refered to above that contains [r...@192.168.x.y /] #cat /usr/local/etc/amanda/DailySet1/samba_exclude.gtar ./Shares/Backup ./Shares/score_testing ./Shares/SharedBinariesWinNT ./Shares/SharedBinariesWin2000 Those directories are full of stuff that does not need to be backed up, but when I look in the gnutar-list directory of that client: [r...@192.168.x.y /var/lib/amanda/gnutar-lists] #grep ./Shares\/Backup 192.168.X.Y_samba_1 2067 12304757 ./Shares/Backup/Software/DevStudio_SourceCode/VC/SAMPLES/SDK/OLE/INOLE2/CHAP07/PATRON 2067 11026541 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/custom/layout/modules 2067 278714 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa5.4.1/expl/productupdates/shared 2067 246195 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa/locale/ja 2067 10896037 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/temp/k9D2wD/SugarPro-Upgrade-4.0.1-to-4.5.0d/include/ytree/TreeView/css 2067 10354877 ./Shares/Backup/ServerBackup/Documents/2000_11_31/Assessment Services/SPQ 2067 12370086 ./Shares/Backup/WebsServers/VPS_april2006/localicons_b4_update/Activate2/small 2067 12763398 ./Shares/Backup/WebsServers/Moja_aug2008/home/httpd/www.activate.com.au/dynamic/HVP_TP/pdf/ask 2067 11043546 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/modules/Studio/wizards 2067 11405291 ./Shares/Backup/Software/DevStudio_SourceCode/MyProjects/Colin Herbert 2067 10322192 ./Shares/Backup/ServerBackup/Documents/1998_12_31/Z_SystemAdministration/motherboards 2067 82041 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa.org.02_04_08/bin/etc 2067 10797122 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/patch/SugarPro-Patch-3.5.1b-restore/include 2067 10715411 ./Shares/Backup/ServerBackup/Documents/2007_and_earlier/Consulting/Proposals/2002/Tender Submissions/NESA Aug02 So I am not sure why it does not work, basically it is 50GB of crap I do not need and is only required for historical purposes. What am I doing wrong? Jobst
Re: exclude list not working on clients.
Thanks for replying, Jean-Louis. I have trouble finding them for the host AND partitions in question. I can only find the sendbackup.*.debug files for the tape host itself but not the client machines (none of them). They should be in /tmp/amanda but they are not (except on the tape host itslef). Bugger. Need to do some research. Jobst -Original Message- From: Jean-Louis Martineau martin...@zmanda.com To: Jobst Schmalenbach jo...@barrett.com.au Cc: amanda-users@amanda.org Subject: Re: exclude list not working on clients. Date: Fri, 26 Feb 2010 10:03:21 -0500 Jobst, What you do looks good, post the sendbackup.*.debug file. Jean-Louis Jobst Schmalenbach wrote: Hi. I have one client host that refuses to listen to the exclude list option in the dumptype. I know it works, for example I backup a 120GB /amanda directory/parition on the tape host that contains the directory holdingdisk, that is excluded in the dumptype: define dumptype amandadir { global exclude list /usr/local/etc/amanda/DailySet1/amanda_exclude.gtar program GNUTAR } and that file contains one line and one line only: [r...@tapehost /usr/local/etc/amanda/DailySet1] #cat amanda_exclude.gtar ./holdingdisk and that [thankfully ;-) ] is not backed up. So here is what I have (well the important stuff) with regards to the config I have some problems with: On the amanda tape server in amanda.conf: define dumptype sambadir { global exclude list /usr/local/etc/amanda/DailySet1/samba_exclude.gtar program GNUTAR } On the amanda tape server in disklist (amongst other partitions and hosts): ... 192.168.X.Y /var standard 192.168.X.Y /src standard 192.168.X.Y /samba sambadir ... on the client 192.168.X.Y I have a file that is refered to above that contains [r...@192.168.x.y /] #cat /usr/local/etc/amanda/DailySet1/samba_exclude.gtar ./Shares/Backup ./Shares/score_testing ./Shares/SharedBinariesWinNT ./Shares/SharedBinariesWin2000 Those directories are full of stuff that does not need to be backed up, but when I look in the gnutar-list directory of that client: [r...@192.168.x.y /var/lib/amanda/gnutar-lists] #grep ./Shares\/Backup 192.168.X.Y_samba_1 2067 12304757 ./Shares/Backup/Software/DevStudio_SourceCode/VC/SAMPLES/SDK/OLE/INOLE2/CHAP07/PATRON 2067 11026541 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/custom/layout/modules 2067 278714 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa5.4.1/expl/productupdates/shared 2067 246195 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa/locale/ja 2067 10896037 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/temp/k9D2wD/SugarPro-Upgrade-4.0.1-to-4.5.0d/include/ytree/TreeView/css 2067 10354877 ./Shares/Backup/ServerBackup/Documents/2000_11_31/Assessment Services/SPQ 2067 12370086 ./Shares/Backup/WebsServers/VPS_april2006/localicons_b4_update/Activate2/small 2067 12763398 ./Shares/Backup/WebsServers/Moja_aug2008/home/httpd/www.activate.com.au/dynamic/HVP_TP/pdf/ask 2067 11043546 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/modules/Studio/wizards 2067 11405291 ./Shares/Backup/Software/DevStudio_SourceCode/MyProjects/Colin Herbert 2067 10322192 ./Shares/Backup/ServerBackup/Documents/1998_12_31/Z_SystemAdministration/motherboards 2067 82041 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa.org.02_04_08/bin/etc 2067 10797122 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/patch/SugarPro-Patch-3.5.1b-restore/include 2067 10715411 ./Shares/Backup/ServerBackup/Documents/2007_and_earlier/Consulting/Proposals/2002/Tender Submissions/NESA Aug02 So I am not sure why it does not work, basically it is 50GB of crap I do not need and is only required for historical purposes. What am I doing wrong? Jobst
Re: exclude list not working on clients.
They are on the client machine. Jean-Louis Jobst Schmalenbach wrote: Thanks for replying, Jean-Louis. I have trouble finding them for the host AND partitions in question. I can only find the sendbackup.*.debug files for the tape host itself but not the client machines (none of them). They should be in /tmp/amanda but they are not (except on the tape host itslef). Bugger. Need to do some research. Jobst -Original Message- From: Jean-Louis Martineau martin...@zmanda.com To: Jobst Schmalenbach jo...@barrett.com.au Cc: amanda-users@amanda.org Subject: Re: exclude list not working on clients. Date: Fri, 26 Feb 2010 10:03:21 -0500 Jobst, What you do looks good, post the sendbackup.*.debug file. Jean-Louis Jobst Schmalenbach wrote: Hi. I have one client host that refuses to listen to the exclude list option in the dumptype. I know it works, for example I backup a 120GB /amanda directory/parition on the tape host that contains the directory holdingdisk, that is excluded in the dumptype: define dumptype amandadir { global exclude list /usr/local/etc/amanda/DailySet1/amanda_exclude.gtar program GNUTAR } and that file contains one line and one line only: [r...@tapehost /usr/local/etc/amanda/DailySet1] #cat amanda_exclude.gtar ./holdingdisk and that [thankfully ;-) ] is not backed up. So here is what I have (well the important stuff) with regards to the config I have some problems with: On the amanda tape server in amanda.conf: define dumptype sambadir { global exclude list /usr/local/etc/amanda/DailySet1/samba_exclude.gtar program GNUTAR } On the amanda tape server in disklist (amongst other partitions and hosts): ... 192.168.X.Y /var standard 192.168.X.Y /src standard 192.168.X.Y /samba sambadir ... on the client 192.168.X.Y I have a file that is refered to above that contains [r...@192.168.x.y /] #cat /usr/local/etc/amanda/DailySet1/samba_exclude.gtar ./Shares/Backup ./Shares/score_testing ./Shares/SharedBinariesWinNT ./Shares/SharedBinariesWin2000 Those directories are full of stuff that does not need to be backed up, but when I look in the gnutar-list directory of that client: [r...@192.168.x.y /var/lib/amanda/gnutar-lists] #grep ./Shares\/Backup 192.168.X.Y_samba_1 2067 12304757 ./Shares/Backup/Software/DevStudio_SourceCode/VC/SAMPLES/SDK/OLE/INOLE2/CHAP07/PATRON 2067 11026541 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/custom/layout/modules 2067 278714 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa5.4.1/expl/productupdates/shared 2067 246195 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa/locale/ja 2067 10896037 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/temp/k9D2wD/SugarPro-Upgrade-4.0.1-to-4.5.0d/include/ytree/TreeView/css 2067 10354877 ./Shares/Backup/ServerBackup/Documents/2000_11_31/Assessment Services/SPQ 2067 12370086 ./Shares/Backup/WebsServers/VPS_april2006/localicons_b4_update/Activate2/small 2067 12763398 ./Shares/Backup/WebsServers/Moja_aug2008/home/httpd/www.activate.com.au/dynamic/HVP_TP/pdf/ask 2067 11043546 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/modules/Studio/wizards 2067 11405291 ./Shares/Backup/Software/DevStudio_SourceCode/MyProjects/Colin Herbert 2067 10322192 ./Shares/Backup/ServerBackup/Documents/1998_12_31/Z_SystemAdministration/motherboards 2067 82041 ./Shares/Backup/WebsServers/Moja_aug2008/home/sympa.org.02_04_08/bin/etc 2067 10797122 ./Shares/Backup/ServerBackup/SomeStuffPre2008/BCG_CRM_4.5.0d/cache/upload/upgrades/patch/SugarPro-Patch-3.5.1b-restore/include 2067 10715411 ./Shares/Backup/ServerBackup/Documents/2007_and_earlier/Consulting/Proposals/2002/Tender Submissions/NESA Aug02 So I am not sure why it does not work, basically it is 50GB of crap I do not need and is only required for historical purposes. What am I doing wrong? Jobst
Re: Exclude list entries
Gee, can we be sure of anything with gnutar ;) I have that small project in my mind that would do a tar cv and a tar cv --exclude-from and compare both output. I will do that when I have a day free :) Olivier
Re: Exclude list entries
Thanks for your replies. I could have phrased my question slightly better. I want to know what the exclude list should look like to make sure I exclude all files with these extensions in any directory/subdirectory contained under the disklist entries: .ora .dbf .dmp .dmp.gz.xx If I understand you correctly, then using the following will only exclude files with those extensions located within top level directories, and not necessarily subdirectories, e.g. ./*.ora Is that correct? Olivier Nicole wrote: No there really is a difference between excluding /foo, ./foo, and foo. As you are backing up ., /foo will not match anything. Of course ./foo will match any foo in the top level directory . foo will match any foo in any directory under . Not so sure: banyanon: /usr/local/bin/tar --version tar (GNU tar) 1.13.25 Copyright (C) 2001 Free Software Foundation, Inc. This program comes with NO WARRANTY, to the extent permitted by law. You may redistribute it under the terms of the GNU General Public License; see the file named COPYING for details. Written by John Gilmore and Jay Fenlason. banyanon: /usr/local/bin/tar cfv /dev/null . | grep jpg ./Images/Picture-2 003.jpg ./Images/Picture-2 005.jpg ./maman.jpg banyanon: cat ../excl ./*.jpg banyanon: /usr/local/bin/tar cfv /dev/null --exclude-from=../excl . | grep jpg banyanon: cat ../excl2 *.jpg banyanon: /usr/local/bin/tar cfv /dev/null --exclude-from=../excl2 . | grep jpg Excl contains ./*.jpg and excl2 contains *.jpg but they both exclude any .jpg file in any level of the hierarchy. OK, funny enough, tar 1.13.19 on another machine gives different results, but still, tar 1.13.25 is recommeneded version isn't it? Olivier -- View this message in context: http://www.nabble.com/Exclude-list-entries-tf1936459.html#a5323781 Sent from the Amanda - Users forum at Nabble.com.
Re: Exclude list entries
I want to know what the exclude list should look like to make sure I exclude all files with these extensions in any directory/subdirectory contained under the disklist entries: .ora .dbf .dmp .dmp.gz.xx It seems that the safest way is: *.ora *.dbf *.dmp *.dmp.gz.* without the initial ./ Olivier
Re: Exclude list entries
On Fri, Jul 14, 2006 at 04:41:37PM +0700, Olivier Nicole wrote: I want to know what the exclude list should look like to make sure I exclude all files with these extensions in any directory/subdirectory contained under the disklist entries: .ora .dbf .dmp .dmp.gz.xx It seems that the safest way is: *.ora *.dbf *.dmp *.dmp.gz.* without the initial ./ And to confirm it will work, put your test patterns in a file, say exclude-pats. Then cd to the directory of your DLE. Run something like: tar -cvf /dev/null --exclude-from exclude-pats 21 | egrep 'ora|dbf|dmp' There should be nothing in the output that matches your list. Might be something like oracle.txt, but that shouldn't be excluded. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Exclude list entries
On Fri, Jul 14, 2006 at 10:54:07AM +0700, Olivier Nicole wrote: No there really is a difference between excluding /foo, ./foo, and foo. As you are backing up ., /foo will not match anything. Of course ./foo will match any foo in the top level directory . foo will match any foo in any directory under . Not so sure: Gee, can we be sure of anything with gnutar ;) banyanon: /usr/local/bin/tar --version tar (GNU tar) 1.13.25 Copyright (C) 2001 Free Software Foundation, Inc. This program comes with NO WARRANTY, to the extent permitted by law. You may redistribute it under the terms of the GNU General Public License; see the file named COPYING for details. Written by John Gilmore and Jay Fenlason. banyanon: /usr/local/bin/tar cfv /dev/null . | grep jpg ./Images/Picture-2 003.jpg ./Images/Picture-2 005.jpg ./maman.jpg banyanon: cat ../excl ./*.jpg banyanon: /usr/local/bin/tar cfv /dev/null --exclude-from=../excl . | grep jpg banyanon: cat ../excl2 *.jpg banyanon: /usr/local/bin/tar cfv /dev/null --exclude-from=../excl2 . | grep jpg Excl contains ./*.jpg and excl2 contains *.jpg but they both exclude any .jpg file in any level of the hierarchy. OK, funny enough, tar 1.13.19 on another machine gives different results, but still, tar 1.13.25 is recommeneded version isn't it? I thought either. Fessing up, my test was with tar 1.15.1 and actually I used --exclude 'pattern' rather than --exclude-from file_of_patterns. Beginning to sound like its try til you get what works for you. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Exclude list entries
Dear all, I've now configured an exclusion list for use with Amanda. At first I used these entries: *.ora *.dbf *.dmp *.dmp.gz* but then I re-read the Amanda documentation, which states: When AMANDA attempts to exclude a file or directory it does so relative to the area being archived. For example if /var is in your disklist and you want to exclude /var/log/somefile, then your exclude file would contain ./log/somefile So I've changed the entries to these: ./*.ora ./*.dbf ./*.dmp ./*.dmp.gz* amcheck config reports no errors or problems with either set of entries. I really just want to confirm whether or not I could have used the entries without ./ and what the difference between the different sets of entries would be. Thanks very much. Joe -- View this message in context: http://www.nabble.com/Exclude-list-entries-tf1936459.html#a5305717 Sent from the Amanda - Users forum at Nabble.com.
Re: Exclude list entries
When AMANDA attempts to exclude a file or directory it does so relative to the area being archived. For example if /var is in your disklist and you want to exclude /var/log/somefile, then your exclude file would contain ./log/somefile I understand that rather as a warning not to use /var in the path I really just want to confirm whether or not I could have used the entries without ./ and what the difference between the different sets of entries would be. I think it does not make any difference (except maybe for path globing?), it will exclude any file ending with the suffix .ora, in any subdirectory that you are backuping. The exclude list is handed as-is to gnu-tar, so one way to make sure it does what you want is to try it with gnu-tar manually. Best regards, Olivier
Re: Exclude list entries
On Fri, Jul 14, 2006 at 09:47:22AM +0700, Olivier Nicole wrote: When AMANDA attempts to exclude a file or directory it does so relative to the area being archived. For example if /var is in your disklist and you want to exclude /var/log/somefile, then your exclude file would contain ./log/somefile I understand that rather as a warning not to use /var in the path I really just want to confirm whether or not I could have used the entries without ./ and what the difference between the different sets of entries would be. I think it does not make any difference (except maybe for path globing?), it will exclude any file ending with the suffix .ora, in any subdirectory that you are backuping. No there really is a difference between excluding /foo, ./foo, and foo. As you are backing up ., /foo will not match anything. ./foo will match any foo in the top level directory . foo will match any foo in any directory under . The exclude list is handed as-is to gnu-tar, so one way to make sure it does what you want is to try it with gnu-tar manually. and it is fast if you do it to /dev/null, as in gtar cvf /dev/null -exclude 'whatever' . -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Exclude list and tar on SUSE OES.
Hello, I've been struggling with this for a while. I have this defined in lots of places now: exclude list /usr/local/amanda/exclude.conf e.g. define dumptype root-tar { global program GNUTAR comment root partitions dumped with tar compress none index exclude list /usr/local/amanda/exclude.conf priority low } but it doesn't create the right command : /bin/tar --create --file /dev/null --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/localhost__0.new --sparse --ignore-failed-read --totals . at the client. Help? Thanks, Owen.
Re: Exclude list and tar on SUSE OES.
You did put the exclude file on the client, didn't you ? http://www.amanda.org/docs/exclude.html#id2533384 Regards, Bert De Ridder PeopleWare NV - Head Office Cdt.Weynsstraat 85 B-2660 Hoboken Tel: +32 3 448.33.38 Fax: +32 3 448.32.66 PeopleWare NV - Branch Office Geel Kleinhoefstraat 5 B-2440 Geel Tel: +32 14 57.00.90 Fax: +32 14 58.13.25 http://www.peopleware.be http://www.mobileware.be Owen Williams [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 19/10/2005 12:21 To amanda-users@amanda.org cc Subject Exclude list and tar on SUSE OES. Hello, I've been struggling with this for a while. I have this defined in lots of places now: exclude list /usr/local/amanda/exclude.conf e.g. define dumptype root-tar { global program GNUTAR comment root partitions dumped with tar compress none index exclude list /usr/local/amanda/exclude.conf priority low } but it doesn't create the right command : /bin/tar --create --file /dev/null --directory / --one-file-system --listed-incremental /var/lib/amanda/gnutar-lists/localhost__0.new --sparse --ignore-failed-read --totals . at the client. Help? Thanks, Owen.
Re: Exclude list and tar on SUSE OES.
Bert, You did put the exclude file on the client, didn't you ? http://www.amanda.org/docs/exclude.html#id2533384 Thanks, but yes. Owen. -- Owen Williams [EMAIL PROTECTED] Work0116 2506349 Home0116 2259109 Mobile 0771 5790631 Senior Computing Officer | Software Engineer Consultant | RedHat Certified Engineer DMU Libraries http://www.blue.dmu.ac.uk/
Re: exclude list optional not working?
Graeme Humphries wrote: On Wed, 2005-07-06 at 18:34 +0200, Stefan G. Weichinger wrote: After reading all that thread I have to ask: Do you all agree with me editing the man-page as Jon suggested? It seems reasonable to me. Edited and committed to the xml-docs-cvs. -- Stefan G. Weichinger AMANDA core team member mailto://[EMAIL PROTECTED] -- oops! linux consulting implementation http://www.oops.co.at --
Re: exclude list optional not working?
Jon LaBadie wrote: On Tue, Jul 05, 2005 at 11:14:19PM +0200, Paul Bijnens wrote: Amanda will not complain - if the exclude file on the client is not there at all In this case amanda can construct a gtar argument list that does not contain the exclude list of a non-existing file. - or if the file is not readable: in that case amanda trusts the suid-root runtar executable so that gnutar can read the contents of the file, never mind the permissions. But Amanda does need to verify if the file is there or not, otherwise gnutar will complain about having handed a nonexisting file in the argument list. Paul, I probably should look it up myself, but I'm feeling lazy. Lazy, but you're correct! I was too lazy to look it up in the source yesterday evening. My mistake. I thought that the various exclude features were generalized so that amanda would make up its own exclude file from the various list and appends. In this way a dle could use a generic list from an exclude file plus some other individual appends. Yes it is like you say. in amanda-2.4.5 sources, client-src/client-util.c, lines 276-277 do the test, and it only tests for ENOENT, not EPERM to suppress error msgs. And the exclude list file building is done as user amanda, without suid root at that time. But even in that case, I believe this should flag an error, as implemented currently, otherwise the user would believe he created an exclude file, while amanda silently ignores it because she cannot read it. In the case which started this, the EPERM is even in the directory above, so that a dumb administrator (or me, 20 years ago too) would change the permissions of the file to be world-readable, and even then amanda could not access it. Adding to the puzzle why the backups don't fit on tape, and fiddling again with the compression settings of the tape, changing the syntax in the exclude files from good to bad again and flooding the mailinglist with problems about exclude lists not working. If I'm not all wet, then does amanda ever just pass on the name of the user supplied file? No, you're right completely, as usual. -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * init 0, kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ... * * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: exclude list optional not working?
On Wed, Jul 06, 2005 at 10:06:07AM +0200, Paul Bijnens wrote: Jon LaBadie wrote: Paul, I probably should look it up myself, but I'm feeling lazy. Lazy, but you're correct! I thought that the various exclude features were generalized so that amanda would make up its own exclude file from the various list and appends. In this way a dle could use a generic list from an exclude file plus some other individual appends. ... . in amanda-2.4.5 sources, client-src/client-util.c, lines 276-277 do the test, and it only tests for ENOENT, not EPERM to suppress error msgs. And the exclude list file building is done as user amanda, without suid root at that time. But even in that case, I believe this should flag an error, as implemented currently, otherwise the user would believe he created an exclude file, while amanda silently ignores it because she cannot read it. Thanks for checking Paul, I completely concur with your belief that the current behavior is appropriate. An optional exclude list file is fine. But it is entirely a different thing for file system permissions to prevent the checking of the existance of the file. Particularly so when those permissions would also make an existing exclude list file non-functional. Seems to me the only thing that needs changing is the amanda.conf man page. Currently it says: ... With exclude list, the string is a file name on the client containing GNU-tar exclude expressions. ... If optional is specified for exclude list, then amcheck will not complain if the file doesn't exist or is not readable. As a suggested alternative this is a little verbose but gets the message across I hope. ... With exclude list, the string is a file name on the client containing GNU-tar exclude expressions. The path to the specified exclude list file must be accessible to to the amanda user and the exclude list file, if present (see description of 'optional' below), must be readable by the amanda user. ... If optional is specified for exclude list, then amcheck will not complain if the file doesn't exist. BTW, is it just amcheck, or amdump as well, that does or does not complain? -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: exclude list optional not working?
On Tue, 2005-07-05 at 22:02 -0400, Gene Heskett wrote: amanda cannot do an suid root if it was built as root, no way around it due to the failure of the suid command if its already owned by root. Certainly, but I don't think the Debian packages were built as root. I just choose to run amdump as root so that I can get into ACL protected directories that I can't touch the permissions on.
Re: exclude list optional not working?
On Wed, 2005-07-06 at 11:01 -0400, Jon LaBadie wrote: BTW, is it just amcheck, or amdump as well, that does or does not complain? In my experience it's only been amcheck that complains, amdump is still happy to do the backups, so this confusion hasn't been critical. ;) -- Graeme Humphries ([EMAIL PROTECTED]) Linux Administrator VCom Inc. (306) 955-7075 ext 485 My views and comments do not necessarily reflect the views of my employer.
Re: exclude list optional not working?
On Wed, Jul 06, 2005 at 09:32:13AM -0600, Graeme Humphries wrote: On Wed, 2005-07-06 at 11:01 -0400, Jon LaBadie wrote: BTW, is it just amcheck, or amdump as well, that does or does not complain? In my experience it's only been amcheck that complains, amdump is still happy to do the backups, so this confusion hasn't been critical. ;) Then even more important to have amcheck complain. There could be an existing, but unreachable or unreadable, exclude list file that based on amdump reports and logs would appear to be functional but in fact was not. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: exclude list optional not working?
Jon LaBadie wrote: Seems to me the only thing that needs changing is the amanda.conf man page. Currently it says: ... With exclude list, the string is a file name on the client containing GNU-tar exclude expressions. ... If optional is specified for exclude list, then amcheck will not complain if the file doesn't exist or is not readable. As a suggested alternative this is a little verbose but gets the message across I hope. ... With exclude list, the string is a file name on the client containing GNU-tar exclude expressions. The path to the specified exclude list file must be accessible to to the amanda user and the exclude list file, if present (see description of 'optional' below), must be readable by the amanda user. ... If optional is specified for exclude list, then amcheck will not complain if the file doesn't exist. After reading all that thread I have to ask: Do you all agree with me editing the man-page as Jon suggested? Stefan. -- Stefan G. Weichinger AMANDA core team member mailto://[EMAIL PROTECTED] -- oops! linux consulting implementation http://www.oops.co.at --
Re: exclude list optional not working?
On Wed, 2005-07-06 at 18:34 +0200, Stefan G. Weichinger wrote: After reading all that thread I have to ask: Do you all agree with me editing the man-page as Jon suggested? It seems reasonable to me.
Re: exclude list optional not working?
On Wednesday 06 July 2005 11:31, Graeme Humphries wrote: On Tue, 2005-07-05 at 22:02 -0400, Gene Heskett wrote: amanda cannot do an suid root if it was built as root, no way around it due to the failure of the suid command if its already owned by root. Certainly, but I don't think the Debian packages were built as root. I just choose to run amdump as root so that I can get into ACL protected directories that I can't touch the permissions on. I think this is going to be problematic, Graeme. But I'll defer to someone who is a bit more cognizant of the actual code. I do know that I cannot run either amcheck or amdump here as root, the exit, complaining about it, is instant or nearly so. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) 99.35% setiathome rank, not too shabby for a WV hillbilly Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
Re: exclude list optional not working?
On Wed, 2005-07-06 at 13:59 -0400, Gene Heskett wrote: I think this is going to be problematic, Graeme. But I'll defer to someone who is a bit more cognizant of the actual code. I do know that I cannot run either amcheck or amdump here as root, the exit, complaining about it, is instant or nearly so. This is the strange thing, though. I'm able to run the binaries from the Debian package just fine as root, or as backup (the amanda user). I'm not having any problems actually backing up with my config, just in terms of getting error messages from amcheck that are somewhat erroneous.
exclude list optional not working?
I've just added exclude lists as per the examples in the online documentation, with the following configuration: exclude list optional .amanda.excludes However, amcheck now complains on every item in the disklist for a *single* host, that it [Can't open exclude file '/filepath' : Permission denied]. I thought optional was supposed to prevent this from being an error condition? And, even stranger, it doesn't seem to generate any similar errors on any of the other client systems, just a single one. Any ideas? -- Graeme Humphries ([EMAIL PROTECTED]) Linux Administrator VCom Inc. (306) 955-7075 ext 485 My views and comments do not necessarily reflect the views of my employer.
Re: exclude list optional not working?
Graeme Humphries wrote: I've just added exclude lists as per the examples in the online documentation, with the following configuration: exclude list optional .amanda.excludes However, amcheck now complains on every item in the disklist for a *single* host, that it [Can't open exclude file '/filepath' : Permission denied]. I thought optional was supposed to prevent this from being an error condition? And, even stranger, it doesn't seem to generate any similar errors on any of the other client systems, just a single one. Any ideas? As you have read the docs, this file should be specified relative to the DLE on the client. I'd suggest setting exclude list optional ./.amanda.excludes instead and check that this file is accessible by the AMANDA-user. The optional means that the existence of the file is optional, not its accessibility due to wrong permissions if existing. If it exists, it has to be accessible. -- Stefan G. Weichinger AMANDA core team member mailto://[EMAIL PROTECTED] -- oops! linux consulting implementation http://www.oops.co.at --
Re: exclude list optional not working?
--On Tuesday, July 05, 2005 09:51:36 -0600 Graeme Humphries [EMAIL PROTECTED] wrote: I've just added exclude lists as per the examples in the online documentation, with the following configuration: exclude list optional .amanda.excludes However, amcheck now complains on every item in the disklist for a *single* host, that it [Can't open exclude file '/filepath' : Permission denied]. I thought optional was supposed to prevent this from being an error condition? And, even stranger, it doesn't seem to generate any similar errors on any of the other client systems, just a single one. Any ideas? amcheck runs as your Amanda user and may not have permissions to the directory where your exclude file lives. amdump is suid root so it will be able to access the file. I consider it a bug but others consider it a feature (fewer suid programs). 'optional' just means it's ok to not be there, but 'permission denied' is an error and Amanda reports it as such. Frank -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: exclude list optional not working?
On Tue, 2005-07-05 at 11:41 -0500, Frank Smith wrote: amcheck runs as your Amanda user and may not have permissions to the directory where your exclude file lives. I *thought* I had amanda running as root on the client, but I may be wrong. It doesn't complain that it can't access any of those shares, which only root should be able to have full access to. 'optional' just means it's ok to not be there, but 'permission denied' is an error and Amanda reports it as such. That's not what the online man page seems to say: If optional is specified for exclude list, then amcheck will not complain if the file doesn't exist or is not readable. The file is not readable (permission denied entering the directory that contains is). Maybe it only checks for permission problems on the exact file, and not on containing directories? Graeme
Re: exclude list optional not working?
On Tue, 2005-07-05 at 18:07 +0200, Stefan G. Weichinger wrote: As you have read the docs, this file should be specified relative to the DLE on the client. Yep. I want it to, like shown in the docs, just look for a .amanda.excludes file in the root of every share I'm backing up. I'd suggest setting exclude list optional ./.amanda.excludes Hmmm, would that really make a difference? Given a disklist entry like /files/share, wouldn't that just cause it to look for /files/share/./.amanda.excludes? The optional means that the existence of the file is optional, not its accessibility due to wrong permissions if existing. If it exists, it has to be accessible. But the file *doesn't* exist in any of these directories yet. I just added the config change, I haven't actually created the file anywhere yet. Graeme
Re: exclude list optional not working?
--On Tuesday, July 05, 2005 10:47:59 -0600 Graeme Humphries [EMAIL PROTECTED] wrote: On Tue, 2005-07-05 at 11:41 -0500, Frank Smith wrote: amcheck runs as your Amanda user and may not have permissions to the directory where your exclude file lives. I *thought* I had amanda running as root on the client, but I may be wrong. It doesn't complain that it can't access any of those shares, which only root should be able to have full access to. You really shouldn't be running Amanda as root, but as a separate user. When you run 'make install' as root it installs the executables that need root access suid root. Then when your backups run it can access everything necessary. 'optional' just means it's ok to not be there, but 'permission denied' is an error and Amanda reports it as such. That's not what the online man page seems to say: If optional is specified for exclude list, then amcheck will not complain if the file doesn't exist or is not readable. The file is not readable (permission denied entering the directory that contains is). Maybe it only checks for permission problems on the exact file, and not on containing directories? In my experience the error occurs if Amanda can't access the directory to see if the file is there. Perhaps the docs need to be rephrased. The error is harmless, runtar can access the exclude file (as root) and will do what you want (assuming your file is correct). Frank Graeme -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: exclude list optional not working?
On Tue, 2005-07-05 at 12:07 -0500, Frank Smith wrote: You really shouldn't be running Amanda as root, but as a separate user. When you run 'make install' as root it installs the executables that need root access suid root. Then when your backups run it can access everything necessary. I've installed via Debian (Ubuntu) package, and the reason I need to run as root is because on our fileserver, giving access to the amanda user in the unix permissions isn't feasible. You can thank Samba and Windows clients that decide to automatically overwrite ACLs for that. ;P In my experience the error occurs if Amanda can't access the directory to see if the file is there. Perhaps the docs need to be rephrased. The error is harmless, runtar can access the exclude file (as root) and will do what you want (assuming your file is correct). I'd rather just have any permission related error ignored if the optional keyword is used. Is there a situation where we *would* want it to hard error out on an optional exclude list if it can't get into the directory the exclude list is supposed to be in? Graeme
Re: exclude list optional not working?
On Tue, Jul 05, 2005 at 11:19:09AM -0600, Graeme Humphries wrote: In my experience the error occurs if Amanda can't access the directory to see if the file is there. Perhaps the docs need to be rephrased. I'd rather just have any permission related error ignored if the optional keyword is used. Is there a situation where we *would* want it to hard error out on an optional exclude list if it can't get into the directory the exclude list is supposed to be in? I'd have to peek at the docs/source to be certain of this, but I haven't done so. While runtar is setuid root, I don't think the pieces that set up the runtar command line are run setuid root. They still need normal amanda user access to the exclude file. If there were no exclude file, then the admin can reasonably feel that it is not contributing to the list of excluded file. Thus no error on setting optional is reasonable. But what about an existing exclude file without access permissions? If that was not an error, how would the admin know the exclude file was not contributing anything to the list of excluded files?? -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: exclude list optional not working?
On Tue, 2005-07-05 at 16:50 -0500, Frank Smith wrote: You can do what I do, just ignore the errors from amcheck that you know are bogus. That's probably what I'll end up doing, but I know that for me it's generally bad practice, because it means that eventually I'll just stop paying attention to the warnings amcheck mails me, and I'll miss something important. ;)
Re: exclude list optional not working?
Title: Re: exclude list optional not working? On Tue, 2005-07-05 at 23:14 +0200, Paul Bijnens wrote: - or if the file is not readable: in that case amanda trusts the suid-root runtar executable so that gnutar can read the contents of the file, never mind the permissions. But Amanda does need to verify if the file is there or not, otherwise gnutar will complain about having handed a nonexisting file in the argument list. Ahhh, I see, it just passes off the file to tar if it can be verified as existing. In my case, it can't verify that the file exists or not, so it chokes. Interesting. I wonder how I'll work around that. :)
Re: exclude list optional not working?
Title: Re: exclude list optional not working? On Tue, 2005-07-05 at 15:34 -0400, Jon LaBadie wrote: If there were no exclude file, then the admin can reasonably feel that it is not contributing to the list of excluded file. Thus no error on setting optional is reasonable. But what about an existing exclude file without access permissions? If that was not an error, how would the admin know the exclude file was not contributing anything to the list of excluded files?? That's a good question, but I don't think it directly affects the issue. The way I'm looking at it is this: the docs claim that optional will ignore errors if it can't read the exclude list file. Regardless of whether or not this is a good design decision, if that's the intended functionality, it should work regardless of if the permission problems are on the file itself or on the enclosing directory (structure). Does that make sense? Graeme
Re: exclude list optional not working?
Graeme Humphries wrote: I'd rather just have any permission related error ignored if the optional keyword is used. Is there a situation where we *would* want it to hard error out on an optional exclude list if it can't get into the directory the exclude list is supposed to be in? Amanda will not complain - if the exclude file on the client is not there at all In this case amanda can construct a gtar argument list that does not contain the exclude list of a non-existing file. - or if the file is not readable: in that case amanda trusts the suid-root runtar executable so that gnutar can read the contents of the file, never mind the permissions. But Amanda does need to verify if the file is there or not, otherwise gnutar will complain about having handed a nonexisting file in the argument list. -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: exclude list optional not working?
On Tuesday 05 July 2005 12:47, Graeme Humphries wrote: On Tue, 2005-07-05 at 11:41 -0500, Frank Smith wrote: amcheck runs as your Amanda user and may not have permissions to the directory where your exclude file lives. I *thought* I had amanda running as root on the client, but I may be wrong. It doesn't complain that it can't access any of those shares, which only root should be able to have full access to. This is beginning to sound as if the install isn't quite right. 1) amanda MUST be built by the user who will run it, in most cases this would be the user 'amanda'. 2) The user amanda MUST be a member of a high ranking group such as 'disk', 'backup', or I've seen one case of 'sys'. 3) amanda MUST be installed by root after being built. 4) the user running amanda must be the one who built it. 5) only when all the above is followed, will all the exec permissions be correct. Amanda will do its own suid to root when it needs those perms. This fails if root built it. If you installed from an rpm, I've seen a couple of broken rpms, and generally speaking, probably well over 80% of us have built it from tarballs. I build every new snapshot of 2.4.5 thats put up within a day or so of its appearance, and run that one till the next one shows up or I get bit have to back up a version. Thats been a very un-common occurance... And finally, to amanda (and tar) there is a huge difference between an .amanda-excludes file and an ./.amanda-excludes file. It takes the leading ./ to anchor it to the current directory. The second . in front of the amanda-excludes file only serves to hide it from a normal directory listing. Now, you can have such a file as an .amanda-excludes, which contains a list of stuff in ./ style to skip, but in my experience with using it, a full path to the excludes file was needed, such as /amanda/.amanda-excludes. 'optional' just means it's ok to not be there, but 'permission denied' is an error and Amanda reports it as such. That's not what the online man page seems to say: If optional is specified for exclude list, then amcheck will not complain if the file doesn't exist or is not readable. The file is not readable (permission denied entering the directory that contains is). Maybe it only checks for permission problems on the exact file, and not on containing directories? Graeme And this says the install isn't quite up to specs, see above. Its also in the top 10 FAQ I believe. I HTH, Graeme. -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) 99.35% setiathome rank, not too shabby for a WV hillbilly Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
Re: exclude list optional not working?
On Tuesday 05 July 2005 13:19, Graeme Humphries wrote: On Tue, 2005-07-05 at 12:07 -0500, Frank Smith wrote: You really shouldn't be running Amanda as root, but as a separate user. When you run 'make install' as root it installs the executables that need root access suid root. Then when your backups run it can access everything necessary. I've installed via Debian (Ubuntu) package, and the reason I need to run as root is because on our fileserver, giving access to the amanda user in the unix permissions isn't feasible. You can thank Samba and Windows clients that decide to automatically overwrite ACLs for that. ;P amanda cannot do an suid root if it was built as root, no way around it due to the failure of the suid command if its already owned by root. In my experience the error occurs if Amanda can't access the directory to see if the file is there. Perhaps the docs need to be rephrased. The error is harmless, runtar can access the exclude file (as root) and will do what you want (assuming your file is correct). I'd rather just have any permission related error ignored if the optional keyword is used. Is there a situation where we *would* want it to hard error out on an optional exclude list if it can't get into the directory the exclude list is supposed to be in? Graeme -- Cheers, Gene There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order. -Ed Howdershelt (Author) 99.35% setiathome rank, not too shabby for a WV hillbilly Yahoo.com and AOL/TW attorneys please note, additions to the above message by Gene Heskett are: Copyright 2005 by Maurice Eugene Heskett, all rights reserved.
Re: exclude list optional not working?
--On Tuesday, July 05, 2005 15:20:15 -0600 Graeme Humphries [EMAIL PROTECTED] wrote: On Tue, 2005-07-05 at 23:14 +0200, Paul Bijnens wrote: - or if the file is not readable: in that case amanda trusts the suid-root runtar executable so that gnutar can read the contents of the file, never mind the permissions. But Amanda does need to verify if the file is there or not, otherwise gnutar will complain about having handed a nonexisting file in the argument list. Ahhh, I see, it just passes off the file to tar if it can be verified as existing. In my case, it can't verify that the file exists or not, so it chokes. Interesting. I wonder how I'll work around that. :) You can do what I do, just ignore the errors from amcheck that you know are bogus. Frank -- Frank Smith [EMAIL PROTECTED] Sr. Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: Exclude list and disklist
On Thu, 21 Apr 2005 at 12:39pm, Kuas wrote I setup the back-up so that all excluded list are in /var/lib/amanda/exclude.txt on each machine, so that I can use a general dump-type. But I haven't been able to get it work. For example, I backed up all the /var/spool/mail for every user (the emails are in 1 file), except: A and B. I put in the exclude.txt on that machine: /var/spool/mail/A and /var/spool/mail/B. This doesn't work, I wonder if they're absolute path. In the dumptype, I put: Exclude paths are relative. So, if your DLE is /var/spool/mail, you'd have to exclude ./A and ./B. The second question is about disklist. If I have to change the directory layout of a system. And change the disklist, I can see that in the next backup, the new list is used. But, all the old list are still in the index (I can see it in amrecover). I think it's good if it keep the old structure (disklist) until all the tapes that contain that data are reused. Is this the case, if not, how do I delete the entries? The same If you change the disklist, amanda treats those as new entries. The old entries won't be backed up anymore, but will still be recoverable until the tapes are overwritten. as exclude list, if some users update me that some directories in their home are not supposed to be backup, but it was backup before. Does the next amdump run redo the dump not to include that directory or only only when it detects more changes in that directory? Would it exclude the list, or I should restart some of the amanda processes? Amanda leaves no processes running between dumps. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: Exclude list and disklist
Joshua Baker-LePain wrote: On Thu, 21 Apr 2005 at 12:39pm, Kuas wrote I setup the back-up so that all excluded list are in /var/lib/amanda/exclude.txt on each machine, so that I can use a general dump-type. But I haven't been able to get it work. For example, I backed up all the /var/spool/mail for every user (the emails are in 1 file), except: A and B. I put in the exclude.txt on that machine: /var/spool/mail/A and /var/spool/mail/B. This doesn't work, I wonder if they're absolute path. In the dumptype, I put: Exclude paths are relative. So, if your DLE is /var/spool/mail, you'd have to exclude ./A and ./B. Got it. Now, In a situation I want to give flexibility to users, that they are the one that knows if a directory needs to be excluded or backup to be more efficient in the backup process. From the howto and some trial I can specify in the dumptype, instead of the absolute path to the exclude file, but just the name of the file: exclude list exclude.list So each user needs to create this file and has full authority to change it. The effect I saw (from amcheck) is that it will try to find that the file in each of the DLE that uses that dumptype. But the problem I saw, when there is a problem like the file doesn't exist. That will stop all backup processes or at least for that DLE. Has anybody else seen this, or it's just normal behavior when there's a problem, they just stop the backup. Is the syntax of that exclude behavior is prohibited. Has anybody tried doing similiar purpose like this before? Would there be a better way to do it? Thanks, Kuas.
Re: Exclude list and disklist
On Thu, 21 Apr 2005 at 3:58pm, Kuas wrote Now, In a situation I want to give flexibility to users, that they are the one that knows if a directory needs to be excluded or backup to be more efficient in the backup process. From the howto and some trial I can specify in the dumptype, instead of the absolute path to the exclude file, but just the name of the file: exclude list exclude.list So each user needs to create this file and has full authority to change it. The effect I saw (from amcheck) is that it will try to find that the file in each of the DLE that uses that dumptype. But the problem I saw, when there is a problem like the file doesn't exist. That will stop all backup processes or at least for that DLE. Has anybody else seen this, or it's just normal behavior when there's a problem, they just stop the backup. Is the syntax of that exclude behavior is prohibited. Has anybody tried doing similiar purpose like this before? Would there be a better way to do it? From 'man amanda': exclude [ list|file ][[optional][ append ][ string ]+] Default: file. There is two exclude list exclude file and exclude list. With exclude file , the string is a gnutar exclude expression. With exclude list , the string is a file name on the client containing gnutar exclude expression. All exclude expression are concatenated in one file and passed to gnutar as a --exclude-from argument. With the append keyword, the string are appended to the current value of the list, without it, the string overwrite the list. =If optional is specified for exclude list, then amcheck will not =complain if the file doesn't exist or is not readable. For exclude list, If the file name is relative, the disk name being backed up is prepended. So if this is entered: exclude list .amanda.excludes the actual file use would be /var/.amanda.excludes for a backup of /var, /usr/local/.amanda.excludes for a backup of /usr/local, and so on. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: Exclude list and disklist
On Thu, Apr 21, 2005 at 03:58:58PM -0400, Kuas enlightened us: Joshua Baker-LePain wrote: On Thu, 21 Apr 2005 at 12:39pm, Kuas wrote I setup the back-up so that all excluded list are in /var/lib/amanda/exclude.txt on each machine, so that I can use a general dump-type. But I haven't been able to get it work. For example, I backed up all the /var/spool/mail for every user (the emails are in 1 file), except: A and B. I put in the exclude.txt on that machine: /var/spool/mail/A and /var/spool/mail/B. This doesn't work, I wonder if they're absolute path. In the dumptype, I put: Exclude paths are relative. So, if your DLE is /var/spool/mail, you'd have to exclude ./A and ./B. Got it. Now, In a situation I want to give flexibility to users, that they are the one that knows if a directory needs to be excluded or backup to be more efficient in the backup process. From the howto and some trial I can specify in the dumptype, instead of the absolute path to the exclude file, but just the name of the file: exclude list exclude.list So each user needs to create this file and has full authority to change it. The effect I saw (from amcheck) is that it will try to find that the file in each of the DLE that uses that dumptype. But the problem I saw, when there is a problem like the file doesn't exist. That will stop all backup processes or at least for that DLE. Has anybody else seen this, or it's just normal behavior when there's a problem, they just stop the backup. Is the syntax of that exclude behavior is prohibited. Has anybody tried doing similiar purpose like this before? Would there be a better way to do it? I use exclude list optional .amanda.exclude So if the file doesn't exist, it isn't an error. Matt -- Matt Hyclak Department of Mathematics Department of Social Work Ohio University (740) 593-1263 pgpL5TSKYGyuZ.pgp Description: PGP signature
Exclude list syntax.
Is it possible to use todays date as element in a filename in an exclude list? -- Regards, Erik P. Olsen
Re: Exclude list syntax.
exclude list syntax depends on your dump/tar/smbtar...usually, no. All you get is wildcards pretty much. --On Tuesday, March 08, 2005 22:09 +0100 Erik P. Olsen [EMAIL PROTECTED] wrote: Is it possible to use todays date as element in a filename in an exclude list? -- Regards, Erik P. Olsen -- GPG/PGP -- 0xE736BD7E 5144 6A2D 977A 6651 DFBE 1462 E351 88B9 E736 BD7E
Re: Exclude list syntax.
On Tue, Mar 08, 2005 at 10:09:04PM +0100, Erik P. Olsen wrote: Is it possible to use todays date as element in a filename in an exclude list? You can have multiple exclude statements, IIRC, at most one can omit the append argument. Given that, you could have one exclude statement read a file created at the start of your amdump run with an appropriate pattern. echo ./###$(date +%Y%m$d)### /path/to/exclude/pattern/of/the/day/file amdump Replace the ###'s with whatever constant or filename pattern you need and the date command with whatever syntax your date string needs demand. Then your dumptype might have something like exclude list append optional /path/to/exclude/pattern/of/the/day/file The path could be absolute or relative to the root of the DLE. jl -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: Exclude list syntax.
On Tue, 2005-03-08 at 17:13 -0500, Jon LaBadie wrote: On Tue, Mar 08, 2005 at 10:09:04PM +0100, Erik P. Olsen wrote: Is it possible to use todays date as element in a filename in an exclude list? You can have multiple exclude statements, IIRC, at most one can omit the append argument. Given that, you could have one exclude statement read a file created at the start of your amdump run with an appropriate pattern. echo ./###$(date +%Y%m$d)### /path/to/exclude/pattern/of/the/day/file amdump Replace the ###'s with whatever constant or filename pattern you need and the date command with whatever syntax your date string needs demand. Then your dumptype might have something like exclude list append optional /path/to/exclude/pattern/of/the/day/file The path could be absolute or relative to the root of the DLE. Beautiful! Thanks a lot. -- Regards, Erik P. Olsen
samba does not support exclude list
Hi I have enabled the exclude list on the amanda.conf file and i am backing up windows folders through samba. My Dumptype is define dumptype root-tar { global program GNUTAR comment root partitions dumped with tar compress none index exclude list /usr/local/etc/amanda/exclude/exclude priority low } Can any one help me in fixing the below error [EMAIL PROTECTED] amanda]$ amcheck DailySet1 Amanda Tape Server Host Check - Holding disk /amhold: 8498704 KB disk space available, that's plenty NOTE: skipping tape-writable test Tape DailySet104 label ok NOTE: info dir /var/lib/amanda/DailySet1/curinfo/indus.mumbai.redhat.com/_home_ghetto_: does not exist NOTE: index dir /var/lib/amanda/DailySet1/index/indus.mumbai.redhat.com/_home_ghetto_: does not exist Server check took 3.681 seconds Amanda Backup Client Hosts Check ERROR: indus.mumbai.redhat.com: [samba does not support exclude list] ERROR: indus.mumbai.redhat.com: [samba does not support exclude list] Client check: 2 hosts checked in 10.164 seconds, 2 problems found (brought to you by Amanda 2.4.4p1) [EMAIL PROTECTED] amanda]$ Would appreciate if anyone can help me -- Regards, Kaushal Shriyan Technical Engineer Red Hat India Pvt. Ltd. Tel : +91-22-22881326/27 Fax : +91-22-22881318 Cell : +91-9820367783
Re: samba does not support exclude list
Hi, amanda told you the problem: samba does not support a exclude list in conjunction with amanda. for backups using samba you can only define a single file to exclude. The used client does not support more than one file to exclude, nor does it support exclude-lists. Christoph Kaushal Shriyan schrieb: Hi I have enabled the exclude list on the amanda.conf file and i am backing up windows folders through samba. My Dumptype is define dumptype root-tar { global program GNUTAR comment root partitions dumped with tar compress none index exclude list /usr/local/etc/amanda/exclude/exclude priority low } Can any one help me in fixing the below error [EMAIL PROTECTED] amanda]$ amcheck DailySet1 Amanda Tape Server Host Check - Holding disk /amhold: 8498704 KB disk space available, that's plenty NOTE: skipping tape-writable test Tape DailySet104 label ok NOTE: info dir /var/lib/amanda/DailySet1/curinfo/indus.mumbai.redhat.com/_home_ghetto_: does not exist NOTE: index dir /var/lib/amanda/DailySet1/index/indus.mumbai.redhat.com/_home_ghetto_: does not exist Server check took 3.681 seconds Amanda Backup Client Hosts Check ERROR: indus.mumbai.redhat.com: [samba does not support exclude list] ERROR: indus.mumbai.redhat.com: [samba does not support exclude list] Client check: 2 hosts checked in 10.164 seconds, 2 problems found (brought to you by Amanda 2.4.4p1) [EMAIL PROTECTED] amanda]$ Would appreciate if anyone can help me
Re: samba does not support exclude list
Kaushal Shriyan wrote: Can any one help me in fixing the below error ... ERROR: indus.mumbai.redhat.com: [samba does not support exclude list] Means you cannot use exclude list with samba. (you may exclude one file, not a list of files). -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: amrecover performance and exclude list
I have two questions for you: 1) any owners of HP SureStore DLT vs80 unit here ? If yes, what kind of performance do you have with amrecover. I did a recover this morning, it took 67 minutes to recover a single 2 bytes file. The backup take 7 minutes so are this unit is really that slow for reading ? I don't have a such a drive, but I should be much faster. The 7 minutes, is that the dump time (to disk) or the time to write to image to tape? How large was the image? How fast is the tape for writing (see amanda reports)? What filenumber was it on tape? Total Full Daily Estimate Time (hrs:min)0:02 Run Time (hrs:min) 0:47 Dump Time (hrs:min)1:00 1:00 0:00 Output Size (meg)5097.6 5097.60.0 Original Size (meg) 7855.1 7855.10.0 Avg Compressed Size (%)64.9 64.9-- Filesystems Dumped3 3 0 Avg Dump Rate (k/s) 1439.7 1439.7-- Tape Time (hrs:min)0:33 0:33 0:00 Tape Size (meg) 5097.6 5097.60.0 Tape Used (%) 13.4 13.40.0 Filesystems Taped 3 3 0 Avg Tp Write Rate (k/s) 2597.2 2597.2-- Is the client a different machine (look for network problems)? Did you specify amrecover_do_fsf true (should be much faster if the image is near the end of tape). Tried that, it's not faster. 2) our backups are done with 'dump' so I backup whole filesystems, but I want to exclude some directories (example: /var/log and /usr). I read Excludes are for gnutar IRC. (Amanda is only a backup scheduler; it uses external programs to do the actual backups. You're limited by the functionality of those programs. Or does dump on your system allow excludes?) Ok, I switched to 'tar' dumps, added this directive to amanda.conf: exclude list /var/lib/amanda/.amanda_excludes And on each client, I created this file. But even if I put stuff in the exclude list (for example: /var/spool), it's still being backed up. This is my dump type: define dumptype hard-disk-compress { global index yes program GNUTAR compress client fast holdingdisk yes priority high exclude list /var/lib/amanda/.amanda_excludes } And a sample from my disklist: x.cesart.local / hard-disk-compress
Re: amrecover performance and exclude list
Hi, youl have to put relative paths in exclude list. example: mount-point/dir-to-backup is: /var file to exclude:/var/tmp/example-file entry in file: ./tmp/example-file Christoph Pascal Robert wrote: I have two questions for you: 1) any owners of HP SureStore DLT vs80 unit here ? If yes, what kind of performance do you have with amrecover. I did a recover this morning, it took 67 minutes to recover a single 2 bytes file. The backup take 7 minutes so are this unit is really that slow for reading ? I don't have a such a drive, but I should be much faster. The 7 minutes, is that the dump time (to disk) or the time to write to image to tape? How large was the image? How fast is the tape for writing (see amanda reports)? What filenumber was it on tape? Total Full Daily Estimate Time (hrs:min)0:02 Run Time (hrs:min) 0:47 Dump Time (hrs:min)1:00 1:00 0:00 Output Size (meg)5097.6 5097.60.0 Original Size (meg) 7855.1 7855.10.0 Avg Compressed Size (%)64.9 64.9-- Filesystems Dumped3 3 0 Avg Dump Rate (k/s) 1439.7 1439.7-- Tape Time (hrs:min)0:33 0:33 0:00 Tape Size (meg) 5097.6 5097.60.0 Tape Used (%) 13.4 13.40.0 Filesystems Taped 3 3 0 Avg Tp Write Rate (k/s) 2597.2 2597.2-- Is the client a different machine (look for network problems)? Did you specify amrecover_do_fsf true (should be much faster if the image is near the end of tape). Tried that, it's not faster. 2) our backups are done with 'dump' so I backup whole filesystems, but I want to exclude some directories (example: /var/log and /usr). I read Excludes are for gnutar IRC. (Amanda is only a backup scheduler; it uses external programs to do the actual backups. You're limited by the functionality of those programs. Or does dump on your system allow excludes?) Ok, I switched to 'tar' dumps, added this directive to amanda.conf: exclude list /var/lib/amanda/.amanda_excludes And on each client, I created this file. But even if I put stuff in the exclude list (for example: /var/spool), it's still being backed up. This is my dump type: define dumptype hard-disk-compress { global index yes program GNUTAR compress client fast holdingdisk yes priority high exclude list /var/lib/amanda/.amanda_excludes } And a sample from my disklist: x.cesart.local / hard-disk-compress
amrecover performance and exclude list
Hi list, I have two questions for you: 1) any owners of HP SureStore DLT vs80 unit here ? If yes, what kind of performance do you have with amrecover. I did a recover this morning, it took 67 minutes to recover a single 2 bytes file. The backup take 7 minutes so are this unit is really that slow for reading ? 2) our backups are done with 'dump' so I backup whole filesystems, but I want to exclude some directories (example: /var/log and /usr). I read that I only have to create a (empty) .amanda_excludes file in the directory I want to exclude and to add this directive: exclude list .amanda_excludes in amanda.conf. But those directories are still being back up. This is my dumptype definition: define dumptype hard-disk-compress { global comment Back up to hard disk instead of tape - using dump holdingdisk no index yes priority high compress client fast exclude list .amanda_excludes } Thanks.
Re: amrecover performance and exclude list
Pascal Robert wrote: I have two questions for you: 1) any owners of HP SureStore DLT vs80 unit here ? If yes, what kind of performance do you have with amrecover. I did a recover this morning, it took 67 minutes to recover a single 2 bytes file. The backup take 7 minutes so are this unit is really that slow for reading ? I don't have a such a drive, but I should be much faster. The 7 minutes, is that the dump time (to disk) or the time to write to image to tape? How large was the image? How fast is the tape for writing (see amanda reports)? What filenumber was it on tape? Is the client a different machine (look for network problems)? Did you specify amrecover_do_fsf true (should be much faster if the image is near the end of tape). 2) our backups are done with 'dump' so I backup whole filesystems, but I want to exclude some directories (example: /var/log and /usr). I read Excludes are for gnutar IRC. (Amanda is only a backup scheduler; it uses external programs to do the actual backups. You're limited by the functionality of those programs. Or does dump on your system allow excludes?) define dumptype hard-disk-compress { global comment Back up to hard disk instead of tape - using dump strange comment, when you first ask if the DLT drive is fast enough :-) holdingdisk no index yes priority high compress client fast exclude list .amanda_excludes } -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Re: amrecover performance and exclude list
Paul Bijnens wrote: Pascal Robert wrote: I have two questions for you: 1) any owners of HP SureStore DLT vs80 unit here ? If yes, what kind of performance do you have with amrecover. I did a recover this morning, it took 67 minutes to recover a single 2 bytes file. The backup take 7 minutes so are this unit is really that slow for reading ? I don't have a such a drive, but I should be much faster. The 7 minutes, is that the dump time (to disk) or the time to write to image to tape? Time to write to tape How large was the image? How fast is the tape for writing (see amanda reports)? What filenumber was it on tape? Avg Dump Rate (k/s) 1697.2-- 1697.2 Avg Tp Write Rate (k/s) 1690.9-- 1690.9 Is the client a different machine (look for network problems)? Yes, but same subnet. No, we didn't have network problems. The tape was busy all the time. Did you specify amrecover_do_fsf true (should be much faster if the image is near the end of tape). I will try that.
exclude list optional broken?
Hallo! The amanda man page states that the exclude list can be optional. So I defined following dumtype: define dumptype my-global { comp-user-tar exclude list optional .amanda.exclude.gtar include list optional .amanda.include.gtar } But unfortunately, with this definition all the backups just fail on _all_ filesystems. Even those which actually contain such an exclude file failed. Is this a known bug or am I just doing something very stupid? Any hints? BTW: amanda-2.4.4, gnutar-1.13.25 -- -- Josef Wolf -- [EMAIL PROTECTED] --
Re: exclude list optional broken?
Josef Wolf wrote: define dumptype my-global { comp-user-tar exclude list optional .amanda.exclude.gtar include list optional .amanda.include.gtar } But unfortunately, with this definition all the backups just fail on _all_ filesystems. Even those which actually contain such an exclude file failed. Is this a known bug or am I just doing something very stupid? Any hints? Before doing complicated, does it work with only an exclude an no include list? I suppose you did put them on the client on the toplevel directory of the DLE? What are the contents of these files? What's in the debug files on the client in /tmp/amanda/*exclude? -- Paul Bijnens, XplanationTel +32 16 397.511 Technologielaan 21 bus 2, B-3001 Leuven, BELGIUMFax +32 16 397.512 http://www.xplanation.com/ email: [EMAIL PROTECTED] *** * I think I've got the hang of it now: exit, ^D, ^C, ^\, ^Z, ^Q, F6, * * quit, ZZ, :q, :q!, M-Z, ^X^C, logoff, logout, close, bye, /bye, * * stop, end, F3, ~., ^]c, +++ ATH, disconnect, halt, abort, hangup, * * PF4, F20, ^X^X, :D::D, KJOB, F14-f-e, F8-e, kill -1 $$, shutdown, * * kill -9 1, Alt-F4, Ctrl-Alt-Del, AltGr-NumLock, Stop-A, ...* * ... Are you sure? ... YES ... Phew ... I'm out * ***
Upgrading to 2.4.4: amcheck complains about samba exclude list
I've upgraded my Amanda version since I can't get a clue about some strange problems at amverify (note that the backup completes regularly and I need to backup ONLY Windows hosts through smbclient). After recreating the previous configuration following the new 2.4.4 syntax and changes I get amcheck complaining about the samba exclude list support. I've never heard this before... but it would be annoying recompiling ALSO Samba from source just to include a feature I don't need. Can anybody explain me what this error means? uranus:/usr/src/amanda-2.4.4# su backup -c /usr/sbin/amcheck ProsaDaily Amanda Tape Server Host Check - Holding disk /usr/local/.holding: 13655828 KB disk space available, that's plenty NOTE: skipping tape-writable test Tape PROSADAILY03 label ok NOTE: info dir /var/amanda/ProsaDaily/curinfo: does not exist NOTE: it will be created on the next run NOTE: index dir /var/amanda/ProsaDaily/index: does not exist Server check took 12.447 seconds Amanda Backup Client Hosts Check ERROR: uranus.prosa.com: [samba does not support exclude list] ERROR: uranus.prosa.com: [samba does not support exclude list] ERROR: uranus.prosa.com: [samba does not support exclude list] ERROR: uranus.prosa.com: [samba does not support exclude list] ERROR: uranus.prosa.com: [samba does not support exclude list] ERROR: uranus.prosa.com: [samba does not support exclude list] Client check: 1 host checked in 1.642 seconds, 6 problems found (brought to you by Amanda 2.4.4) Thank you in advance, Fabio
Re: Upgrading to 2.4.4: amcheck complains about samba exclude list
On Wed, 19 Mar 2003 at 4:22pm, Fabio Corazza wrote Can anybody explain me what this error means? *snip* Amanda Backup Client Hosts Check ERROR: uranus.prosa.com: [samba does not support exclude list] *snip* It means what it says. In your dumptype, you have an exlude list specified. Backups via samba don't support that (see docs/SAMBA). If previous versions didn't complain about it, it's only because the warning was missing, not the condition. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: Upgrading to 2.4.4: amcheck complains about samba exclude list
Joshua Baker-LePain wrote: On Wed, 19 Mar 2003 at 4:22pm, Fabio Corazza wrote Can anybody explain me what this error means? *snip* Amanda Backup Client Hosts Check ERROR: uranus.prosa.com: [samba does not support exclude list] *snip* It means what it says. In your dumptype, you have an exlude list specified. Backups via samba don't support that (see docs/SAMBA). If previous versions didn't complain about it, it's only because the warning was missing, not the condition. Thank you Joshua, I've just commented the line exclude-list from root-tar* to obtain a clean / 0 errors amcheck output. Now I only need to wait for tomorrow's results and see if upgrading has been useful. HTH, Fabio
Re: Easy exclude list question
On Mon, Jan 20, 2003 at 05:08:36PM -0500, Chris Dahn wrote: Does the exclude list go on the server, client, or both? From amanda_src_dir/docs/EXCLUDE: ** Utilize an Exclude List ** An exclude list is a file that resides on the CLIENT machine and contains paths to be excluded, one per line. This file can be in any location on the CLIENT so long as the same path is specified in the dumptype. Some find /usr/local/etc/amanda an appropriate location, but it is up to you. I personally like to have a subdirectory for exclude files but it is up to you where you place this file. jl -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Include and Exclude list
Is it possible to setup in include list like so ./NL-SFS3-02[4-9] ./NL-SFS3-03[0-2] And only include these directories meaning if they aren't any fo these directories don'ttry to back them up. Craig Hancock
Re: pb with large dump exclude list
are you sure of this functionnality, because I tested the 2.4.3 with multiple same disklist entry, like this: 80.65.xxx.yyy /home/test comp-user-tar-incr-adherent_a 80.65.xxx.yyy /home/test comp-user-tar-incr-adherent_b (only excluding rule change in these 2 dumptypes) but I have always this message (duplicate disk record...). I have read the docs, but nothing about this. Jean-Louis Martineau wrote: On Thu, Oct 17, 2002 at 08:00:08PM -0400, Jon LaBadie wrote: On Thu, Oct 17, 2002 at 06:42:13PM -0400, Jean-Louis Martineau wrote: Hello Eric, On Thu, Oct 17, 2002 at 03:59:14PM +0200, Brunet Eric wrote: I tested with a wildcard file path and all worked fine; my idea was to simulate hash dirs with the exclude option: ./[b-zB-Z]* - exclude which takes only subdirs begins by 'B' or 'b' But when I applied these dumptypes on this dir amanda detected that duplicate disk record Upgrade to 2.4.3, it allow multiple disklist entry for the same directory. Does it back it up multiple times? Should it? It follow your include/exclude specification for each entry. Jean-Louis
Re: pb with large dump exclude list (SOLVED)
ok sorry, I found the example in example/disklist it work fine :) howener, I have noticed that server AND client machine must be in 2.4.3 version. Brunet Eric wrote: are you sure of this functionnality, because I tested the 2.4.3 with multiple same disklist entry, like this: 80.65.xxx.yyy /home/test comp-user-tar-incr-adherent_a 80.65.xxx.yyy /home/test comp-user-tar-incr-adherent_b (only excluding rule change in these 2 dumptypes) but I have always this message (duplicate disk record...). I have read the docs, but nothing about this. Jean-Louis Martineau wrote: On Thu, Oct 17, 2002 at 08:00:08PM -0400, Jon LaBadie wrote: On Thu, Oct 17, 2002 at 06:42:13PM -0400, Jean-Louis Martineau wrote: Hello Eric, On Thu, Oct 17, 2002 at 03:59:14PM +0200, Brunet Eric wrote: I tested with a wildcard file path and all worked fine; my idea was to simulate hash dirs with the exclude option: ./[b-zB-Z]* - exclude which takes only subdirs begins by 'B' or 'b' But when I applied these dumptypes on this dir amanda detected that duplicate disk record Upgrade to 2.4.3, it allow multiple disklist entry for the same directory. Does it back it up multiple times? Should it? It follow your include/exclude specification for each entry. Jean-Louis
pb with large dump exclude list
hello, I've been using amanda for one year, but now i have to save a large directory(composed of a lot of subdirs) ( 10GB today). At the beginning all worked fine(full and incremental dump) report example: 80.65.xxx.xxx -e/adherent 0 10150780 9724672 95.8 91:50 1765.0 N/A N/A Now the full dump (using gtar) doesn't finish after 24hours (gtar process continue) i didn't see anything different in the logs (because this dump is not finish) I assume that is not a bandwidth problem (100Mb/s between 2 machines) or a problem of holding disk space. So I have to kill dumper/taper processes in order to prepare the next dump during the following night :( (it generates no space on holding disk in the mail report, this reason is false of course) So I suppose that problem is about size (chunksize is 1GB on the server machine); and I directed my attention to the exclude functionality of gtar. I tested with a wildcard file path and all worked fine; my idea was to simulate hash dirs with the exclude option: ./[b-zB-Z]* - exclude which takes only subdirs begins by 'B' or 'b' But when I applied these dumptypes on this dir amanda detected that duplicate disk record constraint: I can't change this directory into multiple hash dirs. so, Must i patch amanda to not verify duplicates in disklist?? or are there any other solutions??? --technical details: --- -server: amanda 2.4.2p2 (rh 7.1 with gtar 1.13.19, kernel 2.4.19) -client(with the large partition): amanda 2.4.3b2 (FreeBSD 4.5, gtar 1.13.25) regards
Re: pb with large dump exclude list
Hello Eric, On Thu, Oct 17, 2002 at 03:59:14PM +0200, Brunet Eric wrote: I tested with a wildcard file path and all worked fine; my idea was to simulate hash dirs with the exclude option: ./[b-zB-Z]* - exclude which takes only subdirs begins by 'B' or 'b' But when I applied these dumptypes on this dir amanda detected that duplicate disk record Upgrade to 2.4.3, it allow multiple disklist entry for the same directory. Jean-Louis -- Jean-Louis Martineau email: [EMAIL PROTECTED] Departement IRO, Universite de Montreal C.P. 6128, Succ. CENTRE-VILLETel: (514) 343-6111 ext. 3529 Montreal, Canada, H3C 3J7Fax: (514) 343-5834
Re: pb with large dump exclude list
On Thu, Oct 17, 2002 at 06:42:13PM -0400, Jean-Louis Martineau wrote: Hello Eric, On Thu, Oct 17, 2002 at 03:59:14PM +0200, Brunet Eric wrote: I tested with a wildcard file path and all worked fine; my idea was to simulate hash dirs with the exclude option: ./[b-zB-Z]* - exclude which takes only subdirs begins by 'B' or 'b' But when I applied these dumptypes on this dir amanda detected that duplicate disk record Upgrade to 2.4.3, it allow multiple disklist entry for the same directory. Does it back it up multiple times? Should it? -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: pb with large dump exclude list
On Thu, Oct 17, 2002 at 08:00:08PM -0400, Jon LaBadie wrote: On Thu, Oct 17, 2002 at 06:42:13PM -0400, Jean-Louis Martineau wrote: Hello Eric, On Thu, Oct 17, 2002 at 03:59:14PM +0200, Brunet Eric wrote: I tested with a wildcard file path and all worked fine; my idea was to simulate hash dirs with the exclude option: ./[b-zB-Z]* - exclude which takes only subdirs begins by 'B' or 'b' But when I applied these dumptypes on this dir amanda detected that duplicate disk record Upgrade to 2.4.3, it allow multiple disklist entry for the same directory. Does it back it up multiple times? Should it? It follow your include/exclude specification for each entry. Jean-Louis -- Jean-Louis Martineau email: [EMAIL PROTECTED] Departement IRO, Universite de Montreal C.P. 6128, Succ. CENTRE-VILLETel: (514) 343-6111 ext. 3529 Montreal, Canada, H3C 3J7Fax: (514) 343-5834
exclude list problems (PEBKAC?)
I've checked the docs/FAQ/google/archives and I must be missing something really simple. :\ I have an amanda client that I'm backing having problems backing up. We're using tar for /usr on this client: SNIP disklist snippet client /usr no-mirror-tar SNIP Here's the def for no-mirror-tar: SNIP amanda.conf snippet define dumptype no-mirror-tar { global program GNUTAR compress client fast comment usr partitions dumped with tar exclude list .amanda-exclude.gtar priority medium } SNIP The global config only has a comment and 'index yes.' The exclude list (on the client) has this: SNIP $ cat /usr/.amanda-exclude.gtar ./local/mirror ./local/www/logs SNIP The runtar.debug looks like this: SNIP gtar: version 2.4.2 running: /usr/local/bin/tar: gtar --create --directory /usr --listed-incremental /usr/local/var/amanda/gnutar-lists/client_usr_0.new --sparse --one-file-system --ignore-failed-read --totals --file - --exclude-from /usr/.amanda-exclude.gtar . SNIP And the backup of this partition errors out with this: SNIP /-- client /usr lev 0 FAILED [/usr/local/bin/tar returned 2] sendbackup: start [client:/usr level 0] sendbackup: info BACKUP=/usr/local/bin/tar sendbackup: info RECOVER_CMD=/usr/local/bin/gzip -dc |/usr/local/bin/tar -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end ? gtar: Cannot add file ./local/mirror/pub/redhat/i386 . . . | Total bytes written: 8484853760 ? gtar: Error exit delayed from previous errors sendbackup: error [/usr/local/bin/tar returned 2] \ SNIP There's also a 'file changed as we read it' error, but all of these errors come from ./local (/usr/local) in places that are excluded (see above). Suggestions? Thanks, Mike
Re: exclude list problems (PEBKAC?)
--On Friday, October 11, 2002 10:26:58 -0500 pointer [EMAIL PROTECTED] wrote: I've checked the docs/FAQ/google/archives and I must be missing something really simple. :\ I have an amanda client that I'm backing having problems backing up. We're using tar for /usr on this client: SNIP disklist snippet client /usr no-mirror-tar SNIP Here's the def for no-mirror-tar: SNIP amanda.conf snippet define dumptype no-mirror-tar { global program GNUTAR compress client fast comment usr partitions dumped with tar exclude list .amanda-exclude.gtar priority medium } SNIP The global config only has a comment and 'index yes.' The exclude list (on the client) has this: SNIP $ cat /usr/.amanda-exclude.gtar ./local/mirror ./local/www/logs SNIP The runtar.debug looks like this: SNIP gtar: version 2.4.2 running: /usr/local/bin/tar: gtar --create --directory /usr --listed-incremental /usr/local/var/amanda/gnutar-lists/client_usr_0.new --sparse --one-file-system --ignore-failed-read --totals --file - --exclude-from /usr/.amanda-exclude.gtar . SNIP And the backup of this partition errors out with this: SNIP /-- client /usr lev 0 FAILED [/usr/local/bin/tar returned 2] sendbackup: start [client:/usr level 0] sendbackup: info BACKUP=/usr/local/bin/tar sendbackup: info RECOVER_CMD=/usr/local/bin/gzip -dc |/usr/local/bin/tar -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end ? gtar: Cannot add file ./local/mirror/pub/redhat/i386 . . . | Total bytes written: 8484853760 ? gtar: Error exit delayed from previous errors sendbackup: error [/usr/local/bin/tar returned 2] \ SNIP There's also a 'file changed as we read it' error, but all of these errors come from ./local (/usr/local) in places that are excluded (see above). Suggestions? Thanks, Mike Your config looks correct to me. What does '/usr/local/bin/tar --version' show? Can your backup user read the exclude file? Frank -- Frank Smith[EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: exclude list problems (PEBKAC?)
On Fri, Oct 11, 2002 at 10:26:58AM -0500, pointer wrote: I've checked the docs/FAQ/google/archives and I must be missing something really simple. :\ I have an amanda client that I'm backing having problems backing up. We're using tar for /usr on this client: SNIP /-- client /usr lev 0 FAILED [/usr/local/bin/tar returned 2] sendbackup: start [client:/usr level 0] sendbackup: info BACKUP=/usr/local/bin/tar sendbackup: info RECOVER_CMD=/usr/local/bin/gzip -dc |/usr/local/bin/tar -f... - sendbackup: info COMPRESS_SUFFIX=.gz sendbackup: info end ? gtar: Cannot add file ./local/mirror/pub/redhat/i386 . . . | Total bytes written: 8484853760 ? gtar: Error exit delayed from previous errors sendbackup: error [/usr/local/bin/tar returned 2] \ SNIP Even when gnutar ignores failed reads, it exits with an error message and an error status (the returned 2 above). You can compile amanda to consider this exit message and status as normal by placing in config.h a line like #define IGNORE_TAR_ERRORS 1 -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: exclude list problems (PEBKAC?)
Frank, On Fri, 2002-10-11 at 10:50, Frank Smith wrote: Your config looks correct to me. What does '/usr/local/bin/tar --version' show? SNIP $ /usr/local/bin/tar --version tar (GNU tar) 1.13 Copyright (C) 1988, 92,93,94,95,96,97,98, 1999 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by John Gilmore and Jay Fenlason. SNIP Can your backup user read the exclude file? Yes. SNIP $ ls -al /usr/.amanda-exclude.gtar -rw-r--r-- 1 root other 32 Oct 10 10:22 /usr/.amanda-exclude.gtar SNIP Cheers, Mike
Re: exclude list problems (PEBKAC?)
On 11 Oct 2002 at 11:58am, pointer wrote On Fri, 2002-10-11 at 10:50, Frank Smith wrote: Your config looks correct to me. What does '/usr/local/bin/tar --version' show? SNIP $ /usr/local/bin/tar --version tar (GNU tar) 1.13 Bad. Bad bad bad. If you're using indexing, they're broken. amrecover won't work. Run, don't walk, to download 1.13.25 from ftp://alpha.gnu.org. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University
Re: exclude list problems (PEBKAC?)
--On Friday, October 11, 2002 11:58:27 -0500 pointer [EMAIL PROTECTED] wrote: Frank, On Fri, 2002-10-11 at 10:50, Frank Smith wrote: Your config looks correct to me. What does '/usr/local/bin/tar --version' show? SNIP $ /usr/local/bin/tar --version tar (GNU tar) 1.13 Copyright (C) 1988, 92,93,94,95,96,97,98, 1999 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by John Gilmore and Jay Fenlason. SNIP Can your backup user read the exclude file? Yes. SNIP $ ls -al /usr/.amanda-exclude.gtar -rw-r--r-- 1 root other 32 Oct 10 10:22 /usr/.amanda-exclude.gtar SNIP Cheers, Mike I'm stumped on why your excludes aren't working. Possible unrelated problem for you: does anyone on the list remember if tar 1.13 was one of the versions with the bad index file problem (the infamous 'big numbers')? Frank -- Frank Smith[EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: exclude list problems (PEBKAC?)
Joshua, On Fri, 2002-10-11 at 11:50, Joshua Baker-LePain wrote: Bad. Bad bad bad. If you're using indexing, they're broken. amrecover won't work. Run, don't walk, to download 1.13.25 from ftp://alpha.gnu.org. Thanks for the heads-up. I'd seen this, but hadn't gotten around to updating it. Yes, bad, bad, bad! Anyone know when a new version is coming out to fix CAN-2002-0399? Did I misunderstand the vuln announcement, or is this really only exploitable when a superuser extracts files from a tarball without looking at the contents...? SNIP $ tar --version tar (GNU tar) 1.13.25 Copyright (C) 2001 Free Software Foundation, Inc. This program comes with NO WARRANTY, to the extent permitted by law. You may redistribute it under the terms of the GNU General Public License; see the file named COPYING for details. Written by John Gilmore and Jay Fenlason. SNIP Cheers, Mike
Re: exclude list problems (PEBKAC?)
On Friday 11 October 2002 12:58, Frank Smith wrote: --On Friday, October 11, 2002 11:58:27 -0500 pointer [EMAIL PROTECTED] wrote: Frank, On Fri, 2002-10-11 at 10:50, Frank Smith wrote: Your config looks correct to me. What does '/usr/local/bin/tar --version' show? SNIP $ /usr/local/bin/tar --version tar (GNU tar) 1.13 Copyright (C) 1988, 92,93,94,95,96,97,98, 1999 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by John Gilmore and Jay Fenlason. SNIP Can your backup user read the exclude file? Yes. SNIP $ ls -al /usr/.amanda-exclude.gtar -rw-r--r-- 1 root other 32 Oct 10 10:22 /usr/.amanda-exclude.gtar SNIP Cheers, Mike I'm stumped on why your excludes aren't working. Possible unrelated problem for you: does anyone on the list remember if tar 1.13 was one of the versions with the bad index file problem (the infamous 'big numbers')? Frank That was officially fixed with 1.13-19 Frank, it and 1.13-25 both seem to work just fine. What burns me a bit is that the usual tar --version, doesn't eject the minor number, so thery are all 1.13's. One must forcibly remove any tar thats there, and reinstall a known good one to be sure. -- Cheers, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.17% setiathome rank, not too shabby for a WV hillbilly
Re: exclude list problems (PEBKAC?)
On Friday 11 October 2002 14:04, pointer wrote: Joshua, On Fri, 2002-10-11 at 11:50, Joshua Baker-LePain wrote: Bad. Bad bad bad. If you're using indexing, they're broken. amrecover won't work. Run, don't walk, to download 1.13.25 from ftp://alpha.gnu.org. Thanks for the heads-up. I'd seen this, but hadn't gotten around to updating it. Yes, bad, bad, bad! Anyone know when a new version is coming out to fix CAN-2002-0399? Did I misunderstand the vuln announcement, or is this really only exploitable when a superuser extracts files from a tarball without looking at the contents...? Thats the way I read that announcement. SNIP $ tar --version tar (GNU tar) 1.13.25 Copyright (C) 2001 Free Software Foundation, Inc. This program comes with NO WARRANTY, to the extent permitted by law. You may redistribute it under the terms of the GNU General Public License; see the file named COPYING for details. Written by John Gilmore and Jay Fenlason. SNIP this is the good one AFAIK. -- Cheers, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.17% setiathome rank, not too shabby for a WV hillbilly
Re: exclude list problems (PEBKAC?)
--On Friday, October 11, 2002 15:11:00 -0400 Gene Heskett [EMAIL PROTECTED] wrote: On Friday 11 October 2002 12:58, Frank Smith wrote: Possible unrelated problem for you: does anyone on the list remember if tar 1.13 was one of the versions with the bad index file problem (the infamous 'big numbers')? Frank That was officially fixed with 1.13-19 Frank, it and 1.13-25 both seem to work just fine. What burns me a bit is that the usual tar --version, doesn't eject the minor number, so thery are all 1.13's. One must forcibly remove any tar thats there, and reinstall a known good one to be sure. Hmm, on my Debian linux server here I get: # /usr/local/bin/tar --version tar (GNU tar) 1.13.19 and on one of my Solaris boxes: # /usr/local/bin/tar --version tar (GNU tar) 1.13.25 Both were compiled from source, but even on a RedHat 7.2 box I see: #/bin/tar --version tar (GNU tar) 1.13.19 Maybe they started including the minor number in the version string somewhere along the line. Frank -- Frank Smith [EMAIL PROTECTED] Systems Administrator Voice: 512-374-4673 Hoover's Online Fax: 512-374-4501
Re: WangDAT 3400DX DDS2-120m tapetype (was: Re: [amanda 2.4.2p2] tar exclude list not working)
On Wednesday 25 September 2002 01:46, Martin Schwarz wrote: On Sun, Sep 22, 2002 at 10:44:35AM +0200, Martin Schwarz wrote: On Sat, Sep 21, 2002 at 04:00:31PM -0400, Gene Heskett wrote: The above looks as if you have the drives compression turned on, I have, although I should know better about this - having read the list for a while. Somehow I never thought about my own setup when reading about the cons of hardware compression... will switch it off. Hardware compression is now switched off (both with the drive's internal dip switch and the mt utility option), and I have rerun tapetype after making sure the tape's internal information on this was cleared (following Gene's instructions - thanks!). The results are as follows (this is a WangDAT 3400DX SCSI DAT drive with a DDS2 120m tape): martin@lissy:~$ sudo -u backup tapetype -h usage: tapetype -h [-e estsize] [-f tapedev] [-t typename] -hdisplay this message -e estsizeestimated tape size (default: 1g == 1024m) -f tapedevtape device name (default: $TAPE) -t typename tapetype name (default: unknown-tapetype) Note: disable hardware compression when running this program. martin@lissy:~$ time sudo -u backup tapetype -e 4g -t DDS2-120m echo done. wrote 121830 32Kb blocks in 93 files in 11355 seconds (short write) wrote 111350 32Kb blocks in 170 files in 11570 seconds (short write) define tapetype DDS2-120m { comment just produced by tapetype program length 4202 mbytes filemark 4355 kbytes speed 325 kps } real384m37.786s user0m1.980s sys 4m6.670s done. martin@lissy:~$ I'm not sure if this is correct. - The length value looks good. How was that influenced by the estimate of 4GByte I gave with the -e option? I had left out this option the last time I ran tapetype. Having the estimate available means that tapetype can have a go at it in pieces sized somewhat for the tape, rather than doing itty-bitty blocks by the hundreds of thousands on pass one. That makes the run take a bit less time since it doesn't have to play blind man feeling his way along with a cane so to speak. - The filemark seems to be unusually high. Can this be correct? Probably. But I don't think that its size, but rather how far apart they are on the tape. Somebody please correct me if that assumption on my part is wrong. When and if you re-run tapetype again, leave out or change the -e option and see if that effects the filemark value returned. But it will take longer to run without the -e option IIRC. Does this mean I'm wasting over 4 MBytes between two files on tape? Is this something that can be changed or is it a property of the drive? I'm not aware of a method to change this, but someone here might know, in which case they should speak up. - The speed is not that high, but might well be correct. The machine the tape drive is connected to is a lowly Pentium-133... The drive can do about 350-375 on hot rod machines, so its possible the p133 is beginning to effect it somewhat. I certainly wouldn't want to run the compression on that machine as I'd think that a dumptype spec of compress server best would be downright painfull to watch, if you could manage to stay awake. :-) Besides, compress client best also means there are fewer bytes to ship over the network, and several clients can be crunching data simultainiously, both of which will speed things up by large amounts. I will try to rerun the tapetype utility on another tape and see if the results are similar. Thanks for your time and your comments! Martin. -- Cheers, Gene AMD K6-III@500mhz 320M Athlon1600XP@1400mhz 512M 99.16% setiathome rank, not too shabby for a WV hillbilly
WangDAT 3400DX DDS2-120m tapetype (was: Re: [amanda 2.4.2p2] tar exclude list not working)
On Sun, Sep 22, 2002 at 10:44:35AM +0200, Martin Schwarz wrote: On Sat, Sep 21, 2002 at 04:00:31PM -0400, Gene Heskett wrote: The above looks as if you have the drives compression turned on, I have, although I should know better about this - having read the list for a while. Somehow I never thought about my own setup when reading about the cons of hardware compression... will switch it off. Hardware compression is now switched off (both with the drive's internal dip switch and the mt utility option), and I have rerun tapetype after making sure the tape's internal information on this was cleared (following Gene's instructions - thanks!). The results are as follows (this is a WangDAT 3400DX SCSI DAT drive with a DDS2 120m tape): martin@lissy:~$ sudo -u backup tapetype -h usage: tapetype -h [-e estsize] [-f tapedev] [-t typename] -hdisplay this message -e estsizeestimated tape size (default: 1g == 1024m) -f tapedevtape device name (default: $TAPE) -t typename tapetype name (default: unknown-tapetype) Note: disable hardware compression when running this program. martin@lissy:~$ time sudo -u backup tapetype -e 4g -t DDS2-120m echo done. wrote 121830 32Kb blocks in 93 files in 11355 seconds (short write) wrote 111350 32Kb blocks in 170 files in 11570 seconds (short write) define tapetype DDS2-120m { comment just produced by tapetype program length 4202 mbytes filemark 4355 kbytes speed 325 kps } real384m37.786s user0m1.980s sys 4m6.670s done. martin@lissy:~$ I'm not sure if this is correct. - The length value looks good. How was that influenced by the estimate of 4GByte I gave with the -e option? I had left out this option the last time I ran tapetype. - The filemark seems to be unusually high. Can this be correct? Does this mean I'm wasting over 4 MBytes between two files on tape? Is this something that can be changed or is it a property of the drive? - The speed is not that high, but might well be correct. The machine the tape drive is connected to is a lowly Pentium-133... I will try to rerun the tapetype utility on another tape and see if the results are similar. Thanks for your time and your comments! Martin.
Re: [amanda 2.4.2p2] tar exclude list not working
Hello Gene, On Sat, Sep 21, 2002 at 04:00:31PM -0400, Gene Heskett wrote: The above looks as if you have the drives compression turned on, I have, although I should know better about this - having read the list for a while. Somehow I never thought about my own setup when reading about the cons of hardware compression... will switch it off. Back up the list here maybe 2 days, and read the procedure I posted for someone else that will fix this. But I forgot to have him That's Message-Id: [EMAIL PROTECTED] - I will look into it, thanks! Greetings, Martin.