Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread Craig O'Brien
This error shows BackupPC_dump segfault, and pointing to libperl.so How do you install your BackupPC ? From source or from RPM? I did a yum install backuppc, which got it from epel That tells you it was unmounted cleanly last time, not that everything checks out OK. Try it with the -f option

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread Craig O'Brien
And this would explain why the elements are not being linked properly to the pool -- though I would have thought the more likely result would be a duplicate pool entry than an unlinked pool entry... It might be interesting to look for pool chains with the same (uncompressed) content and with

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread Les Mikesell
On Fri, Nov 1, 2013 at 8:48 AM, Craig O'Brien cobr...@fishman.com wrote: This error shows BackupPC_dump segfault, and pointing to libperl.so How do you install your BackupPC ? From source or from RPM? I did a yum install backuppc, which got it from epel Do you see any other segfaults in your

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread Timothy J Massey
Craig O'Brien cobr...@fishman.com wrote on 11/01/2013 09:48:23 AM: This error shows BackupPC_dump segfault, and pointing to libperl.so How do you install your BackupPC ? From source or from RPM? I did a yum install backuppc, which got it from epel That's how I do it. That tells you it

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread backuppc
Craig O'Brien wrote at about 10:11:07 -0400 on Friday, November 1, 2013: And this would explain why the elements are not being linked properly to the pool -- though I would have thought the more likely result would be a duplicate pool entry than an unlinked pool entry... It might be

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread Holger Parplies
Hi, I get some diagnostics when reading this with 'use warnings wrong_numbers' ... backu...@kosowsky.org wrote on 2013-11-01 12:18:17 -0400 [Re: [BackupPC-users] Disk space used far higher than reported pool?size]: Craig O'Brien wrote at about 10:11:07 -0400 on Friday, November 1, 2013

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread backuppc
Holger Parplies wrote at about 18:57:05 +0100 on Friday, November 1, 2013: Hi, I get some diagnostics when reading this with 'use warnings wrong_numbers' ... backu...@kosowsky.org wrote on 2013-11-01 12:18:17 -0400 [Re: [BackupPC-users] Disk space used far higher than reported

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread backuppc
Holger Parplies wrote at about 18:57:05 +0100 on Friday, November 1, 2013: 3.) finding unnecessary duplicates can have a normal explanation: if at some point you had more than 31999 copies of one file (content) in your backups, BackupPC would have created a pool duplicate. Some of

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-11-01 Thread Les Mikesell
On Fri, Nov 1, 2013 at 1:46 PM, backu...@kosowsky.org wrote: This is probably not his *primary* issue since the pool is (only) ~3T. But when he started talking about file read errors, I was concerned that if the pool file reads were being truncated, then there likely would be pool duplicates

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Craig O'Brien
for all your help! Regards, Craig On Wed, Oct 30, 2013 at 10:24 PM, Holger Parplies wb...@parplies.de wrote: Hi, Adam Goryachev wrote on 2013-10-31 09:04:48 +1100 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: On 31/10/13 07:51, Holger Parplies wrote

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Les Mikesell
On Thu, Oct 31, 2013 at 7:49 AM, Craig O'Brien cobr...@fishman.com wrote: Unable to read 8388608 bytes from /var/lib/BackupPC//pc/myfileserver/new//ffileshare/RStmp got=0, What is the underlying storage here - nfs? -- Les Mikesell lesmikes...@gmail.com

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Craig O'Brien
What is the underlying storage here - nfs? Local SATA disks in a RAID 5 (5 disks, 3TB each in capacity) Regards, Craig On Thu, Oct 31, 2013 at 10:37 AM, Les Mikesell lesmikes...@gmail.comwrote: On Thu, Oct 31, 2013 at 7:49 AM, Craig O'Brien cobr...@fishman.com wrote: Unable to read

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Les Mikesell
On Thu, Oct 31, 2013 at 9:47 AM, Craig O'Brien cobr...@fishman.com wrote: What is the underlying storage here - nfs? Local SATA disks in a RAID 5 (5 disks, 3TB each in capacity) I think I'd force an fsck just on general principles even though it will take a long time to complete. Google

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Timothy J Massey
Craig O'Brien cobr...@fishman.com wrote on 10/31/2013 08:49:15 AM: The du -hs /backup/pool /backup/cpool /backup/pc/* has finished. Basically I had 1 host that was taking up 6.9 TB of data with 2.8 TB in the cpool directory and most of the other hosts averaging a GB each. Well, there's your

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Timothy J Massey
Holger Parplies wb...@parplies.de wrote on 10/30/2013 10:24:05 PM: as I understand it, the backups from before the change from smb to rsyncd are linked into the pool. Since the change, some or all are not. Whether the change of XferMethod has anything to do with the problem or whether it

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Marcel Meckel
Hi, Example: ls -l /var/lib lrwxrwxrwx. 1 rootroot 22 Apr 22 2013 BackupPC - /data/BackupPC/TopDir/ mount /dev/sda1 on /data type ext4 (rw) out of curiosity - why don't you just configure /data/BackupPC/TopDir in config.pl as the TopDir?

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Les Mikesell
On Thu, Oct 31, 2013 at 11:36 AM, Marcel Meckel mailinglist+backuppc-us...@foobar0815.de wrote: Hi, Example: ls -l /var/lib lrwxrwxrwx. 1 rootroot 22 Apr 22 2013 BackupPC - /data/BackupPC/TopDir/ mount /dev/sda1 on /data type ext4 (rw) out of

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Craig O'Brien
Just out of curiosity, why hadn't you already done that?!? I didn't know which host was the problem and didn't think of it. Although I'll readily admit it seems painfully obvious to me now. :) The big question is, though, why they aren't linking. I'd really start at the bottom of the stack

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Sharuzzaman Ahmat Raslan
On Fri, Nov 1, 2013 at 1:33 AM, Craig O'Brien cobr...@fishman.com wrote: messages-20131006:Sep 30 13:53:24 servername kernel: BackupPC_dump[15365]: segfault at a80 ip 00310f695002 sp 7fff438c9770 error 4 in libperl.so[310f60+162000] This error shows BackupPC_dump segfault, and

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Les Mikesell
On Thu, Oct 31, 2013 at 12:33 PM, Craig O'Brien cobr...@fishman.com wrote: fsck the filesystem. bash-4.1$ fsck /dev/sda1 fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) /dev/sda1: clean, 20074506/2929688576 files, 2775975889/2929686016 blocks bash-4.1$ That tells you it was

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Timothy J Massey
Craig O'Brien cobr...@fishman.com wrote on 10/31/2013 01:33:30 PM: Just out of curiosity, why hadn't you already done that?!? I didn't know which host was the problem and didn't think of it. Although I'll readily admit it seems painfully obvious to me now. :) Just so you're sufficiently

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Timothy J Massey
Les Mikesell lesmikes...@gmail.com wrote on 10/31/2013 01:54:24 PM: On Thu, Oct 31, 2013 at 12:33 PM, Craig O'Brien cobr...@fishman.com wrote: fsck the filesystem. bash-4.1$ fsck /dev/sda1 fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) /dev/sda1: clean,

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread backuppc
Timothy J Massey wrote at about 11:52:35 -0400 on Thursday, October 31, 2013: Craig O'Brien cobr...@fishman.com wrote on 10/31/2013 08:49:15 AM: Just out of curiosity, why hadn't you already done that?!? Unable to read 8388608 bytes from /var/lib/BackupPC//pc/

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread backuppc
Craig O'Brien wrote at about 08:49:15 -0400 on Thursday, October 31, 2013: The du -hs /backup/pool /backup/cpool /backup/pc/* has finished. Basically I had 1 host that was taking up 6.9 TB of data with 2.8 TB in the cpool directory and most of the other hosts averaging a GB each. The 1

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Les Mikesell
On Thu, Oct 31, 2013 at 12:51 PM, Sharuzzaman Ahmat Raslan sharuzza...@gmail.com wrote: On Fri, Nov 1, 2013 at 1:33 AM, Craig O'Brien cobr...@fishman.com wrote: messages-20131006:Sep 30 13:53:24 servername kernel: BackupPC_dump[15365]: segfault at a80 ip 00310f695002 sp 7fff438c9770

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread backuppc
Les Mikesell wrote at about 10:15:42 -0500 on Thursday, October 31, 2013: On Thu, Oct 31, 2013 at 9:47 AM, Craig O'Brien cobr...@fishman.com wrote: What is the underlying storage here - nfs? Local SATA disks in a RAID 5 (5 disks, 3TB each in capacity) I think I'd force an fsck just

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Holger Parplies
Hi, I've spent far too long writing an email and trying to make it make sense and then discarding it again. Just one thought I want to rescue: the RStmp file was really *large* (something like 1.5 GB), your backup trees are really *large* (1.4 TB), your pool FS is really *full* (27.5 GB free).

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Les Mikesell
On Thu, Oct 31, 2013 at 2:20 PM, Holger Parplies wb...@parplies.de wrote: That doesn't explain your situation, but it still might be something to think about (and we might be seeing one problem on top of and as result of another). I agree with Jeffrey - an Unable to read ... error *without* a

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-31 Thread Sharuzzaman Ahmat Raslan
In my experience, segfault in libraries usually caused by installing it from different source. For example, when I install BackupPC for CentOS, I use the one in EPEL repo. I make sure that all the libraries (perl and others), only come from CentOS base repo, and not from other, as installing

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Timothy J Massey
Craig O'Brien cobr...@fishman.com wrote on 10/29/2013 08:21:11 PM: I'm not sure how I can go about determining if a particular backup is using the pool or just storing the files in the PC folder. What's the best way to check if a given backup set is represented in the pool or not? Would

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Adam Goryachev
On 31/10/13 00:04, Timothy J Massey wrote: The only other thing that I can think of is that you did something wrong with archiving and accidentally archived data somewhere within the BackupPC tree. In my case, I archive to a removable hard drive and sometimes the drive is not mounted when the

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Timothy J Massey
Adam Goryachev mailingli...@websitemanagers.com.au wrote on 10/30/2013 09:18:59 AM: Not really relevant to this thread, but I have in the past added a empty file to each of the removable drives, then test if the file exists before creating the archives. If the drive isn't mounted, the file

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Tyler J. Wagner
On 2013-10-29 17:53, Craig O'Brien wrote: On the General Server Information page, it says Pool is 2922.42GB comprising 6061942 files and 4369 directories, but our pool file system which contains nothing but backuppc and is 11 TB in size is 100% full. Did you forget to exclude the path to

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Craig O'Brien
I'm fairly sure: du -sm /backup/pool /backup/cpool /backup/pc/* It should count all the data under pool and cpool, and there should be minimal space used for the pc folders (because it counts the space for the first time the inode is seen) I'm trying that now. I'll report back when it

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Holger Parplies
Hi, I'll reply here, because I think the issue is visible here. Craig O'Brien wrote on 2013-10-29 15:30:46 -0400 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: The topdir is /var/lib/BackupPC which is a link to /backup I believe that may be your problem. I'm

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Les Mikesell
On Wed, Oct 30, 2013 at 10:12 AM, Holger Parplies wb...@parplies.de wrote: Craig O'Brien wrote on 2013-10-29 15:30:46 -0400 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: The topdir is /var/lib/BackupPC which is a link to /backup I believe that may be your

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Holger Parplies
Just to add two things ... Holger Parplies wrote on 2013-10-30 16:12:02 +0100 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: [...] Like Tim, I also wouldn't bet my life on it, but I'm fairly sure you'll find large amounts of BackupPC_link got error -4 when calling

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Les Mikesell
On Wed, Oct 30, 2013 at 10:48 AM, Holger Parplies wb...@parplies.de wrote: Also note that at least for *rsync backups* files will be hardlinked to identical copies in previous backups even if pooling isn't working. Since you have just written that you are, in fact, using rsync, I should add

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Holger Parplies
Hi, Les Mikesell wrote on 2013-10-30 11:28:26 -0500 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: On Wed, Oct 30, 2013 at 10:48 AM, Holger Parplies wb...@parplies.de wrote: Also note that at least for *rsync backups* files will be hardlinked to identical copies

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Adam Goryachev
On 31/10/13 07:51, Holger Parplies wrote: Yes, as it's basically an extension of start off fresh with the addition of keep old history around in parallel. The notable thing is that you need to *make sure* you have eliminated the problem for there to be any point in starting over. Aside from

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread backuppc
Holger Parplies wrote at about 16:48:11 +0100 on Wednesday, October 30, 2013: Jeffrey, I think we need a script to check pooling? My (still unfinished) BackupPC_copyPool can generate a (huge) list of files, which can be sort(1)ed by inode number. Parsing that should easily reveal anything

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-30 Thread Holger Parplies
Hi, Adam Goryachev wrote on 2013-10-31 09:04:48 +1100 [Re: [BackupPC-users] Disk space used far higher than reported pool size]: On 31/10/13 07:51, Holger Parplies wrote: [...] Aside from that, I would think it might be worth the effort of determining whether all hosts are affected

[BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Craig O'Brien
On the General Server Information page, it says Pool is 2922.42GB comprising 6061942 files and 4369 directories, but our pool file system which contains nothing but backuppc and is 11 TB in size is 100% full. I'm confused how this happened and even ran the BackupPC_nightly script by hand which

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Marcel Meckel
Hi, On the General Server Information page, it says Pool is 2922.42GB comprising 6061942 files and 4369 directories, but our pool file system which contains nothing but backuppc and is 11 TB in size is 100% full. some details would be great! It's a bit hard to guess your setup details...

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Craig O'Brien
File Type is ext4. bash-4.1$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg_harp-lv_root 51615740 10132616 40958900 20% / tmpfs 4019640 828 4018812 1% /dev/shm /dev/md127p1495844135596

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Timothy J Massey
Craig O'Brien cobr...@fishman.com wrote on 10/29/2013 01:53:31 PM: On the General Server Information page, it says Pool is 2922.42GB comprising 6061942 files and 4369 directories, but our pool file system which contains nothing but backuppc and is 11 TB in size is 100% full. My strong

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Craig O'Brien
The topdir is /var/lib/BackupPC which is a link to /backup If I do an ls -l /var/lib I get a bunch of other directories as well as: lrwxrwxrwx. 1 rootroot 7 Dec 17 2011 BackupPC - /backup bash-4.1$ ls -l /backup total 20 drwxr-x---. 18 backuppc root 4096 Oct 25 21:01 cpool

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Marcel Meckel
I've deleted computers inside of the pc directory that I no longer needed to backup. From my understanding that combined with removing the pc from the /etc/BackupPC/hosts file would free up any space those backups used to use in the pool. No. You'll have to wait until the next

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread backuppc
Craig O'Brien wrote at about 13:53:31 -0400 on Tuesday, October 29, 2013: On the General Server Information page, it says Pool is 2922.42GB comprising 6061942 files and 4369 directories, but our pool file system which contains nothing but backuppc and is 11 TB in size is 100% full. I'm

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Timothy J Massey
Craig O'Brien cobr...@fishman.com wrote on 10/29/2013 03:30:46 PM: The topdir is /var/lib/BackupPC which is a link to /backup I missed that in your previous e-mail. Stupid proportional fonts... (And you might want add a -h for commands like du and df: the -h is for human-readable... When

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Les Mikesell
On Tue, Oct 29, 2013 at 4:30 PM, Timothy J Massey tmas...@obscorp.com wrote: Check lost+found and trash while you're at it and see what's in there. They should both be empty. I'm with Jeff: I think that you have multiple PC trees that are not part of the pool. How you managed that I'm

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread backuppc
Les Mikesell wrote at about 16:51:12 -0500 on Tuesday, October 29, 2013: On Tue, Oct 29, 2013 at 4:30 PM, Timothy J Massey tmas...@obscorp.com wrote: Check lost+found and trash while you're at it and see what's in there. They should both be empty. I'm with Jeff: I think

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Craig O'Brien
The folder /backup is the root of the disk. I mounted the disk there, doing the ls -l /backup showed all the root folders on the disk. Perhaps there is something going on with the PC folders, as the lost+found and trash folders are both empty. I'm not sure how I can go about determining if a

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Adam Goryachev
On 30/10/13 11:21, Craig O'Brien wrote: The folder /backup is the root of the disk. I mounted the disk there, doing the ls -l /backup showed all the root folders on the disk. Perhaps there is something going on with the PC folders, as the lost+found and trash folders are both empty. I'm

Re: [BackupPC-users] Disk space used far higher than reported pool size

2013-10-29 Thread Sharuzzaman Ahmat Raslan
Have you removed some PC from the backup list? If you have, the folder to that PC is still available in /backup/pc/pc name . You have to remove the folder manually. I believe that will cause high disk usage, as it is not linking to the pool. Note at the bottom of Edit Hosts: To delete a host,