Re: [Bacula-users] spool disk filesystem, checksums
On 12/05/2014 02:11 PM, Daniel Pocock wrote: On 05/12/14 13:57, Josh Fisher wrote: On 12/5/2014 1:03 AM, Daniel Pocock wrote: On 05/12/14 00:43, Cejka Rudolf wrote: Daniel Pocock wrote (2014/12/04): On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled Hard drives use error correction/detection codes, so single bit error without any error indication is unlikely. Especially in case of spool disks, where datas are read shortly after write. Unfortunately, that is completely untrue. Disks and IO subsystems do not provide any guarantees that they will return the exact data that was written. That is why modern filesystems have checksums. The actual corruption of data often occurs in the writing phase, so whether you read back the sector in 5 minutes or 5 weeks, it will always come back with some bit changed. I've seen this more than once unfortunately. A write error is not detected without a read. There is no way to know, even at the hardware level, whether a particular area of the disk has the correct magnetization / charge without a subsequent read. Then there are RAM buffers and controllers in between the FS and the disk platter / MLC cell. When the FS detects a checksum error, it really has no way to know whether it was due to an incorrect area of disk or an incorrect bit of RAM, but it knows that it didn't read back what should have been written. Without the hardware error detection, the FS may detect false positives, while without the FS checksum there is no way to detect false negatives. Both hardware level and FS level error detection are required. That is why I think the ZFS claim of not needing any special hardware is a bit misleading, or at least depends on the definition of special hardware. Should this go in the bug tracker then? A feature request for Bacula to assume the spool disk filesystem may not be using checksums and therefore Bacula should checksum content on the spool disk itself when handing it off to tape? If you look at the statistics for what is a performance problem in the SD, you will find that it spends a very large amount of time in the crc32 algorithm to create block checksums. Were it an SHA1 or SHA-256 algorithm it would be even more expensive. Thus I am not too enthusiastic to start adding checksums to the spool writing/reading code. If you need or want this kind of checking, it seems to me much simpler to use a checksumming filesystem such as ZFS or btrfs -- that the performance hit may be much less than if the code were done in Bacula because the kernel is more multithread, and more importantly, it will allow Bacula developers to concentrate on adding new features. Best regards, Kern -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
Hello, 2014-12-05 20:01 GMT+01:00 Daniel Pocock dan...@pocock.pro: On 05/12/14 19:20, Dimitri Maziuk wrote: On 12/05/2014 07:48 AM, Daniel Pocock wrote: The purpose of such a checksum wouldn't be to prove that the tapes are correct, it would simply be to catch any failure of the spool disk at the earliest opportunity Monitoring your hardware is not bacula's job. Appropriate tools for this particular job include smartctl, fsck, and checksumming filesystems. Quite a number of applications keep checksums within their own file formats. gzip is one of them, while tar is not. A tgz file is therefore more secure than a tar file alone. The same Bacula! Compare apples to apples and pears to pears! Bacula HAS a block level checksum in the archive volume format like gzip. It does not have a temporary files checksumming like any other application (I suppose, I did not check all other). best regards -- Radosław Korzeniewski rados...@korzeniewski.net -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 06/12/14 00:02, Cejka Rudolf wrote: Daniel Pocock wrote (2014/12/05): Now it is in the bug tracker: http://bugs.bacula.org/view.php?id=2111 ... Data corruption occurs on disks sometimes, e.g. the write to disk didn't record data correctly on the platter and any attempt to read it returns the corrupted data. This may just be 1 bit and it may only occur once in millions of writes but it is known to happen. ... You wrote just about disks first, so I responded just about disks too. After this, you responded with disks and IO subsystems. And now in bug tracker, you again write just about disks. Please: 1) Read atleast Error rates and handling in http://en.wikipedia.org/wiki/Hard_disk_drive 2) Be clear on what problem do you have 3) Think about RAM at first - in PCs without ECC, in servers with just very simple ECC, much weaker than used in disks and on the tapes 4) ... and what about the caches, all storages and all communication paths along the path? What all we know about them? Almost nothing? Checking just the spool disks (with all related subsystems needed to check these) is very narrow view of such a complex thing I'm not trying to avoid those issues - I just used a very simple description of the problem in the bug/feature request details. I included a link to the mailing list thread so that anybody looking in Mantis can see everything that has been discussed. -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 12/5/2014 1:03 AM, Daniel Pocock wrote: On 05/12/14 00:43, Cejka Rudolf wrote: Daniel Pocock wrote (2014/12/04): On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled Hard drives use error correction/detection codes, so single bit error without any error indication is unlikely. Especially in case of spool disks, where datas are read shortly after write. Unfortunately, that is completely untrue. Disks and IO subsystems do not provide any guarantees that they will return the exact data that was written. That is why modern filesystems have checksums. The actual corruption of data often occurs in the writing phase, so whether you read back the sector in 5 minutes or 5 weeks, it will always come back with some bit changed. I've seen this more than once unfortunately. A write error is not detected without a read. There is no way to know, even at the hardware level, whether a particular area of the disk has the correct magnetization / charge without a subsequent read. Then there are RAM buffers and controllers in between the FS and the disk platter / MLC cell. When the FS detects a checksum error, it really has no way to know whether it was due to an incorrect area of disk or an incorrect bit of RAM, but it knows that it didn't read back what should have been written. Without the hardware error detection, the FS may detect false positives, while without the FS checksum there is no way to detect false negatives. Both hardware level and FS level error detection are required. That is why I think the ZFS claim of not needing any special hardware is a bit misleading, or at least depends on the definition of special hardware. This looks at some of the issues; https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 05/12/14 13:57, Josh Fisher wrote: On 12/5/2014 1:03 AM, Daniel Pocock wrote: On 05/12/14 00:43, Cejka Rudolf wrote: Daniel Pocock wrote (2014/12/04): On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled Hard drives use error correction/detection codes, so single bit error without any error indication is unlikely. Especially in case of spool disks, where datas are read shortly after write. Unfortunately, that is completely untrue. Disks and IO subsystems do not provide any guarantees that they will return the exact data that was written. That is why modern filesystems have checksums. The actual corruption of data often occurs in the writing phase, so whether you read back the sector in 5 minutes or 5 weeks, it will always come back with some bit changed. I've seen this more than once unfortunately. A write error is not detected without a read. There is no way to know, even at the hardware level, whether a particular area of the disk has the correct magnetization / charge without a subsequent read. Then there are RAM buffers and controllers in between the FS and the disk platter / MLC cell. When the FS detects a checksum error, it really has no way to know whether it was due to an incorrect area of disk or an incorrect bit of RAM, but it knows that it didn't read back what should have been written. Without the hardware error detection, the FS may detect false positives, while without the FS checksum there is no way to detect false negatives. Both hardware level and FS level error detection are required. That is why I think the ZFS claim of not needing any special hardware is a bit misleading, or at least depends on the definition of special hardware. Should this go in the bug tracker then? A feature request for Bacula to assume the spool disk filesystem may not be using checksums and therefore Bacula should checksum content on the spool disk itself when handing it off to tape? -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 12/5/2014 8:11 AM, Daniel Pocock wrote: On 05/12/14 13:57, Josh Fisher wrote: On 12/5/2014 1:03 AM, Daniel Pocock wrote: On 05/12/14 00:43, Cejka Rudolf wrote: Daniel Pocock wrote (2014/12/04): On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled Hard drives use error correction/detection codes, so single bit error without any error indication is unlikely. Especially in case of spool disks, where datas are read shortly after write. Unfortunately, that is completely untrue. Disks and IO subsystems do not provide any guarantees that they will return the exact data that was written. That is why modern filesystems have checksums. The actual corruption of data often occurs in the writing phase, so whether you read back the sector in 5 minutes or 5 weeks, it will always come back with some bit changed. I've seen this more than once unfortunately. A write error is not detected without a read. There is no way to know, even at the hardware level, whether a particular area of the disk has the correct magnetization / charge without a subsequent read. Then there are RAM buffers and controllers in between the FS and the disk platter / MLC cell. When the FS detects a checksum error, it really has no way to know whether it was due to an incorrect area of disk or an incorrect bit of RAM, but it knows that it didn't read back what should have been written. Without the hardware error detection, the FS may detect false positives, while without the FS checksum there is no way to detect false negatives. Both hardware level and FS level error detection are required. That is why I think the ZFS claim of not needing any special hardware is a bit misleading, or at least depends on the definition of special hardware. Should this go in the bug tracker then? A feature request for Bacula to assume the spool disk filesystem may not be using checksums and therefore Bacula should checksum content on the spool disk itself when handing it off to tape? Well, that would be a feature request, rather than a bug, but I don't think that is needed. The only way to be sure a tape is correct is to read it. Bacula already has verify jobs that read the tape and check it against the client's file checksums. -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 05/12/14 14:25, Josh Fisher wrote: On 12/5/2014 8:11 AM, Daniel Pocock wrote: On 05/12/14 13:57, Josh Fisher wrote: On 12/5/2014 1:03 AM, Daniel Pocock wrote: On 05/12/14 00:43, Cejka Rudolf wrote: Daniel Pocock wrote (2014/12/04): On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled Hard drives use error correction/detection codes, so single bit error without any error indication is unlikely. Especially in case of spool disks, where datas are read shortly after write. Unfortunately, that is completely untrue. Disks and IO subsystems do not provide any guarantees that they will return the exact data that was written. That is why modern filesystems have checksums. The actual corruption of data often occurs in the writing phase, so whether you read back the sector in 5 minutes or 5 weeks, it will always come back with some bit changed. I've seen this more than once unfortunately. A write error is not detected without a read. There is no way to know, even at the hardware level, whether a particular area of the disk has the correct magnetization / charge without a subsequent read. Then there are RAM buffers and controllers in between the FS and the disk platter / MLC cell. When the FS detects a checksum error, it really has no way to know whether it was due to an incorrect area of disk or an incorrect bit of RAM, but it knows that it didn't read back what should have been written. Without the hardware error detection, the FS may detect false positives, while without the FS checksum there is no way to detect false negatives. Both hardware level and FS level error detection are required. That is why I think the ZFS claim of not needing any special hardware is a bit misleading, or at least depends on the definition of special hardware. Should this go in the bug tracker then? A feature request for Bacula to assume the spool disk filesystem may not be using checksums and therefore Bacula should checksum content on the spool disk itself when handing it off to tape? Well, that would be a feature request, rather than a bug, but I don't think that is needed. The only way to be sure a tape is correct is to read it. Bacula already has verify jobs that read the tape and check it against the client's file checksums. The purpose of such a checksum wouldn't be to prove that the tapes are correct, it would simply be to catch any failure of the spool disk at the earliest opportunity If it is only discovered when verifying the tape then it won't be clear where the corruption originated (e.g. the spool disk, the tape or something else) -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 05/12/14 14:48, Daniel Pocock wrote: On 05/12/14 14:25, Josh Fisher wrote: On 12/5/2014 8:11 AM, Daniel Pocock wrote: On 05/12/14 13:57, Josh Fisher wrote: On 12/5/2014 1:03 AM, Daniel Pocock wrote: On 05/12/14 00:43, Cejka Rudolf wrote: Daniel Pocock wrote (2014/12/04): On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled Hard drives use error correction/detection codes, so single bit error without any error indication is unlikely. Especially in case of spool disks, where datas are read shortly after write. Unfortunately, that is completely untrue. Disks and IO subsystems do not provide any guarantees that they will return the exact data that was written. That is why modern filesystems have checksums. The actual corruption of data often occurs in the writing phase, so whether you read back the sector in 5 minutes or 5 weeks, it will always come back with some bit changed. I've seen this more than once unfortunately. A write error is not detected without a read. There is no way to know, even at the hardware level, whether a particular area of the disk has the correct magnetization / charge without a subsequent read. Then there are RAM buffers and controllers in between the FS and the disk platter / MLC cell. When the FS detects a checksum error, it really has no way to know whether it was due to an incorrect area of disk or an incorrect bit of RAM, but it knows that it didn't read back what should have been written. Without the hardware error detection, the FS may detect false positives, while without the FS checksum there is no way to detect false negatives. Both hardware level and FS level error detection are required. That is why I think the ZFS claim of not needing any special hardware is a bit misleading, or at least depends on the definition of special hardware. Should this go in the bug tracker then? A feature request for Bacula to assume the spool disk filesystem may not be using checksums and therefore Bacula should checksum content on the spool disk itself when handing it off to tape? Well, that would be a feature request, rather than a bug, but I don't think that is needed. The only way to be sure a tape is correct is to read it. Bacula already has verify jobs that read the tape and check it against the client's file checksums. The purpose of such a checksum wouldn't be to prove that the tapes are correct, it would simply be to catch any failure of the spool disk at the earliest opportunity If it is only discovered when verifying the tape then it won't be clear where the corruption originated (e.g. the spool disk, the tape or something else) Now it is in the bug tracker: http://bugs.bacula.org/view.php?id=2111 -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 12/05/2014 07:48 AM, Daniel Pocock wrote: The purpose of such a checksum wouldn't be to prove that the tapes are correct, it would simply be to catch any failure of the spool disk at the earliest opportunity Monitoring your hardware is not bacula's job. Appropriate tools for this particular job include smartctl, fsck, and checksumming filesystems. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 05/12/14 19:20, Dimitri Maziuk wrote: On 12/05/2014 07:48 AM, Daniel Pocock wrote: The purpose of such a checksum wouldn't be to prove that the tapes are correct, it would simply be to catch any failure of the spool disk at the earliest opportunity Monitoring your hardware is not bacula's job. Appropriate tools for this particular job include smartctl, fsck, and checksumming filesystems. Quite a number of applications keep checksums within their own file formats. gzip is one of them, while tar is not. A tgz file is therefore more secure than a tar file alone. smartctl and fsck will not always know if bits are written incorrectly within the data of a file. fsck will only notice if the corruption has occurred in the filesystem metadata but not in the file content. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iQIcBAEBCAAGBQJUggEFAAoJEGxlgOd711bEP+4P/2hna4NByzIwrl3zJWkS+QZQ KFcT3YmbpfQdUdpPyOqSPVaph9VCN29dGz5dBwRhrMsX2mmTlw/1b3QwatV4kmMz 135lnNq+Ylb7/IRqqzqSy5S7aftIgWIie7X+wFru0Us70WfaxB6r+Ne1qHoKWZbR aK4FhNDkP/9GZwHEbwrWMP0z2lw79h0uf5VACfA4hJ7fW3fAlc35jcccEsy/QpMJ T6dww9sug0LJR5jxdcMUiu7k9OO5ulryyZsbSfJjlycrkBQeeUWR9HP5SBpz1LBi nqxw+6xalBUS7dQ9nCxx+jrQlID0aGWLFrMLRm6pqeT0Y22kCA9HzsYxs9o7sfC/ mC5KwJzOHWqiIBRs8xBWeH8lRqthGd3COum1orsy1PGMFkh0z3biWPbNW6G1dU3q 3/9+6/5uY8iZyvuFoO5AX7EqezXBuftFm2uv9aKkvVEeG7wJQ3qAGrVOI1oFdAOU ypcMMe7wUgZtvIEyO3aZu5AryiNf5ll/WGyOVbJ6LVNNI0obb7dtAA0j5ziqBBjA 7AHenMC/nUtYL9HyjdR/spbjNDU3PD2J0XzIW+7pDvmerYM6U8pET9BiUQDoZcFy oIfEiFCsYyh0ZGvS61/hqoTcXzOgwreSu9E1vL0sTnXtBuWTvc7FQFiD4JEq52yl PAAmSsIVY7AKXdq1zOFV =bgit -END PGP SIGNATURE- -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 12/05/2014 01:01 PM, Daniel Pocock wrote: smartctl and fsck will not always know if bits are written incorrectly within the data of a file. Buy better disks, then. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu signature.asc Description: OpenPGP digital signature -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
Daniel Pocock wrote (2014/12/05): Now it is in the bug tracker: http://bugs.bacula.org/view.php?id=2111 ... Data corruption occurs on disks sometimes, e.g. the write to disk didn't record data correctly on the platter and any attempt to read it returns the corrupted data. This may just be 1 bit and it may only occur once in millions of writes but it is known to happen. ... You wrote just about disks first, so I responded just about disks too. After this, you responded with disks and IO subsystems. And now in bug tracker, you again write just about disks. Please: 1) Read atleast Error rates and handling in http://en.wikipedia.org/wiki/Hard_disk_drive 2) Be clear on what problem do you have 3) Think about RAM at first - in PCs without ECC, in servers with just very simple ECC, much weaker than used in disks and on the tapes 4) ... and what about the caches, all storages and all communication paths along the path? What all we know about them? Almost nothing? Checking just the spool disks (with all related subsystems needed to check these) is very narrow view of such a complex thing -- Rudolf Cejka cejkar at fit.vutbr.cz http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. Best regards, Kern -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
Daniel Pocock wrote (2014/12/04): On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled Hard drives use error correction/detection codes, so single bit error without any error indication is unlikely. Especially in case of spool disks, where datas are read shortly after write. -- Rudolf Cejka cejkar at fit.vutbr.cz http://www.fit.vutbr.cz/~cejkar Brno University of Technology, Faculty of Information Technology Bozetechova 2, 612 66 Brno, Czech Republic -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] spool disk filesystem, checksums
On 05/12/14 00:43, Cejka Rudolf wrote: Daniel Pocock wrote (2014/12/04): On 04/12/14 18:35, Kern Sibbald wrote: On 12/03/2014 08:49 PM, Daniel Pocock wrote: Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? Unless you are running ZFS for the spool disk, the error will only be noticed when the data is read from the tape. In that case, it sounds like a good idea to use ZFS or Btrfs with checksums enabled Hard drives use error correction/detection codes, so single bit error without any error indication is unlikely. Especially in case of spool disks, where datas are read shortly after write. Unfortunately, that is completely untrue. Disks and IO subsystems do not provide any guarantees that they will return the exact data that was written. That is why modern filesystems have checksums. The actual corruption of data often occurs in the writing phase, so whether you read back the sector in 5 minutes or 5 weeks, it will always come back with some bit changed. I've seen this more than once unfortunately. This looks at some of the issues; https://blogs.oracle.com/bonwick/entry/zfs_end_to_end_data -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] spool disk filesystem, checksums
Does Bacula checksum content on the spool disk before sending it to tape? To be more explicit, if there is a single bit error on the spool disk, will it be noticed before going onto tape or would it only be noticed in future when a file is taken off the tape? -- Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151iu=/4140/ostg.clktrk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
