Re: BTRFS on production: NVR 16+ IP Cameras
On Thu, Nov 15, 2018 at 10:39 AM Juan Alberto Cirez wrote: > > Is BTRFS mature enough to be deployed on a production system to underpin > the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)? > > Based on our limited experience with BTRFS (1+ year) under the above > scenario the answer seems to be no; but I wanted you ask the community > at large for their experience before making a final decision to hold off > on deploying BTRFS on production systems. > > Let us be clear: We think BTRFS has great potential, and as it matures > we will continue to watch its progress, so that at some future point we > can return to using it. > > The issue has been the myriad of problems we have encountered when > deploying BTRFS as the storage fs for the NVR/VMS in cases were the > camera count exceeds 10: Corrupted file systems, sudden read-only file > system, re-balance kernel panics, broken partitions, etc. Performance problems are separate from reliability problems. No matter what, there shouldn't be corruptions or failures when your process is writing through the Btrfs kernel driver. Period. So you've either got significant hardware/firmware problems as the root cause, or your use case is exposing Btrfs bugs. But you're burdened with providing sufficient details about the hardware and storage stack configuration including kernel, btrfs-progs versions and mkfs options and mount options being used. Without a way for a developer to reproduce your problem it's unlikely the source of the problem can be discovered and fixed. > So, again, the question is: is BTRFS mature enough to be used in such > use case and if so, what approach can be used to mitigate such issues. What format are the cameras writing out in? It matters if this is a continuous appending format, or if it's writing them out as individual JPEG files, one per frame, or whatever. What rate, what size, and any other concurrent operations, etc. -- Chris Murphy
Re: BTRFS on production: NVR 16+ IP Cameras
On 11/16/2018 03:51 AM, Nikolay Borisov wrote: On 15.11.18 г. 20:39 ч., Juan Alberto Cirez wrote: Is BTRFS mature enough to be deployed on a production system to underpin the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)? Based on our limited experience with BTRFS (1+ year) under the above scenario the answer seems to be no; but I wanted you ask the community at large for their experience before making a final decision to hold off on deploying BTRFS on production systems. Let us be clear: We think BTRFS has great potential, and as it matures we will continue to watch its progress, so that at some future point we can return to using it. The issue has been the myriad of problems we have encountered when deploying BTRFS as the storage fs for the NVR/VMS in cases were the camera count exceeds 10: Corrupted file systems, sudden read-only file system, re-balance kernel panics, broken partitions, etc. What version of the file system, how many of those issues (if any) have you reported to the upstream mailing list? I don't remember seeing any reports from you. One specific case saw the btrfs drive pool mounted under the /var partition so that upon installation the btrfs pool contained all files under /var; /lib/mysql as well as the video storage. Needless to say this was a catastrophe... BTRFS is not suitable for random workloads, such as databases for example. In those cases it's recommended, at the very least, to disable COW on the database files. Note, disabling CoW doesn't preclude you from creating snapshots, it just means that not every 4k write (for example) will result in a CoW operation. At the other end of the spectrum is a use case where ONLY the video storage was on the btrfs pool; but in that case, the btrfs pool became read-only suddenly and would not re-mount as rw despite all the recovery trick btrfs-tools could throw at it. This, of course prevented the NVR/VMS from recording any footage. Again, you give absolutely no information about how you have configured the filesystem or versions of the software used. So, again, the question is: is BTRFS mature enough to be used in such use case and if so, what approach can be used to mitigate such issues. And.. What is the blocksize used by the application to write into btrfs? And what type of write IO? async, directio or fsync? Thanks, Anand Thank you all for your assistance -Ryan-
Re: BTRFS on production: NVR 16+ IP Cameras
On 2018-11-15 13:39, Juan Alberto Cirez wrote: Is BTRFS mature enough to be deployed on a production system to underpin the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)? For NVR, I'd say no. BTRFS does pretty horribly with append-only workloads, even if they are WORM style. It also does a really bad job with most relational database systems that you would likely use for indexing. If you can suggest your reasoning for wanting to use BTRFS though, I can probably point you at alternatives that would work more reliably for your use case.
Re: BTRFS on production: NVR 16+ IP Cameras
On Thu, 15 Nov 2018 11:39:58 -0700 Juan Alberto Cirez wrote: > Is BTRFS mature enough to be deployed on a production system to underpin > the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)? What are you looking to gain from using Btrfs on an NVR system? It doesn't sound like any of its prime-time features -- such as snapshots, checksumming, compression or reflink copying -- are a must for a bulk video recording scenario. And even if you don't need or use those features, you still pay the price for having them, Btrfs can be 2x to 10x slower than its simpler competitors: https://www.phoronix.com/scan.php?page=article=linux418-nvme-raid=2 If you just meant to use its multi-device support instead of a separate RAID layer, IMO that can't be considered prime-time or depended upon in production (yes, not even RAID 1 or 10). -- With respect, Roman
Re: BTRFS on production: NVR 16+ IP Cameras
On 15.11.18 г. 20:39 ч., Juan Alberto Cirez wrote: > Is BTRFS mature enough to be deployed on a production system to underpin > the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)? > > Based on our limited experience with BTRFS (1+ year) under the above > scenario the answer seems to be no; but I wanted you ask the community > at large for their experience before making a final decision to hold off > on deploying BTRFS on production systems. > > Let us be clear: We think BTRFS has great potential, and as it matures > we will continue to watch its progress, so that at some future point we > can return to using it. > > The issue has been the myriad of problems we have encountered when > deploying BTRFS as the storage fs for the NVR/VMS in cases were the > camera count exceeds 10: Corrupted file systems, sudden read-only file > system, re-balance kernel panics, broken partitions, etc. What version of the file system, how many of those issues (if any) have you reported to the upstream mailing list? I don't remember seeing any reports from you. > > One specific case saw the btrfs drive pool mounted under the /var > partition so that upon installation the btrfs pool contained all files > under /var; /lib/mysql as well as the video storage. Needless to say > this was a catastrophe... BTRFS is not suitable for random workloads, such as databases for example. In those cases it's recommended, at the very least, to disable COW on the database files. Note, disabling CoW doesn't preclude you from creating snapshots, it just means that not every 4k write (for example) will result in a CoW operation. > > At the other end of the spectrum is a use case where ONLY the video > storage was on the btrfs pool; but in that case, the btrfs pool became > read-only suddenly and would not re-mount as rw despite all the recovery > trick btrfs-tools could throw at it. This, of course prevented the > NVR/VMS from recording any footage. Again, you give absolutely no information about how you have configured the filesystem or versions of the software used. > > So, again, the question is: is BTRFS mature enough to be used in such > use case and if so, what approach can be used to mitigate such issues. > > > Thank you all for your assistance > > -Ryan-
BTRFS on production: NVR 16+ IP Cameras
Is BTRFS mature enough to be deployed on a production system to underpin the storage layer of a 16+ ipcameras-based NVR (or VMS if you prefer)? Based on our limited experience with BTRFS (1+ year) under the above scenario the answer seems to be no; but I wanted you ask the community at large for their experience before making a final decision to hold off on deploying BTRFS on production systems. Let us be clear: We think BTRFS has great potential, and as it matures we will continue to watch its progress, so that at some future point we can return to using it. The issue has been the myriad of problems we have encountered when deploying BTRFS as the storage fs for the NVR/VMS in cases were the camera count exceeds 10: Corrupted file systems, sudden read-only file system, re-balance kernel panics, broken partitions, etc. One specific case saw the btrfs drive pool mounted under the /var partition so that upon installation the btrfs pool contained all files under /var; /lib/mysql as well as the video storage. Needless to say this was a catastrophe... At the other end of the spectrum is a use case where ONLY the video storage was on the btrfs pool; but in that case, the btrfs pool became read-only suddenly and would not re-mount as rw despite all the recovery trick btrfs-tools could throw at it. This, of course prevented the NVR/VMS from recording any footage. So, again, the question is: is BTRFS mature enough to be used in such use case and if so, what approach can be used to mitigate such issues. Thank you all for your assistance -Ryan- -- Ryan Burt, B. Sc. for Juan Alberto Cirez, CFPT
