Re: btrfs/125 deadlock using nospace_cache or space_cache=v2

At 02/07/2017 04:02 PM, Anand Jain wrote: Hi Qu, I don't think I have seen this before, I don't know the reason why I wrote this, may be to test encryption, however it was all with default options. Forgot to mention, thanks for the test case. Or we will never find it. Thanks, Qu But

Re: btrfs/125 deadlock using nospace_cache or space_cache=v2

Hi Anand, At 02/07/2017 04:02 PM, Anand Jain wrote: Hi Qu, I don't think I have seen this before, I don't know the reason why I wrote this, may be to test encryption, however it was all with default options. But now I could reproduce and, looks like balance fails to start with IO error

Re: [PATCH] Btrfs: add another missing end_page_writeback on submit_extent_page failure

On 2017/02/07 1:34, Liu Bo wrote: One thing to add, we still need to check whether page has writeback bit before end_page_writeback. Ok, I add PageWriteback check before end_page_writeback. Looks like commit 55e3bd2e0c2e1 also has the same problem although I gave it my reviewed-by. I

BTRFS for OLTP Databases

Hi, I have tried BTRFS from Ubuntu 16.04 LTS for write intensive OLTP MySQL Workload. It did not go very well ranging from multi-seconds stalls where no transactions are completed to the finally kernel OOPS with "no space left on device" error message and filesystem going read only. I'm

Re: BTRFS for OLTP Databases

On Tue, Feb 07, 2017 at 08:53:35AM -0500, Peter Zaitsev wrote: > Hi, > > I have tried BTRFS from Ubuntu 16.04 LTS for write intensive OLTP MySQL > Workload. > > It did not go very well ranging from multi-seconds stalls where no > transactions are completed to the finally kernel OOPS with "no

Re: BTRFS for OLTP Databases

On 2017-02-07 08:53, Peter Zaitsev wrote: Hi, I have tried BTRFS from Ubuntu 16.04 LTS for write intensive OLTP MySQL Workload. It did not go very well ranging from multi-seconds stalls where no transactions are completed to the finally kernel OOPS with "no space left on device" error

Re: BTRFS for OLTP Databases

Hi Hugo, For the use case I'm looking for I'm interested in having snapshot(s) open at all time. Imagine for example snapshot being created every hour and several of these snapshots kept at all time providing quick recovery points to the state of 1,2,3 hours ago. In such case (as I think you

understanding disk space usage

Hello, My system is or seems to be running out of disk space but I can't find out how or why. Might be a BTRFS peculiarity, hence posting on this list. Most indicators seem to suggest I'm filling up, but I can't trace the disk usage to files on the FS. The issue is on my root filesystem on a

Re: BTRFS for OLTP Databases

Hi Peter, Le 07/02/2017 à 15:13, Peter Zaitsev a écrit : > Hi Hugo, > > For the use case I'm looking for I'm interested in having snapshot(s) > open at all time. Imagine for example snapshot being created every > hour and several of these snapshots kept at all time providing quick > recovery

Re: BTRFS for OLTP Databases

On 2017-02-07 10:00, Timofey Titovets wrote: 2017-02-07 17:13 GMT+03:00 Peter Zaitsev : Hi Hugo, For the use case I'm looking for I'm interested in having snapshot(s) open at all time. Imagine for example snapshot being created every hour and several of these snapshots

Re: BTRFS for OLTP Databases

2017-02-07 17:13 GMT+03:00 Peter Zaitsev : > Hi Hugo, > > For the use case I'm looking for I'm interested in having snapshot(s) > open at all time. Imagine for example snapshot being created every > hour and several of these snapshots kept at all time providing quick >

Re: BTRFS for OLTP Databases

>> I think that you have a problem with extent bookkeeping (if i >> understand how btrfs manage extents). >> So for deal with it, try enable compression, as compression will force >> all extents to be fragmented with size ~128kb. > > No, it will compress everything in chunks of 128kB, but it will

Re: BTRFS for OLTP Databases

On 2017-02-07 10:20, Timofey Titovets wrote: I think that you have a problem with extent bookkeeping (if i understand how btrfs manage extents). So for deal with it, try enable compression, as compression will force all extents to be fragmented with size ~128kb. No, it will compress everything

Re: Very slow balance / btrfs-transaction

On Tue, Feb 7, 2017 at 12:22 AM, Qu Wenruo wrote: > > > At 02/07/2017 12:09 AM, Goldwyn Rodrigues wrote: >> >> >> Hi Qu, >> >> On 02/05/2017 07:45 PM, Qu Wenruo wrote: >>> >>> >>> >>> At 02/04/2017 09:47 AM, Jorg Bornschein wrote: February 4, 2017 1:07 AM,

Re: BTRFS for OLTP Databases

> I have tried BTRFS from Ubuntu 16.04 LTS for write intensive > OLTP MySQL Workload. This has a lot of interesting and mostly agreeable information: https://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp The main target of Btrfs is where one wants checksums and occasional

[PATCH] Btrfs: fix use-after-free due to wrong order of destroying work queues

From: Filipe Manana Before we destroy all work queues (and wait for their tasks to complete) we were destroying the work queues used for metadata I/O operations, which can result in a use-after-free problem because most tasks from all work queues do metadata I/O operations.

READ YOUR MESSAGE NOW!!

Greeting's to you and your family! I will be glad if you will be capable to assist me to secure a sum of USD 15.5M) into your bank account in your country. This is a genuine transaction, It just that I cannot operate it alone without the help of a foreign partner that is my reason of

Re: BTRFS for OLTP Databases

Am Tue, 7 Feb 2017 10:06:34 -0500 schrieb "Austin S. Hemmelgarn" : > 4. Try using in-line compression. This can actually significantly > improve performance, especially if you have slow storage devices and > a really nice CPU. Just a side note: With nodatacow there'll be

Re: BTRFS for OLTP Databases

On 2017-02-07 13:59, Peter Zaitsev wrote: Jeff, Thank you very much for explanations. Indeed it was not clear in the documentation - I read it simply as "if you have snapshots enabled nodatacow makes no difference" I will rebuild the database in this mode from scratch and see how performance

Re: BTRFS for OLTP Databases

Am Tue, 7 Feb 2017 14:50:04 -0500 schrieb "Austin S. Hemmelgarn" : > > Also does autodefrag works with nodatacow (ie with snapshot) or > > are these exclusive ? > I'm not sure about this one. I would assume based on the fact that > many other things don't work with

Re: BTRFS for OLTP Databases

Le 07/02/2017 à 21:36, Kai Krakow a écrit : > [...] > I think I've read that btrfs snapshots do not guarantee single point in > time snapshots - the snapshot may be smeared across a longer period of > time while the kernel is still writing data. So parts of your writes > may still end up in the

Re: BTRFS for OLTP Databases

On 2017-02-07 15:36, Kai Krakow wrote: Am Tue, 7 Feb 2017 09:13:25 -0500 schrieb Peter Zaitsev : Hi Hugo, For the use case I'm looking for I'm interested in having snapshot(s) open at all time. Imagine for example snapshot being created every hour and several of these

Re: BTRFS for OLTP Databases

Am Tue, 7 Feb 2017 15:27:34 -0500 schrieb "Austin S. Hemmelgarn" : > >> I'm not sure about this one. I would assume based on the fact that > >> many other things don't work with nodatacow and that regular defrag > >> doesn't work on files which are currently mapped as

Re: BTRFS for OLTP Databases

Le 07/02/2017 à 21:47, Austin S. Hemmelgarn a écrit : > On 2017-02-07 15:36, Kai Krakow wrote: >> Am Tue, 7 Feb 2017 09:13:25 -0500 >> schrieb Peter Zaitsev : >> >>> Hi Hugo, >>> >>> For the use case I'm looking for I'm interested in having snapshot(s) >>> open at all time.

Re: BTRFS for OLTP Databases

Hi Hugo, As I re-read it closely (and also other comments in the thread) I know understand there is a difference how nodatacow works even if snapshot are in place. On autodefrag I wonder is there some more detailed documentation about how autodefrag works. The manual

Re: BTRFS for OLTP Databases

Am Tue, 7 Feb 2017 09:13:25 -0500 schrieb Peter Zaitsev : > Hi Hugo, > > For the use case I'm looking for I'm interested in having snapshot(s) > open at all time. Imagine for example snapshot being created every > hour and several of these snapshots kept at all time

Re: [PATCH] Btrfs: add another missing end_page_writeback on submit_extent_page failure

On Tue, Feb 07, 2017 at 08:09:53PM +0900, takafumi-sslab wrote: > > On 2017/02/07 1:34, Liu Bo wrote: > > > > > One thing to add, we still need to check whether page has writeback bit > > before > > end_page_writeback. > > Ok, I add PageWriteback check before end_page_writeback. > > > > > >

Re: BTRFS for OLTP Databases

On 2017-02-07 14:39, Kai Krakow wrote: Am Tue, 7 Feb 2017 10:06:34 -0500 schrieb "Austin S. Hemmelgarn" : 4. Try using in-line compression. This can actually significantly improve performance, especially if you have slow storage devices and a really nice CPU. Just a

Re: BTRFS for OLTP Databases

Am Tue, 7 Feb 2017 10:43:11 -0500 schrieb "Austin S. Hemmelgarn" : > > I mean that: > > You have a 128MB extent, you rewrite random 4k sectors, btrfs will > > not split 128MB extent, and not free up data, (i don't know > > internal algo, so i can't predict when this will

Re: BTRFS for OLTP Databases

Am Tue, 7 Feb 2017 22:25:29 +0100 schrieb Lionel Bouton : > Le 07/02/2017 à 21:47, Austin S. Hemmelgarn a écrit : > > On 2017-02-07 15:36, Kai Krakow wrote: > >> Am Tue, 7 Feb 2017 09:13:25 -0500 > >> schrieb Peter Zaitsev : > >> > [...] >

Re: BTRFS for OLTP Databases

On 2017-02-07 14:31, Peter Zaitsev wrote: Hi Hugo, As I re-read it closely (and also other comments in the thread) I know understand there is a difference how nodatacow works even if snapshot are in place. On autodefrag I wonder is there some more detailed documentation about how autodefrag

Re: Very slow balance / btrfs-transaction

Am Mon, 6 Feb 2017 08:19:37 -0500 schrieb "Austin S. Hemmelgarn" : > > MDRAID uses stripe selection based on latency and other measurements > > (like head position). It would be nice if btrfs implemented similar > > functionality. This would also be helpful for selecting a

Re: Very slow balance / btrfs-transaction

On 2017-02-07 14:47, Kai Krakow wrote: Am Mon, 6 Feb 2017 08:19:37 -0500 schrieb "Austin S. Hemmelgarn" : MDRAID uses stripe selection based on latency and other measurements (like head position). It would be nice if btrfs implemented similar functionality. This would

Re: BTRFS for OLTP Databases

On 2017-02-07 15:19, Kai Krakow wrote: Am Tue, 7 Feb 2017 14:50:04 -0500 schrieb "Austin S. Hemmelgarn" : Also does autodefrag works with nodatacow (ie with snapshot) or are these exclusive ? I'm not sure about this one. I would assume based on the fact that many other

Re: BTRFS for OLTP Databases

Austin, I recognize there are other components too. In this case I'm actually comparing BTRFS to XFS and EXT4 so I'm 100% sure it is file system related. Also I'm using O_DIRECT asynchronous IO with MySQL which means there are no significant dirty block size on the file system level. I'll

Re: BTRFS for OLTP Databases

On Tue, 7 Feb 2017 09:13:25 -0500 Peter Zaitsev wrote: > Hi Hugo, > > For the use case I'm looking for I'm interested in having snapshot(s) > open at all time. Imagine for example snapshot being created every > hour and several of these snapshots kept at all time providing

Re: BTRFS for OLTP Databases

On 02/07/2017 07:59 PM, Peter Zaitsev wrote: > > So far the most frustating for me was periodic stalls for many seconds > (running sysbench workload). What was the most puzzling I get this > even if I run workload at the 50% or less of the full load - Ie > database can handle 1000

Re: dup vs raid1 in single disk

Am Thu, 19 Jan 2017 15:02:14 -0500 schrieb "Austin S. Hemmelgarn" : > On 2017-01-19 13:23, Roman Mamedov wrote: > > On Thu, 19 Jan 2017 17:39:37 +0100 > > "Alejandro R. Mosteo" wrote: > > > >> I was wondering, from a point of view of data safety, if

Re: [PATCH] btrfs-progs: better document btrfs receive security

Am Fri, 3 Feb 2017 08:48:58 -0500 schrieb "Austin S. Hemmelgarn" : > +user who is running receive, and then move then into the final > destination Typo? s/then/them/? -- Regards, Kai Replies to list-only preferred. -- To unsubscribe

Re: dup vs raid1 in single disk

On 02/07/2017 11:28 PM, Kai Krakow wrote: > Am Thu, 19 Jan 2017 15:02:14 -0500 > schrieb "Austin S. Hemmelgarn" : > >> On 2017-01-19 13:23, Roman Mamedov wrote: >>> On Thu, 19 Jan 2017 17:39:37 +0100 >>> [...] >>> And the DUP mode is still useful on SSDs, for cases when one

Re: Very slow balance / btrfs-transaction

At 02/07/2017 11:55 PM, Filipe Manana wrote: On Tue, Feb 7, 2017 at 12:22 AM, Qu Wenruo wrote: At 02/07/2017 12:09 AM, Goldwyn Rodrigues wrote: Hi Qu, On 02/05/2017 07:45 PM, Qu Wenruo wrote: At 02/04/2017 09:47 AM, Jorg Bornschein wrote: February 4, 2017

Re: dup vs raid1 in single disk

On 8 February 2017 at 08:28, Kai Krakow wrote: > I still thinks it's a myth... The overhead of managing inline > deduplication is just way too high to implement it without jumping > through expensive hoops. Most workloads have almost zero deduplication > potential. And even

Re: BTRFS for OLTP Databases

On 02/07/2017 10:35 PM, Kai Krakow wrote: > Am Tue, 7 Feb 2017 22:25:29 +0100 > schrieb Lionel Bouton : > >> Le 07/02/2017 à 21:47, Austin S. Hemmelgarn a écrit : >>> On 2017-02-07 15:36, Kai Krakow wrote: Am Tue, 7 Feb 2017 09:13:25 -0500 schrieb

Re: [lustre-devel] [PATCH 04/24] fs: Provide infrastructure for dynamic BDIs in filesystems

On Feb 2, 2017, at 10:34, Jan Kara wrote: > > Provide helper functions for setting up dynamically allocated > backing_dev_info structures for filesystems and cleaning them up on > superblock destruction. > > CC: linux-...@lists.infradead.org > CC: linux-...@vger.kernel.org > CC:

user_subvol_rm_allowed? Is there a user_subvol_create_deny|allowed?

Dear btrfs community, Please accept my apologies in advance if I missed something in recent btrfs development; my MUA tells me I'm ~1500 unread messages out-of-date. :/ I recently read about "mount -t btrfs -o user_subvol_rm_allowed" while doing reading up on LXC handling of snapshots with the

Re: understanding disk space usage

At 02/08/2017 12:44 AM, Vasco Visser wrote: Hello, My system is or seems to be running out of disk space but I can't find out how or why. Might be a BTRFS peculiarity, hence posting on this list. Most indicators seem to suggest I'm filling up, but I can't trace the disk usage to files on the

Troubleshooting crash due to running out of space issues

I'm running BTRFS on Ubuntu 16.04 - I was testing intensive database IO which ends up with pretty fragmented data file: root@blinky:/var/lib/mysql/sbtest# filefrag sbtest1.ibd sbtest1.ibd: 13415923 extents found This is 500G device which is some 60% full: /dev/nvme0n1500107608

Re: raid1: cannot add disk to replace faulty because can only mount fs as read-only.

Greetings, On 2017-02-02 10:06, Austin S. Hemmelgarn wrote: > On 2017-02-02 09:25, Adam Borowski wrote: >> On Thu, Feb 02, 2017 at 07:49:50AM -0500, Austin S. Hemmelgarn wrote: >>> This is a severe bug that makes a not all that uncommon (albeit bad) use >>> case fail completely. The fix had no

[PATCH] btrfs: qgroup: Move half of the qgroup accounting time out of commit trans

Just as Filipe pointed out, the most time consuming part of qgroup is btrfs_qgroup_account_extents() and btrfs_qgroup_prepare_account_extents(). Which both call btrfs_find_all_roots() to get old_roots and new_roots ulist. However for old_roots, we don't really need to calculate it at transaction

Re: BTRFS for OLTP Databases

Hi Kai, I guess your message did not make it to me as I'm not subscribed to the list. I totally understand what the the snapshot is "crash consistent" - consistent to the state of the disk you would find if you shut down the power with no notice, for many applications it is a problem however it

Re: BTRFS for OLTP Databases

Jeff, Thank you very much for explanations. Indeed it was not clear in the documentation - I read it simply as "if you have snapshots enabled nodatacow makes no difference" I will rebuild the database in this mode from scratch and see how performance changes. So far the most frustating for me

Re: BTRFS for OLTP Databases

On 2/7/17 8:53 AM, Peter Zaitsev wrote: > Hi, > > I have tried BTRFS from Ubuntu 16.04 LTS for write intensive OLTP MySQL > Workload. > > It did not go very well ranging from multi-seconds stalls where no > transactions are completed to the finally kernel OOPS with "no space left > on device"

Re: [PATCH] btrfs-progs: better document btrfs receive security

On Fri, Feb 03, 2017 at 08:48:58AM -0500, Austin S. Hemmelgarn wrote: > This adds some extra documentation to the btrfs-receive manpage that > explains some of the security related aspects of btrfs-receive. The > first part covers the fact that the subvolume being received is writable > until the