Re: [PATCH 00/10] btrfs: Support for DAX devices

2018-12-05 Thread Robert White
On 12/5/18 9:37 PM, Jeff Mahoney wrote: The high level idea that Jan Kara and I came up with in our conversation at Labs conf is pretty expensive.  We'd need to set a flag that pauses new page faults, set the WP bit on affected ranges, do the snapshot, commit, clear the flag, and wake up the

What if TRIM issued a wipe on devices that don't TRIM?

2018-12-05 Thread Robert White
(1) Automatic and selective wiping of unused and previously used disk blocks is a good security measure, particularly when there is an encryption layer beneath the file system. (2) USB attached devices _never_ support TRIM and they are the most likely to fall into strangers hands. (3) I

Theoretical Question about commit=n

2017-11-12 Thread Robert White
Is the commit interval monotonic, or is it seconds after sync? What I mean is that if I manually call sync(2) does the commit timer reset? I'm thinking it does not, but I can imagine a workload where it ideally would. (Again, this is purely theoretical, I have no such workload as I am about to

A good "Boot Maintenance" scheme (WAS: New file system with same issue)

2016-03-18 Thread Robert White
On 03/14/2016 01:13 PM, Marc Haber wrote: This was not asked, and I didn't try. Since this is an encrypted root filesystem, is it a workable way to add clear_cache to /etc/fstab, rebuild initramfs and reboot? Or do you recommend using a rescue system? You should be able to boot to single user

Stupid (?) Idea about extent lifetimes.

2016-03-16 Thread Robert White
It occurs to me that it would be desirable to mark extents as "least favoured nations" and so all new writes would like to not be written there and any data written there would have a desire to be somewhere else. So lets say the wholly unallocated space has a natural status of 100. Allocated

Any Arguments for/against --bind mounts?

2015-03-13 Thread Robert White
Is there any practical reason to prefer bind mounts or separately mounting a subvolume? e.g. assuming /locationA and /locationB are arbitrarily far apart in the file system tree, is there any reason to prefer one of the following over the other mount -t btrfs -o subvolume=/thing /dev/sdN1

Re: Documenting MS_LAZYTIME

2015-02-22 Thread Robert White
On 02/20/2015 12:50 AM, Michael Kerrisk wrote: * As at Linux 3.20, this option is supported only on ext4. As of Linux 3.20 is more correct. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo

Re: scrub problem

2015-02-22 Thread Robert White
On 02/20/2015 01:03 PM, Bob Williams wrote: /var/lib/btrfs/scrub.status.0b07b829-9a0e-44ab-89ee-14b36a45199e (the last bit of the filename is the filesystem uuid) Look for a line that ends with finished:0 and change it to say finished:1 Why does this data item even exist? The

Re: Btrfs subvolume question

2015-02-04 Thread Robert White
On 02/04/2015 06:27 PM, Chris Murphy wrote: On Wed, Feb 4, 2015 at 3:54 PM, Markus Moeller hua...@moeller.plus.com wrote: Hi , I am new to btrfs and wonder what I need to do to move subvolumes to the right filesystem. I see the following: df -h Filesystem Size Used

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-28 Thread Robert White
On 12/28/2014 04:07 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White: Now: The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-28 Thread Robert White
On 12/28/2014 07:42 AM, Martin Steigerwald wrote: Am Sonntag, 28. Dezember 2014, 06:52:41 schrieb Robert White: On 12/28/2014 04:07 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White: Now: The complaining party has verified the minimum, repeatable case

Re: fstrim not working on one of three BTRFS filesystems

2014-12-28 Thread Robert White
On 12/28/2014 08:58 AM, Martin Steigerwald wrote: Hi! After my recent tests with my /home filesystem and the up and downsizing of it I get: merkaba:~ LANG=C fstrim -v /home /home: 0 B (0 bytes) trimmed merkaba:~ LANG=C fstrim -v / /: 24.5 GiB (26257555456 bytes) trimmed merkaba:~ LANG=C

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 02:54 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills: On Sat, Dec 27, 2014 at 10:01:17AM +0100, Martin Steigerwald wrote: Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White: On 12/26/2014 05:37 AM, Martin Steigerwald wrote

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 03:11 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills: I only see the lockups of BTRFS is the trees *occupy* all space on the device. No, the trees occupy 3.29 GiB of your 5 GiB of mirrored metadata space. What's more, balance does

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 05:16 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 03:52:56 schrieb Robert White: My theory from watching the Windows XP defragmentation case is this: - For writing into the file BTRFS needs to actually allocate and use free space in the current tree allocation

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 05:16 AM, Martin Steigerwald wrote: It can easily be reproduced without even using Virtualbox, just by a nice simple fio job. TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file... #!/bin/bash # not tested, so correct any syntax errors

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 06:00 AM, Robert White wrote: On 12/27/2014 05:16 AM, Martin Steigerwald wrote: It can easily be reproduced without even using Virtualbox, just by a nice simple fio job. TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file... #!/bin/bash

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
in Virtualbox for reproducing the issue. Next I will try to reproduce with a freshly creating filesystem. Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills: On Sat, Dec 27, 2014 at 10:01:17AM +0100, Martin Steigerwald wrote: Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 06:21 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 15:14:05 schrieb Martin Steigerwald: Am Samstag, 27. Dezember 2014, 06:00:48 schrieb Robert White: On 12/27/2014 05:16 AM, Martin Steigerwald wrote: It can easily be reproduced without even using Virtualbox, just

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
Semi off-topic questions... On 12/27/2014 08:26 AM, Hugo Mills wrote: This is... badly mistaken, at best. The problem of where to write a file into a set of free extents is definitely *not* an NP-hard problem. It's a P problem, with an O(n log n) solution, where n is the number of free

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 08:01 AM, Martin Steigerwald wrote: From how you write I get the impression that you think everyone else beside you is just silly and dumb. Please stop this assumption. I may not always get terms right, and I may make a mistake as with the wrong df figure. But I also highly

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 05:01 PM, Bardur Arantsson wrote: On 2014-12-28 01:25, Robert White wrote: On 12/27/2014 08:01 AM, Martin Steigerwald wrote: From how you write I get the impression that you think everyone else beside you is just silly and dumb. Please stop this assumption. I may not always get

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-26 Thread Robert White
On 12/26/2014 05:37 AM, Martin Steigerwald wrote: Hello! First: Have a merry christmas and enjoy a quiet time in these days. Second: At a time you feel like it, here is a little rant, but also a bug report: I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache,

Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-26 Thread Robert White
On 12/23/2014 04:31 AM, Dongsheng Yang wrote: On 12/18/2014 12:07 PM, Robert White wrote: I don't disagree with the _ideal_ of your patch. I just think that it's impossible to implement it without lying to the user or making things just as bad in a different way. I would _like_ you to be right

Re: Can BTRFS handle XATTRs larger than 4K?

2014-12-22 Thread Robert White
On 12/22/2014 12:44 PM, Austin S Hemmelgarn wrote: On 2014-12-22 15:06, Richard Sharpe wrote: On Mon, Dec 22, 2014 at 10:43 AM, Chris Murphy li...@colorremedies.com wrote: On Mon, Dec 22, 2014 at 11:09 AM, Austin S Hemmelgarn ahferro...@gmail.com wrote: Personally, I'd love to see unlimited

Re: Can BTRFS handle XATTRs larger than 4K?

2014-12-22 Thread Robert White
So I'll ask again... On 12/22/2014 03:15 PM, ronnie sahlberg wrote: On Mon, Dec 22, 2014 at 2:52 PM, Robert White rwh...@pobox.com wrote: So skipping the full ADS, what's the current demand/payoff for large XATTR space? So skipping the full ADS, what is the current demand

Re: Can BTRFS handle XATTRs larger than 4K?

2014-12-22 Thread Robert White
On 12/22/2014 02:55 PM, Richard Sharpe wrote: On Mon, Dec 22, 2014 at 2:52 PM, Robert White rwh...@pobox.com wrote: So skipping the full ADS, what's the current demand/payoff for large XATTR space? Windows Security Descriptors (sometimes incorrectly called ACLs) stored by Samba. Ah. I

Re: Can BTRFS handle XATTRs larger than 4K?

2014-12-22 Thread Robert White
On 12/22/2014 03:58 PM, Richard Sharpe wrote: On Mon, Dec 22, 2014 at 3:55 PM, Robert White rwh...@pobox.com wrote: So I'll ask again... On 12/22/2014 03:15 PM, ronnie sahlberg wrote: On Mon, Dec 22, 2014 at 2:52 PM, Robert White rwh...@pobox.com wrote: So skipping the full ADS, what's

Re: Oddly slow read performance with near-full largish FS

2014-12-21 Thread Robert White
On 12/21/2014 08:32 AM, Charles Cazabon wrote: Hi, Robert, Thanks for the response. Many of the things you mentioned I have tried, but for completeness: Have you taken SMART (smartmotools etc) to these disks There are no errors or warnings from SMART for the disks. Do make sure you are

Re: Uncorrectable errors on RAID-1?

2014-12-21 Thread Robert White
On 12/21/2014 11:34 AM, constantine wrote: Some months ago I had 6 uncorrectable errors. I deleted the files that contained them and then after scrubbing I had 0 uncorrectable errors. After some weeks I encountered new uncorrectable errors. Question 1: Why do I have uncorrectable errors on a

Re: Oddly slow read performance with near-full largish FS

2014-12-21 Thread Robert White
On 12/21/2014 02:53 PM, Charles Cazabon wrote: Hi, Robert, My performance issues with btrfs are more-or-less resolved now -- the performance under btrfs still seems quite variable compared to other filesystems -- my rsync speed is now varying between 40MB and ~90MB/s, with occasional intervals

Re: Oddly slow read performance with near-full largish FS

2014-12-20 Thread Robert White
On 12/19/2014 09:33 AM, Duncan wrote: Charles Cazabon posted on Fri, 19 Dec 2014 10:58:49 -0600 as excerpted: This configuration is one I've been using for many years. It's only recently that I've noticed it being particularly slow with btrfs -- I don't know if that's because the filesystem

Re: Oddly slow read performance with near-full largish FS

2014-12-20 Thread Robert White
On 12/16/2014 06:42 PM, Charles Cazabon wrote: Hi, I've been running btrfs for various filesystems for a few years now, and have recently run into problems with a large filesystem becoming *really* slow for basic reading. None of the debugging/testing suggestions I've come across in the wiki

Re: btrfs is using 25% more disk than it should

2014-12-20 Thread Robert White
On 12/19/2014 01:17 PM, Josef Bacik wrote: tl;dr: Cow means you can in the worst case end up using 2 * filesize - blocksize of data on disk and the file will appear to be filesize. Thanks, Doesn't the worst case more like N^log(N) (when N is file in blocksize) in the pernicious case?

Re: btrfs is using 25% more disk than it should

2014-12-20 Thread Robert White
On 12/20/2014 03:39 AM, Josef Bacik wrote: On 12/20/2014 06:23 AM, Robert White wrote: On 12/19/2014 01:17 PM, Josef Bacik wrote: tl;dr: Cow means you can in the worst case end up using 2 * filesize - blocksize of data on disk and the file will appear to be filesize. Thanks, Doesn't

Re: btrfs is using 25% more disk than it should

2014-12-20 Thread Robert White
On 12/19/2014 01:10 PM, Josef Bacik wrote: On 12/18/2014 09:59 AM, Daniele Testa wrote: Hey, I am hoping you guys can shed some light on my issue. I know that it's a common question that people see differences in the disk used when running different calculations, but I still think that my

Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-17 Thread Robert White
I don't disagree with the _ideal_ of your patch. I just think that it's impossible to implement it without lying to the user or making things just as bad in a different way. I would _like_ you to be right. But my thing is finding and quantifying failure cases and the entire question is full of

Re: [PATCH 6/7] Print the summary

2014-12-17 Thread Robert White
On 12/16/2014 01:05 AM, Hugo Mills wrote: On Mon, Dec 15, 2014 at 07:47:06PM -0800, Robert White wrote: I prefer slice, not that I am totally happy with that word either. But by the time you get through loopback devices, memory map devices, the device files that represent parts of partitioned

Re: btrfs receive being very slow

2014-12-15 Thread Robert White
On 12/14/2014 11:41 PM, Nick Dimov wrote: Hi, thanks for the answer, I will answer between the lines. On 15.12.2014 08:45, Robert White wrote: On 12/14/2014 08:50 PM, Nick Dimov wrote: Hello everyone! First, thanks for amazing work on btrfs filesystem! Now the problem: I use a ssd as my

Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.

2014-12-15 Thread Robert White
On 12/15/2014 12:26 AM, Dongsheng Yang wrote: On 12/15/2014 03:49 PM, Robert White wrote: On 12/14/2014 10:06 PM, Robert White wrote: On 12/14/2014 05:21 PM, Dongsheng Yang wrote: Anyone have some suggestion about it? (... strong advocacy for raw numbers...) Hi Robert, thanx for your so

Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-15 Thread Robert White
On 12/15/2014 01:36 AM, Robert White wrote: So we don't just hand-wave over statfs(). We include the dev_item.bytes_excluded in the superblock and we decide once-and-for-all (with any geometry creation, or completed conversion) how many bytes just _can't_ be reached but only once we _know_

Re: [PATCH 6/7] Print the summary

2014-12-15 Thread Robert White
On 12/15/2014 05:58 PM, Duncan wrote: * Please s/disk/device/, here and possibly elsewhere. I know I'm not the only one who is trying to make the switch in my own usage, as it looks a bit foolish (and/or marks the user as an old fogey who's likely to start lecturing about how a GiB isn't small,

Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-15 Thread Robert White
On 12/15/2014 07:30 PM, Robert White wrote: The above would be ideal. But POSIX says no. f_blocks is defined (only Correction the linux kernel says total data blocks, POSIX says total blocks -- it was a mental typo... 8-) in the comments) as total data blocks in the filesystem and /bin/df

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-14 Thread Robert White
On 12/13/2014 03:56 PM, Robert White wrote: ... Dangit... On re-reading I think I was still less than optimally clear. I kept using the word resent when I should have been using a word like re-written or re-stored (as opposed to restored). On re-reading I'm not sure what the least confusing

Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.

2014-12-14 Thread Robert White
On 12/14/2014 05:21 PM, Dongsheng Yang wrote: Does it make sense to you? I understood what you were saying but it didn't make sense to me... As there are 2 complaints for the same change of @size in df, I have to say it maybe not so easy to understand. Anyone have some suggestion about

Re: btrfs receive being very slow

2014-12-14 Thread Robert White
On 12/14/2014 08:50 PM, Nick Dimov wrote: Hello everyone! First, thanks for amazing work on btrfs filesystem! Now the problem: I use a ssd as my system drive (/dev/sda2) and use daily snapshots on it. Then, from time to time, i sync those on HDD (/dev/sdb4) by using btrfs send / receive like

Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.

2014-12-14 Thread Robert White
On 12/14/2014 10:06 PM, Robert White wrote: On 12/14/2014 05:21 PM, Dongsheng Yang wrote: Anyone have some suggestion about it? (... strong advocacy for raw numbers...) Concise Example to attempt to be clearer: /dev/sda == 1TiB /dev/sdb == 2TiB /dev/sdc == 3TiB /dev/sdd == 3TiB mkfs.btrfs

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-13 Thread Robert White
On 12/13/2014 12:16 AM, Tomasz Chmielewski wrote: On 2014-12-12 23:58, Robert White wrote: I don't have the history to answer this definitively, but I don't think you have a choice. Nothing else is going to touch that error. I have not seen any oh my god, btrfsck just ate my filesystem errors

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-13 Thread Robert White
On 12/13/2014 05:53 AM, Tomasz Chmielewski wrote: My usage case is quite simple: - skinny extents, extended inode refs okay - mount compress-force=zlib I'd, personally, never force compression. This can increase the size of files by five or more percent if it is an inherently incompressible

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-13 Thread Robert White
On 12/13/2014 01:52 PM, Tomasz Chmielewski wrote: On 2014-12-13 21:54, Robert White wrote: - rsync many remote data sources (-a -H --inplace --partial) + snapshot Using --inplace on a Copy On Write filesystem has only one effect, it increases fragmentation... a lot... ...if the file

Re: subvolume / folder compression flag

2014-12-13 Thread Robert White
On 12/13/2014 06:59 PM, Ali AlipourR wrote: 2- ... and rsync files without compression flag ... The --compress flag for rsync has nothing to do with how the files are stored on either end. It determines whether the data is compressed as it passes from the source rsync to the destination

Re: mkfs.btrfs limits odd [and maybe a failed phantom device?]

2014-12-12 Thread Robert White
On 12/12/2014 01:06 AM, David Taylor wrote: The above quote is discussing two device RAID5, you are discussing three device RAID5. Heresy! (yes, some humor is required here.) There is no such thing as a two device RAID5. That's what RAID1 is for. Saying The above quote is discussing a two

Re: A note on spotting bugs [Was: ENOSPC after conversion]

2014-12-12 Thread Robert White
On 12/11/2014 10:42 PM, Patrik Lundquist wrote: On 11 December 2014 at 23:00, Robert White rwh...@pobox.com wrote: On 12/11/2014 12:18 AM, Patrik Lundquist wrote: * Full balance, that ended with 98 enospc errors during balance. Assuming that quote is an actual quote from the output

Re: Balance scrub defrag

2014-12-12 Thread Robert White
On 12/12/2014 01:17 AM, Erkki Seppala wrote: Robert White rwh...@pobox.com writes: You need to buy better disks. 8-) Where can one buy these better disks with reasonable prices?-) Disks are best thought of as consumables. A good disk is only about 9% more expensive. So like the WD green

Re: mkfs.btrfs limits odd [and maybe a failed phantom device?]

2014-12-12 Thread Robert White
On 12/12/2014 08:45 AM, Zygo Blaxell wrote: On Thu, Dec 11, 2014 at 10:01:06PM -0800, Robert White wrote: So RAID5 with three media M is MMM MMM D1 D2 P(a) D3 P(b) D4 P(c) D5 D6 RAID5 with two media is well defined, and looks like this: MMM D1 P(a) P(b) D2 D3 P(c

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-12 Thread Robert White
On 12/12/2014 01:46 PM, Tomasz Chmielewski wrote: On 2014-12-12 22:36, Robert White wrote: In another thread [that was discussing SMART] you talked about replacing a drive and then needing to do some patching-up of the result because of drive failures. Is this the same filesystem where

RAID0 extent sizes?

2014-12-12 Thread Robert White
I've seen it mentioned here that generally data extents are 1G and metadata extents are 256M. Is that per-drive or per-stripe in the case of RAID0? That is, if I have data mode raid0 across N drives does the system allocate one 1G extent on each drive making the full stripe allocation

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-12 Thread Robert White
On 12/12/2014 02:46 PM, Tomasz Chmielewski wrote: On 2014-12-12 23:34, Robert White wrote: On 12/12/2014 01:46 PM, Tomasz Chmielewski wrote: On 2014-12-12 22:36, Robert White wrote: In another thread [that was discussing SMART] you talked about replacing a drive and then needing to do some

Re: RAID0 extent sizes?

2014-12-12 Thread Robert White
On 12/12/2014 02:59 PM, Hugo Mills wrote: On Fri, Dec 12, 2014 at 02:54:24PM -0800, Robert White wrote: I've seen it mentioned here that generally data extents are 1G and metadata extents are 256M. Is that per-drive or per-stripe in the case of RAID0? That is, if I have data mode raid0 across

Re: A note on spotting bugs [Was: ENOSPC after conversion]

2014-12-12 Thread Robert White
On 12/12/2014 05:12 PM, Duncan wrote: Robert White posted on Fri, 12 Dec 2014 05:29:58 -0800 as excerpted: This still doesnt say _anything_ is wrong with your filesystem except that it doesn't have enough _raw_ space to create a 2-ish gig extent. What's wrong with the filesystem

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-11 Thread Robert White
On 12/10/2014 05:36 AM, Patrik Lundquist wrote: On 10 December 2014 at 13:17, Robert White rwh...@pobox.com wrote: On 12/09/2014 11:19 PM, Patrik Lundquist wrote: BUT FIRST UNDERSTAND: you do _not_ need to balance a newly converted filesystem. That is, the recommended balance (and recursive

Re: out of space warning?

2014-12-11 Thread Robert White
On 12/11/2014 01:01 AM, Erkki Seppala wrote: Robert White rwh...@pobox.com writes: You don't check your car's gas tank every time you put your foot on the brake, you don't want to check your free space every time your system finishes every tiny command you type. Well, actually my car makes

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

2014-12-11 Thread Robert White
So far I don't see a bug. On 12/11/2014 12:18 AM, Patrik Lundquist wrote: I'll reboot the thread with a recap and my latest findings. * Half full 3TB disk converted from ext4 to Btrfs, after first verifying it with fsck. * Undo subvolume deleted after being happy with the conversion. *

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-11 Thread Robert White
On 12/11/2014 01:55 AM, Patrik Lundquist wrote: On 11 December 2014 at 09:42, Robert White rwh...@pobox.com wrote: On 12/10/2014 05:36 AM, Patrik Lundquist wrote: On 10 December 2014 at 13:17, Robert White rwh...@pobox.com wrote: On 12/09/2014 11:19 PM, Patrik Lundquist wrote: BUT FIRST

A note on spotting bugs [Was: ENOSPC after conversion]

2014-12-11 Thread Robert White
On 12/11/2014 12:18 AM, Patrik Lundquist wrote: * Full balance, that ended with 98 enospc errors during balance. Assuming that quote is an actual quote from the output of the balance... We can strongly infer that this sort of occurrence is expected since there is code to keep track of it and

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

2014-12-11 Thread Robert White
On 12/11/2014 03:01 PM, Patrik Lundquist wrote: On 11 December 2014 at 11:18, Robert White rwh...@pobox.com wrote: So far I don't see a bug. Fair enough, lets call it a huge problem with btrfs convert. I think it warrants a note in the wiki. On 12/11/2014 12:18 AM, Patrik Lundquist wrote

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

2014-12-11 Thread Robert White
TL;DR version On 12/11/2014 03:01 PM, Patrik Lundquist wrote: Of course the filesystem is in a problematic state after the conversion, even if it's not a bug. ~1.5TB of free space and yet out of space and it can't be fixed with a balance. It might not be wrong per se but it's very problematic

Re: Balance scrub defrag

2014-12-11 Thread Robert White
On 12/11/2014 05:00 PM, Russell Coker wrote: On Wed, 10 Dec 2014 17:17:28 Robert White wrote: A _monthly_ scrub is maybe worth scheduling if you have a lot of churn in your disk contents. I do weekly scrubs. I recently had 2 disks in a RAID-1 array develop read errors within a month of each

Re: [PATCH] Btrfs: get more accurate output in fd command.

2014-12-10 Thread Robert White
On 12/09/2014 05:08 PM, Dongsheng Yang wrote: On 12/10/2014 02:47 AM, Goffredo Baroncelli wrote: Hi Dongsheng On 12/09/2014 12:20 PM, Dongsheng Yang wrote: When function btrfs_statfs() calculate the tatol size of fs, it is calculating the total size of disks and then dividing it by a factor.

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-10 Thread Robert White
On 12/09/2014 11:19 PM, Patrik Lundquist wrote: On 10 December 2014 at 00:13, Robert White rwh...@pobox.com wrote: On 12/09/2014 02:29 PM, Patrik Lundquist wrote: Label: none uuid: 770fe01d-6a45-42b9-912e-e8f8b413f6a4 Total devices 1 FS bytes used 1.35TiB devid1 size 2.73TiB

Re: [PATCH] Btrfs: get more accurate output in fd command.

2014-12-10 Thread Robert White
On 12/10/2014 05:21 AM, Duncan wrote: Robert White posted on Wed, 10 Dec 2014 02:53:40 -0800 as excerpted: On 12/09/2014 05:08 PM, Dongsheng Yang wrote: On 12/10/2014 02:47 AM, Goffredo Baroncelli wrote: Hi Dongsheng On 12/09/2014 12:20 PM, Dongsheng Yang wrote: When function btrfs_statfs

mkfs.btrfs limits odd [and maybe a failed phantom device?]

2014-12-10 Thread Robert White
So I started looking at the mkfs.btrfs manual page with an eye towards documenting some of the tidbits like metadata automatically switching from dup to raid1 when more than one device is used. In experimenting I ended up with some questions... (1) why is the dup profile for data restricted

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-10 Thread Robert White
On 12/10/2014 10:56 AM, Patrik Lundquist wrote: On 10 December 2014 at 14:11, Duncan 1i5t5.dun...@cox.net wrote: Assuming no snapshots still contain the file, of course, and that the ext* saved subvolume has already been deleted. Got no snapshots or subvolumes. Keeping it simple for now.

Re: Possibility to have a transient snapshot?

2014-12-10 Thread Robert White
On 12/10/2014 11:52 AM, James West wrote: I was just looking into using overlayfs, and although it has some promise, I think it's biggest drawback is the upperdir will have to be some sort of storage backed filesystem. From my limited understanding of tmpfs, it's not supposed to be the greatest

Re: Balance scrub defrag

2014-12-10 Thread Robert White
On 12/10/2014 02:15 PM, sys.syphus wrote: I am working on a script that i can run daily that will do maintenance on my btrfs mountpoints. is there any reason not to concurrently do all of the above? possibly including discards as well. also, is there anything existing currently that will do

Re: out of space warning?

2014-12-10 Thread Robert White
On 12/10/2014 02:54 PM, sys.syphus wrote: I would like to avoid running out of space. is there a way to know that I am getting close? i'd like to make a script that runs as part of my bash prompt and lets me know when i am getting close. i know there are several ways you can run out of space and

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-09 Thread Robert White
On 12/09/2014 02:29 PM, Patrik Lundquist wrote: Label: none uuid: 770fe01d-6a45-42b9-912e-e8f8b413f6a4 Total devices 1 FS bytes used 1.35TiB devid1 size 2.73TiB used 1.36TiB path /dev/sdc1 Data, single: total=1.35TiB, used=1.35TiB System, single: total=32.00MiB, used=112.00KiB

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-09 Thread Robert White
On 12/09/2014 02:29 PM, Patrik Lundquist wrote: Data, single: total=1.35TiB, used=1.35TiB System, single: total=32.00MiB, used=112.00KiB Metadata, single: total=3.00GiB, used=1.55GiB GlobalReserve, single: total=512.00MiB, used=0.00B P.S. you should re-balance your System and Metadata as DUP

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-09 Thread Robert White
On 12/09/2014 02:29 PM, Patrik Lundquist wrote: (stuff depicting a nearly full file system). Having taken another look at it all, I'd bet (there is not sufficient information to be _sure_ from the output you've provided) that you don't have the necessary 1Gb free on your disk slice to

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-09 Thread Robert White
On 12/09/2014 03:48 PM, Robert White wrote: On 12/09/2014 02:29 PM, Patrik Lundquist wrote: (stuff depicting a nearly full file system). Having taken another look at it all, I'd bet (there is not sufficient information to be _sure_ from the output you've provided) that you don't have

Re: PROBLEM: #89121 BTRFS mixes up mounted devices with their snapshots

2014-12-08 Thread Robert White
On 12/07/2014 04:32 PM, Konstantin wrote: I know this and I'm using 0.9 on purpose. I need to boot from these disks so I can't use 1.2 format as the BIOS wouldn't recognize the partitions. Having an additional non-RAID disk for booting introduces a single point of failure which contrary to the

Re: PROBLEM: #89121 BTRFS mixes up mounted devices with their snapshots

2014-12-08 Thread Robert White
On 12/08/2014 02:38 PM, Konstantin wrote: For more important systems there are high availability solutions which alleviate many of the problems you mention of but that's not the point here when speaking about the major bug in BTRFS which can make your system crash. I think you missed the part

Re: Why is the actual disk usage of btrfs considered unknowable?

2014-12-07 Thread Robert White
On 12/07/2014 07:15 AM, Shriramana Sharma wrote: IIUC: 1) btrfs fi df already shows the alloc-ed space and the space used out of that. 2) Despite snapshots, CoW and compression, the tree knows how many extents of data and metadata there are, and how many bytes on disk these occcupy, no matter

Re: Why is the actual disk usage of btrfs considered unknowable?

2014-12-07 Thread Robert White
On 12/07/2014 07:40 AM, Martin Steigerwald wrote: Well what would be possible I bet would be a kind of system call like this: I need to write 5 GB of data in 100 of files to /opt/mynewshinysoftware, can I do it *and* give me a guarentee I can. So like a more flexible fallocate approach as

Re: Why is the actual disk usage of btrfs considered unknowable?

2014-12-07 Thread Robert White
On 12/07/2014 10:20 PM, ashf...@whisperpc.com wrote: Martin, Excellent analysis. On 12/07/2014 07:40 AM, Martin Steigerwald wrote: So while the core problem isn't insoluble, in real life it is _not_ _worth_ _solving_. Your email quoting things is messed up... I wrote that analysis... 8-)

General Question: ctime, mtime, and xattrs

2014-12-05 Thread Robert White
So I was reading the wiki on the internal layout. The INODE description says st_ctime. Also updated when xattrs change. Why isn't changing the xattrs a modification (st_mtime) event? It just seems odd to me... -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body

Re: General Question: ctime, mtime, and xattrs

2014-12-05 Thread Robert White
... What a maroon. 8-) On 12/05/2014 03:03 PM, cwillu wrote: xattrs are commonly used to implement acls, which wouldn't typically be considered a content modification. On Dec 5, 2014 4:08 PM, Robert White rwh...@pobox.com wrote: So I was reading the wiki on the internal layout. The INODE description

Re: btrfs fi show not accepting mount path as arg?

2014-12-04 Thread Robert White
On 12/04/2014 05:50 AM, Shriramana Sharma wrote: [samjnaa:~] mount | grep btrfs /dev/sdb1 on /run/media/samjnaa/BRIHATII type btrfs (rw,nosuid,nodev,relatime,space_cache,uhelper=udisks2) [samjnaa:~] sudo btrfs fi show /run/media/samjnaa/BRIHATII/ Btrfs v3.17+20141103 [samjnaa:~] But the manpage

Re: btrfs df not really doing df?

2014-12-04 Thread Robert White
On 12/04/2014 05:51 AM, Shriramana Sharma wrote: IIUC df means disk free and is supposed to display the disk's (or partition's) free space -- but while btrfs fi df displays the allocated and used sizes, it doesn't actually display the total capacity of the devices, and subtract the allocated

Re: System/single + Metadata/single as leftover cruft of mkfs?

2014-12-04 Thread Robert White
I _think_ the extra single entries are the entries that deal with the disk/partition itself and therefore can not ever be distributed. So like the BTRFS signatures that say this is the third disk of this array (as opposed to the first, second, or fourth etc) and this is where my superblocks

Re: Crazy idea of cleanup the inode_record btrfsck things with SQL?

2014-12-03 Thread Robert White
FILEENTRIES_HPP /* Part of Underdog. https://underdog.sourceforge.net * Released under GPLv3. Other licenses may be available. * Robert White © Copyright 2014 rwh...@pobox.com */ #include dirent.h /* Defines DT_* constants */ #include fcntl.h #include stdio.h #include unistd.h #include stdlib.h #include

Re: Moving an entire subvol?

2014-12-02 Thread Robert White
On 12/02/2014 07:11 AM, Shriramana Sharma wrote: On Tue, Dec 2, 2014 at 6:58 PM, David Sterba dste...@suse.cz wrote: A subvolume is also a snapshotting barrier, so it's convenient to create subvolumes in well-known paths that contain data that should not be rolled back (/var/log, /srv,

Re: Possible to undo subvol delete?

2014-12-01 Thread Robert White
On 12/01/2014 08:40 AM, Shriramana Sharma wrote: IIUC you can only specify RO while creating but you can always cheaply create a RW snapshot of an RO one or an RO snapshot of an RW one... You can turn ReadOnly status on and off (er. true and false) with btrfs property get/set ro=true/false

Re: PROBLEM: #89121 BTRFS mixes up mounted devices with their snapshots

2014-12-01 Thread Robert White
On 12/01/2014 04:56 AM, MegaBrutal wrote: Since the other thread went off into theoretical debates about UUIDs and their generic relation to BTRFS, their everyday use cases, and the philosophical meaning behind uniqueness of copies and UUIDs; I'd like to specifically ask you to only post here

Re: Crazy idea of cleanup the inode_record btrfsck things with SQL?

2014-12-01 Thread Robert White
On 11/30/2014 10:18 PM, Qu Wenruo wrote: (advocacy for using SQL internally for btrfsck) All of these ideas you want to toss a entire SQL front end on are more simply handled with simple data structures. In C++ terms mapinode,parent and/or mapparent,vectorchildren beats the heck out of

Re: pro/cons of raid1 with mdadm/lvm2

2014-12-01 Thread Robert White
On 12/01/2014 01:26 AM, Gour wrote: On Mon, 01 Dec 2014 09:06:19 +1100 Russell Coker russ...@coker.com.au wrote: When the 2 disks have different data mdadm has no way of knowing which one is correct and has a 50% chance of overwriting good data. But BTRFS does checksums on all reads and solves

Re: btrfs stuck with lot's of files

2014-12-01 Thread Robert White
On 12/01/2014 03:46 AM, Peter Volkov wrote: Hi, guys. (stuff about getting hung up trying to write to one drive) That drive (/dev/sdn) is probably starting to fail. Some older drives basically go unresponsive when they start to go bad. Particularly if they've gone bad enough to have run out

Re: Online Drive Replacement: BTRFS with RAID 6

2014-12-01 Thread Robert White
On 12/01/2014 06:47 AM, Oliver wrote: Hi All, on a testing machine I installed four HDDs and they are configured as RAID6. For a test I removed one of the drives (/dev/sdk) while the volume was mounted and data was written to it. This worked well, as far as I can see. Some I/O errors were

Re: PROBLEM: #89121 BTRFS mixes up mounted devices with their snapshots

2014-12-01 Thread Robert White
On 12/01/2014 02:10 PM, MegaBrutal wrote: Since having duplicate UUIDs on devices is not a problem for me since I can tell them apart by LVM names, the discussion is of little relevance to my use case. Of course it's interesting and I like to read it along, it is not about the actual problem at

Re: Crazy idea of cleanup the inode_record btrfsck things with SQL?

2014-11-30 Thread Robert White
On 11/30/2014 05:58 PM, Qu Wenruo wrote: (why not use SQL to... suggestion) SQL, as in Structured Query Language, is _terrible_ for recursion. It expresses all of its elements in terms of set theory and really can only implement union and intersection of flat sets. Several companies offer

  1   2   >