What if TRIM issued a wipe on devices that don't TRIM?

2018-12-05 Thread Robert White
(1) Automatic and selective wiping of unused and previously used disk blocks is a good security measure, particularly when there is an encryption layer beneath the file system. (2) USB attached devices _never_ support TRIM and they are the most likely to fall into strangers hands. (3) I vagu

Re: [PATCH 00/10] btrfs: Support for DAX devices

2018-12-05 Thread Robert White
On 12/5/18 9:37 PM, Jeff Mahoney wrote: The high level idea that Jan Kara and I came up with in our conversation at Labs conf is pretty expensive.  We'd need to set a flag that pauses new page faults, set the WP bit on affected ranges, do the snapshot, commit, clear the flag, and wake up the wa

Feature Request: Directory property to upconvert mkdir/rmdir to subvol create/delete

2019-05-17 Thread Robert White
Howdy, For several reasons it would be really convenient if there was a way to mark a btrfs directory such that the directories created in the marked directory would actually be automatically converted to subvolume creation and destruction. NFS4 particularly pivots on file system boundaries,

Re: Unbootable root btrfs

2019-05-17 Thread Robert White
On 5/18/19 4:06 AM, Chris Murphy wrote: On Fri, May 17, 2019 at 2:18 AM Lee Fleming wrote: I didn't see that particular warning. I did see a warning that it could cause damage and should be tried after trying some other things which I did. The data on this drive isn't important. I just wante

Stupid (?) Idea about extent lifetimes.

2016-03-15 Thread Robert White
It occurs to me that it would be desirable to mark extents as "least favoured nations" and so all new writes would like to not be written there and any data written there would have a desire to be somewhere else. So lets say the wholly unallocated space has a natural status of 100. Allocated b

A good "Boot Maintenance" scheme (WAS: New file system with same issue)

2016-03-18 Thread Robert White
On 03/14/2016 01:13 PM, Marc Haber wrote: This was not asked, and I didn't try. Since this is an encrypted root filesystem, is it a workable way to add clear_cache to /etc/fstab, rebuild initramfs and reboot? Or do you recommend using a rescue system? You should be able to boot to single user

Theoretical Question about commit=n

2017-11-12 Thread Robert White
Is the commit interval monotonic, or is it seconds after sync? What I mean is that if I manually call sync(2) does the commit timer reset? I'm thinking it does not, but I can imagine a workload where it ideally would. (Again, this is purely theoretical, I have no such workload as I am about to de

BUG: Replacing a directory with a subvolume breaks incremental snapshots

2014-05-23 Thread Robert White
Howdy, If you remove an existing directory and then create a subvolume with the same name the incremental send (btrfs send -p) will die with errno==2 (file not found). Steps to Reproduce: btrfs subvol create scratch # make a playground mkdir scratch/example btrfs subvol snap -r scratch scrat

Feature Question: stored mount options... is that in there?

2014-06-05 Thread Robert White
Is there a way to store the default mount options in the file system a la ext4 and tune2fs? In particular my initrd mounts the file system and then the system remounts it using fstab, but this tends to lead to rebuilds of the inode cache. The initrd doesn't have the inode_cache option but the

hung task timer + btrfs_convert or btrfs balance = OOPS

2014-04-23 Thread Robert White
The first mount of a non-trivial file system after a btrfs_convert, or an ongoing btrfs balance operation containing large files may lead to an oops (and a pathologically damaged file system) if the hang check timer (CONFIG_DETECT_HUNG_TASK=y) is compiled into the linux kernel and not disabled.

Cycle of send/receive for backup/restore is incomplete...

2014-04-23 Thread Robert White
So the backup/restore system described using snapshots is incomplete because the final restore is a copy operation. As such, the act of restoring from the backup will require restarting the entire backup cycle because the copy operation will scramble the metadata consanguinity. The real choice

Re: hung task timer + btrfs_convert or btrfs balance = OOPS

2014-04-23 Thread Robert White
On 04/23/2014 02:12 PM, Marc MERLIN wrote: On Wed, Apr 23, 2014 at 01:44:21PM -0700, Robert White wrote: As it is, the two options are not happy together. Be sure to echo 0 > /proc/sys/kernel/hung_task_timeout_secs to disable the timer before doing a mount or balance after a But this o

Re: Cycle of send/receive for backup/restore is incomplete...

2014-04-24 Thread Robert White
I am, then, quite confused... On 04/23/2014 09:28 PM, Brendan Hide wrote: Replied inline: btrfs doesn't differentiate snapshots and subvolumes. They're the same first-class citizen. A snapshot is a subvolume that just happens to have some data (automagically/naturally) deduplicated with another

Re: btrfs fi show not accepting mount path as arg?

2014-12-04 Thread Robert White
On 12/04/2014 05:50 AM, Shriramana Sharma wrote: [samjnaa:~] mount | grep btrfs /dev/sdb1 on /run/media/samjnaa/BRIHATII type btrfs (rw,nosuid,nodev,relatime,space_cache,uhelper=udisks2) [samjnaa:~] sudo btrfs fi show /run/media/samjnaa/BRIHATII/ Btrfs v3.17+20141103 [samjnaa:~] But the manpage

Re: btrfs df not really doing df?

2014-12-04 Thread Robert White
On 12/04/2014 05:51 AM, Shriramana Sharma wrote: IIUC df means "disk free" and is supposed to display the disk's (or partition's) free space -- but while btrfs fi df displays the allocated and used sizes, it doesn't actually display the total capacity of the devices, and subtract the allocated si

Re: System/single + Metadata/single as leftover cruft of mkfs?

2014-12-04 Thread Robert White
I _think_ the "extra" single entries are the entries that deal with the disk/partition itself and therefore can not ever be distributed. So like the BTRFS signatures that say "this is the third disk of this array" (as opposed to the first, second, or fourth etc) and "this is where my superbloc

General Question: ctime, mtime, and xattrs

2014-12-05 Thread Robert White
So I was reading the wiki on the internal layout. The INODE description says "st_ctime. Also updated when xattrs change." Why isn't changing the xattrs a modification (st_mtime) event? It just seems odd to me... -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the bo

Re: General Question: ctime, mtime, and xattrs

2014-12-05 Thread Robert White
cause to think about it again... Never mind. /sigh... What a maroon. 8-) On 12/05/2014 03:03 PM, cwillu wrote: xattrs are commonly used to implement acls, which wouldn't typically be considered a content modification. On Dec 5, 2014 4:08 PM, "Robert White" wrote: So I was reading

Re: Why is the actual disk usage of btrfs considered unknowable?

2014-12-07 Thread Robert White
On 12/07/2014 07:15 AM, Shriramana Sharma wrote: IIUC: 1) btrfs fi df already shows the alloc-ed space and the space used out of that. 2) Despite snapshots, CoW and compression, the tree knows how many extents of data and metadata there are, and how many bytes on disk these occcupy, no matter w

Re: Why is the actual disk usage of btrfs considered unknowable?

2014-12-07 Thread Robert White
On 12/07/2014 07:40 AM, Martin Steigerwald wrote: Well what would be possible I bet would be a kind of system call like this: I need to write 5 GB of data in 100 of files to /opt/mynewshinysoftware, can I do it *and* give me a guarentee I can. So like a more flexible fallocate approach as fallo

Re: Why is the actual disk usage of btrfs considered unknowable?

2014-12-07 Thread Robert White
On 12/07/2014 10:20 PM, ashf...@whisperpc.com wrote: Martin, Excellent analysis. On 12/07/2014 07:40 AM, Martin Steigerwald wrote: So while the core problem isn't insoluble, in real life it is _not_ _worth_ _solving_. Your email quoting things is messed up... I wrote that analysis... 8-)

Re: PROBLEM: #89121 BTRFS mixes up mounted devices with their snapshots

2014-12-08 Thread Robert White
On 12/07/2014 04:32 PM, Konstantin wrote: I know this and I'm using 0.9 on purpose. I need to boot from these disks so I can't use 1.2 format as the BIOS wouldn't recognize the partitions. Having an additional non-RAID disk for booting introduces a single point of failure which contrary to the id

Re: [PATCH V2][BTRFS-PROGS] Don't use LVM snapshot device

2014-12-08 Thread Robert White
On 12/08/2014 09:36 AM, Goffredo Baroncelli wrote: I like this approach, but as I wrote before, it seems that initramfs executes a "btrfs dev scan" (see my previoue email 'Re: PROBLEM: #89121 BTRFS mixes up mounted devices with their snapshots' date 2014/12/03 9:34): Roll your own for now.

Re: PROBLEM: #89121 BTRFS mixes up mounted devices with their snapshots

2014-12-08 Thread Robert White
On 12/08/2014 02:38 PM, Konstantin wrote: For more important systems there are high availability solutions which alleviate many of the problems you mention of but that's not the point here when speaking about the major bug in BTRFS which can make your system crash. I think you missed the part w

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-09 Thread Robert White
On 12/09/2014 02:29 PM, Patrik Lundquist wrote: Label: none uuid: 770fe01d-6a45-42b9-912e-e8f8b413f6a4 Total devices 1 FS bytes used 1.35TiB devid1 size 2.73TiB used 1.36TiB path /dev/sdc1 Data, single: total=1.35TiB, used=1.35TiB System, single: total=32.00MiB, used=112.00KiB Me

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-09 Thread Robert White
On 12/09/2014 02:29 PM, Patrik Lundquist wrote: Data, single: total=1.35TiB, used=1.35TiB System, single: total=32.00MiB, used=112.00KiB Metadata, single: total=3.00GiB, used=1.55GiB GlobalReserve, single: total=512.00MiB, used=0.00B P.S. you should re-balance your System and Metadata as "DUP"

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-09 Thread Robert White
On 12/09/2014 02:29 PM, Patrik Lundquist wrote: > (stuff depicting a nearly full file system). Having taken another look at it all, I'd bet (there is not sufficient information to be _sure_ from the output you've provided) that you don't have the necessary 1Gb free on your disk slice to allocat

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-09 Thread Robert White
On 12/09/2014 03:48 PM, Robert White wrote: On 12/09/2014 02:29 PM, Patrik Lundquist wrote: > (stuff depicting a nearly full file system). Having taken another look at it all, I'd bet (there is not sufficient information to be _sure_ from the output you've provided) that you d

Re: [PATCH] Btrfs: get more accurate output in fd command.

2014-12-10 Thread Robert White
On 12/09/2014 05:08 PM, Dongsheng Yang wrote: On 12/10/2014 02:47 AM, Goffredo Baroncelli wrote: Hi Dongsheng On 12/09/2014 12:20 PM, Dongsheng Yang wrote: When function btrfs_statfs() calculate the tatol size of fs, it is calculating the total size of disks and then dividing it by a factor. Bu

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-10 Thread Robert White
On 12/09/2014 11:19 PM, Patrik Lundquist wrote: On 10 December 2014 at 00:13, Robert White wrote: On 12/09/2014 02:29 PM, Patrik Lundquist wrote: Label: none uuid: 770fe01d-6a45-42b9-912e-e8f8b413f6a4 Total devices 1 FS bytes used 1.35TiB devid1 size 2.73TiB used 1.36TiB

Re: [PATCH] Btrfs: get more accurate output in fd command.

2014-12-10 Thread Robert White
On 12/10/2014 05:21 AM, Duncan wrote: Robert White posted on Wed, 10 Dec 2014 02:53:40 -0800 as excerpted: On 12/09/2014 05:08 PM, Dongsheng Yang wrote: On 12/10/2014 02:47 AM, Goffredo Baroncelli wrote: Hi Dongsheng On 12/09/2014 12:20 PM, Dongsheng Yang wrote: When function btrfs_statfs

mkfs.btrfs limits "odd" [and maybe a "failed" phantom device?]

2014-12-10 Thread Robert White
So I started looking at the mkfs.btrfs manual page with an eye towards documenting some of the tidbits like metadata automatically switching from dup to raid1 when more than one device is used. In experimenting I ended up with some questions... (1) why is the dup profile for data restricted to

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-10 Thread Robert White
On 12/10/2014 10:56 AM, Patrik Lundquist wrote: On 10 December 2014 at 14:11, Duncan <1i5t5.dun...@cox.net> wrote: Assuming no snapshots still contain the file, of course, and that the ext* saved subvolume has already been deleted. Got no snapshots or subvolumes. Keeping it simple for now. D

Re: Possibility to have a "transient" snapshot?

2014-12-10 Thread Robert White
On 12/10/2014 11:52 AM, James West wrote: I was just looking into using overlayfs, and although it has some promise, I think it's biggest drawback is the upperdir will have to be some sort of storage backed filesystem. From my limited understanding of tmpfs, it's not supposed to be the greatest w

Re: Balance & scrub & defrag

2014-12-10 Thread Robert White
On 12/10/2014 02:15 PM, sys.syphus wrote: I am working on a script that i can run daily that will do maintenance on my btrfs mountpoints. is there any reason not to concurrently do all of the above? possibly including discards as well. also, is there anything existing currently that will do mai

Re: out of space warning?

2014-12-10 Thread Robert White
On 12/10/2014 02:54 PM, sys.syphus wrote: I would like to avoid running out of space. is there a way to know that I am getting close? i'd like to make a script that runs as part of my bash prompt and lets me know when i am getting close. i know there are several ways you can run out of space and

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-11 Thread Robert White
On 12/10/2014 05:36 AM, Patrik Lundquist wrote: On 10 December 2014 at 13:17, Robert White wrote: On 12/09/2014 11:19 PM, Patrik Lundquist wrote: BUT FIRST UNDERSTAND: you do _not_ need to balance a newly converted filesystem. That is, the recommended balance (and recursive defrag) is _not_

Re: out of space warning?

2014-12-11 Thread Robert White
On 12/11/2014 01:01 AM, Erkki Seppala wrote: Robert White writes: You don't check your car's gas tank every time you put your foot on the brake, you don't want to check your free space every time your system finishes every tiny command you type. Well, actually my car makes a

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

2014-12-11 Thread Robert White
So far I don't see a "bug". On 12/11/2014 12:18 AM, Patrik Lundquist wrote: I'll reboot the thread with a recap and my latest findings. * Half full 3TB disk converted from ext4 to Btrfs, after first verifying it with fsck. * Undo subvolume deleted after being happy with the conversion. * Recurs

Re: Fixing Btrfs Filesystem Full Problems typo?

2014-12-11 Thread Robert White
On 12/11/2014 01:55 AM, Patrik Lundquist wrote: On 11 December 2014 at 09:42, Robert White wrote: On 12/10/2014 05:36 AM, Patrik Lundquist wrote: On 10 December 2014 at 13:17, Robert White wrote: On 12/09/2014 11:19 PM, Patrik Lundquist wrote: BUT FIRST UNDERSTAND: you do _not_ need

A note on spotting "bugs" [Was: ENOSPC after conversion]

2014-12-11 Thread Robert White
On 12/11/2014 12:18 AM, Patrik Lundquist wrote: * Full balance, that ended with "98 enospc errors during balance." Assuming that quote is an actual quote from the output of the balance... We can strongly infer that this sort of occurrence is expected since there is code to keep track of it an

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

2014-12-11 Thread Robert White
On 12/11/2014 03:01 PM, Patrik Lundquist wrote: On 11 December 2014 at 11:18, Robert White wrote: So far I don't see a "bug". Fair enough, lets call it a huge problem with btrfs convert. I think it warrants a note in the wiki. On 12/11/2014 12:18 AM, Patrik Lundquist w

Re: ENOSPC after conversion [Was: Fixing Btrfs Filesystem Full Problems typo?]

2014-12-11 Thread Robert White
TL;DR version On 12/11/2014 03:01 PM, Patrik Lundquist wrote: Of course the filesystem is in a problematic state after the conversion, even if it's not a bug. ~1.5TB of free space and yet out of space and it can't be fixed with a balance. It might not be wrong per se but it's very problematic fr

Re: Balance & scrub & defrag

2014-12-11 Thread Robert White
On 12/11/2014 05:00 PM, Russell Coker wrote: On Wed, 10 Dec 2014 17:17:28 Robert White wrote: A _monthly_ scrub is maybe worth scheduling if you have a lot of churn in your disk contents. I do weekly scrubs. I recently had 2 disks in a RAID-1 array develop read errors within a month of each

Re: mkfs.btrfs limits "odd" [and maybe a "failed" phantom device?]

2014-12-11 Thread Robert White
On 12/11/2014 07:56 PM, Zygo Blaxell wrote: On Wed, Dec 10, 2014 at 02:18:55PM -0800, Robert White wrote: (3) why can I make a raid5 out of two devices? (I understand that we are currently just making mirrors, but the standard requires three devices in the geometry etc. So I would expect a two

Re: mkfs.btrfs limits "odd" [and maybe a "failed" phantom device?]

2014-12-12 Thread Robert White
On 12/12/2014 01:06 AM, David Taylor wrote: The above quote is discussing two device RAID5, you are discussing three device RAID5. Heresy! (yes, some humor is required here.) There is no such thing as a "two device RAID5". That's what RAID1 is for. Saying "The above quote is discussing a two

Re: A note on spotting "bugs" [Was: ENOSPC after conversion]

2014-12-12 Thread Robert White
On 12/11/2014 10:42 PM, Patrik Lundquist wrote: On 11 December 2014 at 23:00, Robert White wrote: On 12/11/2014 12:18 AM, Patrik Lundquist wrote: * Full balance, that ended with "98 enospc errors during balance." Assuming that quote is an actual quote from the output of the balan

Re: Balance & scrub & defrag

2014-12-12 Thread Robert White
On 12/12/2014 01:17 AM, Erkki Seppala wrote: Robert White writes: You need to buy better disks. 8-) Where can one buy these better disks with reasonable prices?-) Disks are best thought of as consumables. A good disk is only about 9% more expensive. So like the WD "green" disk

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-12 Thread Robert White
On 12/12/2014 06:37 AM, Tomasz Chmielewski wrote: FYI, still seeing this with 3.18 (scrub passes fine on this filesystem). # time btrfs balance start /mnt/lxc2 Segmentation fault real322m32.153s user0m0.000s sys 16m0.930s (...) [20306.981773] BTRFS (device sdd1): parent transi

Re: mkfs.btrfs limits "odd" [and maybe a "failed" phantom device?]

2014-12-12 Thread Robert White
On 12/12/2014 08:45 AM, Zygo Blaxell wrote: On Thu, Dec 11, 2014 at 10:01:06PM -0800, Robert White wrote: So RAID5 with three media M is MMM MMM D1 D2 P(a) D3 P(b) D4 P(c) D5 D6 RAID5 with two media is well defined, and looks like this: MMM D1 P(a) P(b) D2 D3 P(c

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-12 Thread Robert White
On 12/12/2014 01:46 PM, Tomasz Chmielewski wrote: On 2014-12-12 22:36, Robert White wrote: In another thread [that was discussing SMART] you talked about replacing a drive and then needing to do some patching-up of the result because of drive failures. Is this the same filesystem where that

RAID0 extent sizes?

2014-12-12 Thread Robert White
I've seen it mentioned here that generally data extents are 1G and metadata extents are 256M. Is that per-drive or per-stripe in the case of RAID0? That is, if I have data mode raid0 across N drives does the system allocate one 1G extent on each drive making the full stripe allocation N-gigs;

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-12 Thread Robert White
On 12/12/2014 02:46 PM, Tomasz Chmielewski wrote: On 2014-12-12 23:34, Robert White wrote: On 12/12/2014 01:46 PM, Tomasz Chmielewski wrote: On 2014-12-12 22:36, Robert White wrote: In another thread [that was discussing SMART] you talked about replacing a drive and then needing to do some

Re: RAID0 extent sizes?

2014-12-12 Thread Robert White
On 12/12/2014 02:59 PM, Hugo Mills wrote: On Fri, Dec 12, 2014 at 02:54:24PM -0800, Robert White wrote: I've seen it mentioned here that generally data extents are 1G and metadata extents are 256M. Is that per-drive or per-stripe in the case of RAID0? That is, if I have data mode raid0 a

Re: A note on spotting "bugs" [Was: ENOSPC after conversion]

2014-12-12 Thread Robert White
On 12/12/2014 05:12 PM, Duncan wrote: Robert White posted on Fri, 12 Dec 2014 05:29:58 -0800 as excerpted: This still doesnt say _anything_ is wrong with your filesystem except that it doesn't have enough _raw_ space to create a 2-ish gig extent. What's wrong with the filesyst

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-13 Thread Robert White
On 12/13/2014 12:16 AM, Tomasz Chmielewski wrote: On 2014-12-12 23:58, Robert White wrote: I don't have the history to answer this definitively, but I don't think you have a choice. Nothing else is going to touch that error. I have not seen any "oh my god, btrfsck just at

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-13 Thread Robert White
On 12/13/2014 05:53 AM, Tomasz Chmielewski wrote: My usage case is quite simple: - skinny extents, extended inode refs okay - mount compress-force=zlib I'd, personally, "never" force compression. This can increase the size of files by five or more percent if it is an inherently incompressibl

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-13 Thread Robert White
On 12/13/2014 01:52 PM, Tomasz Chmielewski wrote: On 2014-12-13 21:54, Robert White wrote: - rsync many remote data sources (-a -H --inplace --partial) + snapshot Using --inplace on a Copy On Write filesystem has only one effect, it increases fragmentation... a lot... ...if the file was

Re: subvolume / folder compression flag

2014-12-13 Thread Robert White
On 12/13/2014 06:59 PM, Ali AlipourR wrote: 2- ... and rsync files without compression flag ... The --compress flag for rsync has nothing to do with how the files are stored on either end. It determines whether the data is compressed as it passes from the source rsync to the destination rsyn

Re: 3.18.0: kernel BUG at fs/btrfs/relocation.c:242!

2014-12-14 Thread Robert White
On 12/13/2014 03:56 PM, Robert White wrote: ... Dangit... On re-reading I think I was still less than optimally clear. I kept using the word "resent" when I should have been using a word like "re-written" or "re-stored" (as opposed to "restored"). On

Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.

2014-12-14 Thread Robert White
On 12/14/2014 05:21 PM, Dongsheng Yang wrote: Does it make sense to you? I understood what you were saying but it didn't make sense to me... As there are 2 complaints for the same change of @size in df, I have to say it maybe not so easy to understand. Anyone have some suggestion about it

Re: btrfs receive being very slow

2014-12-14 Thread Robert White
On 12/14/2014 08:50 PM, Nick Dimov wrote: Hello everyone! First, thanks for amazing work on btrfs filesystem! Now the problem: I use a ssd as my system drive (/dev/sda2) and use daily snapshots on it. Then, from time to time, i sync those on HDD (/dev/sdb4) by using btrfs send / receive like th

Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.

2014-12-14 Thread Robert White
On 12/14/2014 10:06 PM, Robert White wrote: On 12/14/2014 05:21 PM, Dongsheng Yang wrote: Anyone have some suggestion about it? (... strong advocacy for raw numbers...) Concise Example to attempt to be clearer: /dev/sda == 1TiB /dev/sdb == 2TiB /dev/sdc == 3TiB /dev/sdd == 3TiB mkfs.btrfs

Re: btrfs receive being very slow

2014-12-15 Thread Robert White
On 12/14/2014 11:41 PM, Nick Dimov wrote: Hi, thanks for the answer, I will answer between the lines. On 15.12.2014 08:45, Robert White wrote: On 12/14/2014 08:50 PM, Nick Dimov wrote: Hello everyone! First, thanks for amazing work on btrfs filesystem! Now the problem: I use a ssd as my

Re: [PATCH v2 1/3] Btrfs: get more accurate output in df command.

2014-12-15 Thread Robert White
On 12/15/2014 12:26 AM, Dongsheng Yang wrote: On 12/15/2014 03:49 PM, Robert White wrote: On 12/14/2014 10:06 PM, Robert White wrote: On 12/14/2014 05:21 PM, Dongsheng Yang wrote: Anyone have some suggestion about it? (... strong advocacy for raw numbers...) Hi Robert, thanx for your so

Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-15 Thread Robert White
On 12/15/2014 01:36 AM, Robert White wrote: So we don't just hand-wave over statfs(). We include the dev_item.bytes_excluded in the superblock and we decide once-and-for-all (with any geometry creation, or completed conversion) how many bytes just _can't_ be reached but only once we _

Re: [PATCH 6/7] Print the summary

2014-12-15 Thread Robert White
On 12/15/2014 05:58 PM, Duncan wrote: * Please s/disk/device/, here and possibly elsewhere. I know I'm not the only one who is trying to make the switch in my own usage, as it looks a bit foolish (and/or marks the user as an old fogey who's likely to start lecturing about how a GiB isn't "small"

Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-15 Thread Robert White
On 12/15/2014 07:30 PM, Robert White wrote: The above would be ideal. But POSIX says "no". f_blocks is defined (only Correction the linux kernel says "total data blocks", POSIX says "total blocks" -- it was a mental typo... 8-) in the comments) as "total data

Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-16 Thread Robert White
On 12/16/2014 03:30 AM, Dongsheng Yang wrote: Hi Robert, thanx for your proposal about this. IMHO, output of df command shoud be more friendly to user. Well, I think we have a disagreement on this point, let's take a look at what the zfs is doing. /dev/sda7- 10G /dev/sda8- 10G # zpool create m

Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-17 Thread Robert White
I don't disagree with the _ideal_ of your patch. I just think that it's impossible to implement it without lying to the user or making things just as bad in a different way. I would _like_ you to be right. But my thing is finding and quantifying failure cases and the entire question is full of

Re: [PATCH 6/7] Print the summary

2014-12-17 Thread Robert White
On 12/16/2014 01:05 AM, Hugo Mills wrote: On Mon, Dec 15, 2014 at 07:47:06PM -0800, Robert White wrote: I prefer "slice", not that I am totally happy with that word either. But by the time you get through loopback devices, memory map devices, the "device files" tha

Re: Oddly slow read performance with near-full largish FS

2014-12-20 Thread Robert White
On 12/19/2014 09:33 AM, Duncan wrote: Charles Cazabon posted on Fri, 19 Dec 2014 10:58:49 -0600 as excerpted: This configuration is one I've been using for many years. It's only recently that I've noticed it being particularly slow with btrfs -- I don't know if that's because the filesystem ha

Re: Oddly slow read performance with near-full largish FS

2014-12-20 Thread Robert White
On 12/16/2014 06:42 PM, Charles Cazabon wrote: Hi, I've been running btrfs for various filesystems for a few years now, and have recently run into problems with a large filesystem becoming *really* slow for basic reading. None of the debugging/testing suggestions I've come across in the wiki or

Re: btrfs is using 25% more disk than it should

2014-12-20 Thread Robert White
On 12/19/2014 01:17 PM, Josef Bacik wrote: tl;dr: Cow means you can in the worst case end up using 2 * filesize - blocksize of data on disk and the file will appear to be filesize. Thanks, Doesn't the worst case more like N^log(N) (when N is file in blocksize) in the pernicious case? Stagge

Re: btrfs is using 25% more disk than it should

2014-12-20 Thread Robert White
On 12/20/2014 03:39 AM, Josef Bacik wrote: On 12/20/2014 06:23 AM, Robert White wrote: On 12/19/2014 01:17 PM, Josef Bacik wrote: tl;dr: Cow means you can in the worst case end up using 2 * filesize - blocksize of data on disk and the file will appear to be filesize. Thanks, Doesn'

Re: btrfs is using 25% more disk than it should

2014-12-20 Thread Robert White
On 12/19/2014 01:10 PM, Josef Bacik wrote: On 12/18/2014 09:59 AM, Daniele Testa wrote: Hey, I am hoping you guys can shed some light on my issue. I know that it's a common question that people see differences in the "disk used" when running different calculations, but I still think that my iss

Re: Oddly slow read performance with near-full largish FS

2014-12-21 Thread Robert White
On 12/21/2014 08:32 AM, Charles Cazabon wrote: Hi, Robert, Thanks for the response. Many of the things you mentioned I have tried, but for completeness: Have you taken SMART (smartmotools etc) to these disks There are no errors or warnings from SMART for the disks. Do make sure you are re

Re: Uncorrectable errors on RAID-1?

2014-12-21 Thread Robert White
On 12/21/2014 11:34 AM, constantine wrote: Some months ago I had 6 uncorrectable errors. I deleted the files that contained them and then after scrubbing I had 0 uncorrectable errors. After some weeks I encountered new uncorrectable errors. Question 1: Why do I have uncorrectable errors on a RAI

Re: Oddly slow read performance with near-full largish FS

2014-12-21 Thread Robert White
On 12/21/2014 02:53 PM, Charles Cazabon wrote: Hi, Robert, My performance issues with btrfs are more-or-less resolved now -- the performance under btrfs still seems quite variable compared to other filesystems -- my rsync speed is now varying between 40MB and ~90MB/s, with occasional intervals w

Re: Can BTRFS handle XATTRs larger than 4K?

2014-12-22 Thread Robert White
On 12/22/2014 12:44 PM, Austin S Hemmelgarn wrote: On 2014-12-22 15:06, Richard Sharpe wrote: On Mon, Dec 22, 2014 at 10:43 AM, Chris Murphy wrote: On Mon, Dec 22, 2014 at 11:09 AM, Austin S Hemmelgarn wrote: Personally, I'd love to see unlimited length xattr's like NTFS and HFS+ do, as tha

Re: Can BTRFS handle XATTRs larger than 4K?

2014-12-22 Thread Robert White
So I'll ask again... On 12/22/2014 03:15 PM, ronnie sahlberg wrote: On Mon, Dec 22, 2014 at 2:52 PM, Robert White wrote: So skipping the full ADS, what's the current demand/payoff for large XATTR space? So skipping the full ADS, what is the current demand/payoff

Re: Can BTRFS handle XATTRs larger than 4K?

2014-12-22 Thread Robert White
On 12/22/2014 02:55 PM, Richard Sharpe wrote: On Mon, Dec 22, 2014 at 2:52 PM, Robert White wrote: So skipping the full ADS, what's the current demand/payoff for large XATTR space? Windows Security Descriptors (sometimes incorrectly called ACLs) stored by Samba. Ah. I know that

Re: Can BTRFS handle XATTRs larger than 4K?

2014-12-22 Thread Robert White
On 12/22/2014 03:58 PM, Richard Sharpe wrote: On Mon, Dec 22, 2014 at 3:55 PM, Robert White wrote: So I'll ask again... On 12/22/2014 03:15 PM, ronnie sahlberg wrote: On Mon, Dec 22, 2014 at 2:52 PM, Robert White wrote: So skipping the full ADS, what's the current demand/payoff

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-26 Thread Robert White
On 12/26/2014 05:37 AM, Martin Steigerwald wrote: Hello! First: Have a merry christmas and enjoy a quiet time in these days. Second: At a time you feel like it, here is a little rant, but also a bug report: I have this on 3.18 kernel on Debian Sid with BTRFS Dual SSD RAID with space_cache, ski

Re: Standards Problems [Was: [PATCH v2 1/3] Btrfs: get more accurate output in df command.]

2014-12-26 Thread Robert White
On 12/23/2014 04:31 AM, Dongsheng Yang wrote: On 12/18/2014 12:07 PM, Robert White wrote: I don't disagree with the _ideal_ of your patch. I just think that it's impossible to implement it without lying to the user or making things just as bad in a different way. I would _like_ you t

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 02:54 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills: On Sat, Dec 27, 2014 at 10:01:17AM +0100, Martin Steigerwald wrote: Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White: On 12/26/2014 05:37 AM, Martin Steigerwald wrote

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 03:11 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills: I only see the lockups of BTRFS is the trees *occupy* all space on the device. No, "the trees" occupy 3.29 GiB of your 5 GiB of mirrored metadata space. What's more, balance does

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 05:16 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 03:52:56 schrieb Robert White: My theory from watching the Windows XP defragmentation case is this: - For writing into the file BTRFS needs to actually allocate and use free space in the current tree allocation

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 05:16 AM, Martin Steigerwald wrote: It can easily be reproduced without even using Virtualbox, just by a nice simple fio job. TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file... #!/bin/bash # not tested, so correct any syntax errors

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 06:00 AM, Robert White wrote: On 12/27/2014 05:16 AM, Martin Steigerwald wrote: It can easily be reproduced without even using Virtualbox, just by a nice simple fio job. TL;DR: If you want a worst-case example of consuming a BTRFS filesystem with one single file... #!/bin/bash

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
Virtualbox for reproducing the issue. Next I will try to reproduce with a freshly creating filesystem. Am Samstag, 27. Dezember 2014, 09:30:43 schrieb Hugo Mills: On Sat, Dec 27, 2014 at 10:01:17AM +0100, Martin Steigerwald wrote: Am Freitag, 26. Dezember 2014, 14:48:38 schrieb Robert White: On

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 06:21 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 15:14:05 schrieb Martin Steigerwald: Am Samstag, 27. Dezember 2014, 06:00:48 schrieb Robert White: On 12/27/2014 05:16 AM, Martin Steigerwald wrote: It can easily be reproduced without even using Virtualbox, just

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
Semi off-topic questions... On 12/27/2014 08:26 AM, Hugo Mills wrote: This is... badly mistaken, at best. The problem of where to write a file into a set of free extents is definitely *not* an NP-hard problem. It's a P problem, with an O(n log n) solution, where n is the number of free exten

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 08:01 AM, Martin Steigerwald wrote: From how you write I get the impression that you think everyone else beside you is just silly and dumb. Please stop this assumption. I may not always get terms right, and I may make a mistake as with the wrong df figure. But I also highly dislike

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-27 Thread Robert White
On 12/27/2014 05:01 PM, Bardur Arantsson wrote: On 2014-12-28 01:25, Robert White wrote: On 12/27/2014 08:01 AM, Martin Steigerwald wrote: From how you write I get the impression that you think everyone else beside you is just silly and dumb. Please stop this assumption. I may not always get

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-28 Thread Robert White
On 12/28/2014 04:07 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White: Now: The complaining party has verified the minimum, repeatable case of simple file allocation on a very fragmented system and the responding party and several others have understood

Re: BTRFS free space handling still needs more work: Hangs again

2014-12-28 Thread Robert White
On 12/28/2014 07:42 AM, Martin Steigerwald wrote: Am Sonntag, 28. Dezember 2014, 06:52:41 schrieb Robert White: On 12/28/2014 04:07 AM, Martin Steigerwald wrote: Am Samstag, 27. Dezember 2014, 20:03:09 schrieb Robert White: Now: The complaining party has verified the minimum, repeatable case

Re: fstrim not working on one of three BTRFS filesystems

2014-12-28 Thread Robert White
On 12/28/2014 08:58 AM, Martin Steigerwald wrote: Hi! After my recent tests with my /home filesystem and the up and downsizing of it I get: merkaba:~> LANG=C fstrim -v /home /home: 0 B (0 bytes) trimmed merkaba:~> LANG=C fstrim -v / /: 24.5 GiB (26257555456 bytes) trimmed merkaba:~> LANG=C fst

Re: Btrfs subvolume question

2015-02-04 Thread Robert White
On 02/04/2015 06:27 PM, Chris Murphy wrote: On Wed, Feb 4, 2015 at 3:54 PM, Markus Moeller wrote: Hi , I am new to btrfs and wonder what I need to do to move subvolumes to the right filesystem. I see the following: df -h Filesystem Size Used Avail Use% Mounted on /d

Re: scrub problem

2015-02-22 Thread Robert White
On 02/20/2015 01:03 PM, Bob Williams wrote: /var/lib/btrfs/scrub.status.0b07b829-9a0e-44ab-89ee-14b36a45199e (the last bit of the filename is the filesystem uuid) Look for a line that ends with "finished:0" and change it to say "finished:1" Why does this data item even exist? The filesystem/k

  1   2   >