Re: Untar on empty partition returns ENOSPACE

[Cerem Cem ASLAN]

You don't even need dd_rescue or something for RaspberryPi creation,
you may use 
https://github.com/aktos-io/dcs-tools#cloning-a-target-into-a-new-target
Jean-Denis Girard , 20 Eki 2018 Cmt, 22:00
tarihinde şunu yazdı:
>
> Le 17/10/2018 à 20:43, Henk Slager a écrit :
> > On Thu, Oct 18, 2018 at 6:04 AM Jean-Denis Girard  
> > wrote:
> > This looks like to have a similar root cause as this:
> > https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg81536.html
>
> Yes, seems similar.
>
> > For 50 clones, you could prepare 1 master (sparse) image and change
> > the UUID with btrfstune for every SD-card you have after written the
> > image (optionally sparse write with e.g. dd_rescue).
>
> Thanks for suggestion, I may try this another time, but I can live with
> my workaround.
>
> > But it looks like some change is needed in the kernel code, although I
> > have no clue at the moment where exactly.
>
> Right, this is why I send this message, try to motivate a dev to look at
> this issue ;)
>
>
> Thanks,
> --
> Jean-Denis Girard
>
> SysNux   Systèmes   Linux   en   Polynésie  française
> https://www.sysnux.pf/   Tél: +689 40.50.10.40 / GSM: +689 87.797.527
>
>

Is it possible to construct snapshots manually?

[Cerem Cem ASLAN]

By saying "manually", I mean copying files into a subvolume on a
different mountpoint manually, then mark the target as if it is
created by "btrfs send | btrfs receive".

Rationale:

When we delete all common snapshots from source, we have to send whole
snapshot for next time. This ability will prevent sending everything
from scratch unless it's necessary.
(https://unix.stackexchange.com/q/462451/65781)

Possible workaround:

I've got an idea at the moment: What I want can be achieved by
dropping usage of "btrfs send | btrfs receive" completely and using
only rsync for transfers. After transfer, a snapshot can be created
independently on the remote site. Only problem will be handling the
renamed/relocated files, but `rsync --fuzzy` option might help here:
https://serverfault.com/a/489293/261445

Anyway, it would be good to have a built in support for this.

Re: reproducible builds with btrfs seed feature

[Cerem Cem ASLAN]

Thanks for the explanation, I got it now. I still think this is
related with my needs, so I'll keep an eye on this.

What is the possible use case? I can think of only one scenario: You
have a rootfs that contains a distro installer and you want to
generate distro.img files which uses Btrfs under the hood in different
locations and still have the same hash, so you can publish your
verified image hash by a single source (https://your-distro.org).
You'll sync next release files with the remote servers by using diffs
(btrfs send/receive) and they will generate distro.img independently,
still having the same hash that you'll later verify by
https://your-distro.org.
Chris Murphy , 14 Eki 2018 Paz, 21:10
tarihinde şunu yazdı:
>
> On Sun, Oct 14, 2018 at 6:20 AM, Cerem Cem ASLAN  
> wrote:
> > I'm not sure I could fully understand the desired achievement but it
> > sounds like (or this would be an example of selective perception) it's
> > somehow related with "creating reproducible snapshots"
> > (https://unix.stackexchange.com/q/462451/65781), no?
>
> No the idea is to be able to consistently reproduce a distro installer
> image (like an ISO file) with the same hash. Inside the ISO image, is
> typically a root.img or squash.img which itself contains a file system
> like ext4 or squashfs, to act as the system root. And that root.img is
> the main thing I'm talking about here. There is work to make squashfs
> deterministic, as well as ext4. And I'm wondering if there are sane
> ways to constrain Btrfs features to make it likewise deterministic.
>
> For example:
>
> fallocate -l 5G btrfsroot.img
> losetup /dev/loop0 btrfsroot.img
> mkfs.btrfs -m single -d single -rseed --rootdir /tmp/ -T
> "20181010T1200" --uuidv $X --uuidc $Y --uuidd $Z ...
> shasum btrfsroot.img
>
> And then do it again, and the shasum's should be the same. I realize
> today it's not that way. And that inode assignment, extent allocation
> (number, size, locality) are all factors in making Btrfs quickly
> non-determinstic, and also why I'm assuming this needs to be done in
> user space. That would be the point of the -rseed flag: set the seed
> flag, possibly set a compat_ro flag, fix generation/transid to 1,
> require the use of -T (similar to make_ext4) to set all timestamps to
> this value, and configurable uuid's for everything that uses uuids,
> and whatever other constraints are necessary.
>
>
> --
> Chris Murphy

Re: reproducible builds with btrfs seed feature

[Cerem Cem ASLAN]

I'm not sure I could fully understand the desired achievement but it
sounds like (or this would be an example of selective perception) it's
somehow related with "creating reproducible snapshots"
(https://unix.stackexchange.com/q/462451/65781), no?
Chris Murphy , 14 Eki 2018 Paz, 02:05
tarihinde şunu yazdı:
>
> On Sat, Oct 13, 2018 at 4:28 PM, Chris Murphy  wrote:
> > Is it practical and desirable to make Btrfs based OS installation
> > images reproducible? Or is Btrfs simply too complex and
> > non-deterministic? [1]
> >
> > The main three problems with Btrfs right now for reproducibility are:
> > a. many objects have uuids other than the volume uuid; and mkfs only
> > lets us set the volume uuid
> > b. atime, ctime, mtime, otime; and no way to make them all the same
> > c. non-deterministic allocation of file extents, compression, inode
> > assignment, logical and physical address allocation
>
> d. generation, just pick a consistent default because the entire image
> is made with mkfs and then never rw mounted so it's not a problem
>
> > - Possibly disallow subvolumes and snapshots
>
> There's no actual mechanism to do either of these with mkfs, so it's
> not a problem. And if a sprout is created, it's fine for newly created
> subvolumes to follow the usual behavior of having unique UUID and
> incrementing generation. Thing is, the sprout will inherit the seeds
> preset chunk uuid, which while it shouldn't cause a problem is a kind
> of violation of uuid uniqueness; but ultimately I'm not sure how big
> of a problem it is for such uuids to spread.
>
>
>
> --
> Chris Murphy

Re: DRDY errors are not consistent with scrub results

[Cerem Cem ASLAN]

Chris Murphy , 29 Ağu 2018 Çar, 02:58
tarihinde şunu yazdı:
>
> On Tue, Aug 28, 2018 at 5:04 PM, Cerem Cem ASLAN  
> wrote:
> > What I want to achive is that I want to add the problematic disk as
> > raid1 and see how/when it fails and how BTRFS recovers these fails.
> > While the party goes on, the main system shouldn't be interrupted
> > since this is a production system. For example, I would never expect
> > to be ended up with such a readonly state while trying to add a disk
> > with "unknown health" to the system. Was it somewhat expected?
>
> I don't know. I also can't tell you how LVM or mdraid behave in the
> same situation either though. For sure I've come across bug reports
> where underlying devices go read only and the file system falls over
> totally and developers shrug and say they can't do anything.
>
> This situation is a little different and difficult. You're starting
> out with a one drive setup so the profile is single/DUP or
> single/single, and that doesn't change when adding. So the 2nd drive
> is actually *mandatory* for a brief period of time before you've made
> it raid1 or higher. It's a developer question what is the design, and
> if this is a bug: maybe the device being added should be written to
> with placeholder supers or even just zeros in all the places for 'dev
> add' metadata, and only if that succeeds, to then write real updated
> supers to all devices. It's possible the 'dev add' presently writes
> updated supers to all devices at the same time, and has a brief period
> where the state is fragile and if it fails, it goes read only to
> prevent damaging the file system.

Thinking again, this is totally acceptable. If the requirement was a
good health disk, then I think I must check the disk health by myself.
I may believe that the disk is in a good state, or make a quick test
or make some very detailed tests to be sure.

Likewise, ending up with readonly state is not the end of the world,
even over SSH, because system still functions and all I need to do is
a reboot in the worst case. That's also acceptable *while adding a new
disk*.

>
> Anyway, without a call trace, no idea why it ended up read only. So I
> have to speculate.
>

I may try adding the disk again any time and provide any requested
logs, it is still attached to the server. I'm only not sure if this is
a useful experiment from the point of view of the rest of the people.

>
> >
> > Although we know that disk is about to fail, it still survives.
>
> That's very tenuous rationalization, a drive that rejects even a
> single write is considered failed by the md driver. Btrfs is still
> very tolerant of this, so if it had successfully added and you were
> running in production, you should expect to see thousands of write
> errors dumped to the kernel log

That's exactly what I expected :)

because Btrfs never ejects a bad drive
> still. It keeps trying. And keeps reporting the failures. And all
> those errors being logged can end up causing more write demand if the
> logs are on the same volume as the failing device, even more errors to
> record, and you get an escalating situation with heavy log writing.
>

Good to point this. Maybe I should arrange an on-ram virtual machine
that writes back to local disk if no hardware errors found and start
sending logs in a different server *if* such a hardware failure
occurs.

>
> > Shouldn't we expect in such a scenario that when system tries to read
> > or write some data from/to that BROKEN_DISK and when it recognizes it
> > failed, it will try to recover the part of the data from GOOD_DISK and
> > try to store that recovered data in some other part of the
> > BROKEN_DISK?
>
> Nope. Btrfs can only write supers to fixed locations on the drive,
> same as any other file system. Btrfs metadata could possibly go
> elsewhere because it doesn't have fixed locations, but Btrfs doesn't
> do bad sector tracking. So once it decides metadata goes in location
> X, if X reports a write error it will not try to write elsewhere and
> insofar as I'm aware ext4 and XFS and LVM and md don't either; md does
> have an optional bad block map it will use for tracking bad sectors
> and remap to known good sectors. Normally the drive firmware should do
> this and when that fails the drive is considered toast for production
> purpose

That's also plausible. Thinking again (again? :) if BTRFS would behave
as I expected, that retries might never end if the disk is in a very
bad situation and that would add very intensive IO load on a
production system.

I think in such a situation I should remove the raid device, try to
reformat it and attach it again.

>
> >Or did I misunderstood the whole thing?
>
> Well in a way this is sorta user sabotage. It's a 

Re: DRDY errors are not consistent with scrub results

[Cerem Cem ASLAN]

What I want to achive is that I want to add the problematic disk as
raid1 and see how/when it fails and how BTRFS recovers these fails.
While the party goes on, the main system shouldn't be interrupted
since this is a production system. For example, I would never expect
to be ended up with such a readonly state while trying to add a disk
with "unknown health" to the system. Was it somewhat expected?

Although we know that disk is about to fail, it still survives.
Shouldn't we expect in such a scenario that when system tries to read
or write some data from/to that BROKEN_DISK and when it recognizes it
failed, it will try to recover the part of the data from GOOD_DISK and
try to store that recovered data in some other part of the
BROKEN_DISK? Or did I misunderstood the whole thing?
Chris Murphy , 29 Ağu 2018 Çar, 00:07
tarihinde şunu yazdı:
>
> On Tue, Aug 28, 2018 at 12:50 PM, Cerem Cem ASLAN  
> wrote:
> > I've successfully moved everything to another disk. (The only hard
> > part was configuring the kernel parameters, as my root partition was
> > on LVM which is on LUKS partition. Here are the notes, if anyone
> > needs: 
> > https://github.com/ceremcem/smith-sync/blob/master/create-bootable-backup.md)
> >
> > Now I'm seekin for trouble :) I tried to convert my new system (booted
> > with new disk) into raid1 coupled with the problematic old disk. To do
> > so, I issued:
> >
> > sudo btrfs device add /dev/mapper/master-root /mnt/peynir/
> > /dev/mapper/master-root appears to contain an existing filesystem (btrfs).
> > ERROR: use the -f option to force overwrite of /dev/mapper/master-root
> > aea@aea3:/mnt$ sudo btrfs device add /dev/mapper/master-root /mnt/peynir/ -f
> > ERROR: error adding device '/dev/mapper/master-root': Input/output error
> > aea@aea3:/mnt$ sudo btrfs device add /dev/mapper/master-root /mnt/peynir/
> > sudo: unable to open /var/lib/sudo/ts/aea: Read-only file system
> >
> > Now I ended up with a readonly file system. Isn't it possible to add a
> > device to a running system?
>
> Yes.
>
> The problem is the 2nd error message:
>
> ERROR: error adding device '/dev/mapper/master-root': Input/output error
>
> So you need to look in dmesg to see what Btrfs kernel messages
> occurred at that time. I'm gonna guess it's a failed write. You have a
> few of those in the smartctl log output. Any time a write failure
> happens, the operation is always fatal regardless of the file system.
>
>
>
> --
> Chris Murphy

Re: DRDY errors are not consistent with scrub results

[Cerem Cem ASLAN]

I've successfully moved everything to another disk. (The only hard
part was configuring the kernel parameters, as my root partition was
on LVM which is on LUKS partition. Here are the notes, if anyone
needs: 
https://github.com/ceremcem/smith-sync/blob/master/create-bootable-backup.md)

Now I'm seekin for trouble :) I tried to convert my new system (booted
with new disk) into raid1 coupled with the problematic old disk. To do
so, I issued:

sudo btrfs device add /dev/mapper/master-root /mnt/peynir/
/dev/mapper/master-root appears to contain an existing filesystem (btrfs).
ERROR: use the -f option to force overwrite of /dev/mapper/master-root
aea@aea3:/mnt$ sudo btrfs device add /dev/mapper/master-root /mnt/peynir/ -f
ERROR: error adding device '/dev/mapper/master-root': Input/output error
aea@aea3:/mnt$ sudo btrfs device add /dev/mapper/master-root /mnt/peynir/
sudo: unable to open /var/lib/sudo/ts/aea: Read-only file system

Now I ended up with a readonly file system. Isn't it possible to add a
device to a running system?

Chris Murphy , 28 Ağu 2018 Sal, 04:08
tarihinde şunu yazdı:
>
> On Mon, Aug 27, 2018 at 6:49 PM, Cerem Cem ASLAN  
> wrote:
> > Thanks for your guidance, I'll get the device replaced first thing in
> > the morning.
> >
> > Here is balance results which I think resulted not too bad:
> >
> > sudo btrfs balance start /mnt/peynir/
> > WARNING:
> >
> > Full balance without filters requested. This operation is very
> > intense and takes potentially very long. It is recommended to
> > use the balance filters to narrow down the balanced data.
> > Use 'btrfs balance start --full-balance' option to skip this
> > warning. The operation will start in 10 seconds.
> > Use Ctrl-C to stop it.
> > 10 9 8 7 6 5 4 3 2 1
> > Starting balance without any filters.
> > Done, had to relocate 18 out of 18 chunks
> >
> > I suppose this means I've not lost any data, but I'm very prone to due
> > to previous `smartctl ...` results.
>
>
> OK so nothing fatal anyway. We'd have to see any kernel messages that
> appeared during the balance to see if there were read or write errors,
> but presumably any failure means the balance fails so... might get you
> by for a while actually.
>
>
>
>
>
>
>
> --
> Chris Murphy

Re: DRDY errors are not consistent with scrub results

[Cerem Cem ASLAN]

Thanks for your guidance, I'll get the device replaced first thing in
the morning.

Here is balance results which I think resulted not too bad:

sudo btrfs balance start /mnt/peynir/
WARNING:

Full balance without filters requested. This operation is very
intense and takes potentially very long. It is recommended to
use the balance filters to narrow down the balanced data.
Use 'btrfs balance start --full-balance' option to skip this
warning. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
Done, had to relocate 18 out of 18 chunks

I suppose this means I've not lost any data, but I'm very prone to due
to previous `smartctl ...` results.

Chris Murphy , 28 Ağu 2018 Sal, 03:39
tarihinde şunu yazdı:
>
> On Mon, Aug 27, 2018 at 6:38 PM, Chris Murphy  wrote:
>
> >> Metadata,single: Size:8.00MiB, Used:0.00B
> >>/dev/mapper/master-root 8.00MiB
> >>
> >> Metadata,DUP: Size:2.00GiB, Used:562.08MiB
> >>/dev/mapper/master-root 4.00GiB
> >>
> >> System,single: Size:4.00MiB, Used:0.00B
> >>/dev/mapper/master-root 4.00MiB
> >>
> >> System,DUP: Size:32.00MiB, Used:16.00KiB
> >>/dev/mapper/master-root64.00MiB
> >>
> >> Unallocated:
> >>/dev/mapper/master-root   915.24GiB
> >
> >
> > OK this looks like it maybe was created a while ago, it has these
> > empty single chunk items that was common a while back. There is a low
> > risk to clean it up, but I still advise backup first:
> >
> > 'btrfs balance start -mconvert=dup '
>
> You can skip this advise now, it really doesn't matter. But future
> Btrfs shouldn't have both single and DUP chunks like this one is
> showing, if you're using relatively recent btrfs-progs to create the
> file system.
>
>
> --
> Chris Murphy

DRDY errors are not consistent with scrub results

[Cerem Cem ASLAN]

Hi,

I'm getting DRDY ERR messages which causes system crash on the server:

# tail -n 40 /var/log/kern.log.1
Aug 24 21:04:55 aea3 kernel: [  939.228059] lxc-bridge: port
5(vethI7JDHN) entered disabled state
Aug 24 21:04:55 aea3 kernel: [  939.300602] eth0: renamed from vethQ5Y2OF
Aug 24 21:04:55 aea3 kernel: [  939.328245] IPv6: ADDRCONF(NETDEV_UP):
eth0: link is not ready
Aug 24 21:04:55 aea3 kernel: [  939.328453] IPv6:
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Aug 24 21:04:55 aea3 kernel: [  939.328474] IPv6:
ADDRCONF(NETDEV_CHANGE): vethI7JDHN: link becomes ready
Aug 24 21:04:55 aea3 kernel: [  939.328491] lxc-bridge: port
5(vethI7JDHN) entered blocking state
Aug 24 21:04:55 aea3 kernel: [  939.328493] lxc-bridge: port
5(vethI7JDHN) entered forwarding state
Aug 24 21:04:59 aea3 kernel: [  943.085647] cgroup: cgroup2: unknown
option "nsdelegate"
Aug 24 21:16:15 aea3 kernel: [ 1619.400016] perf: interrupt took too
long (2506 > 2500), lowering kernel.perf_event_max_sample_rate to
79750
Aug 24 21:17:11 aea3 kernel: [ 1675.515815] perf: interrupt took too
long (3137 > 3132), lowering kernel.perf_event_max_sample_rate to
63750
Aug 24 21:17:13 aea3 kernel: [ 1677.080837] cgroup: cgroup2: unknown
option "nsdelegate"
Aug 25 22:38:31 aea3 kernel: [92955.512098] usb 4-2: USB disconnect,
device number 2
Aug 26 02:14:21 aea3 kernel: [105906.035038] lxc-bridge: port
4(vethCTKU4K) entered disabled state
Aug 26 02:15:30 aea3 kernel: [105974.107521] lxc-bridge: port
4(vethO59BPD) entered disabled state
Aug 26 02:15:30 aea3 kernel: [105974.109991] device vethO59BPD left
promiscuous mode
Aug 26 02:15:30 aea3 kernel: [105974.109995] lxc-bridge: port
4(vethO59BPD) entered disabled state
Aug 26 02:15:30 aea3 kernel: [105974.710490] lxc-bridge: port
4(vethBAYODL) entered blocking state
Aug 26 02:15:30 aea3 kernel: [105974.710493] lxc-bridge: port
4(vethBAYODL) entered disabled state
Aug 26 02:15:30 aea3 kernel: [105974.710545] device vethBAYODL entered
promiscuous mode
Aug 26 02:15:30 aea3 kernel: [105974.710598] IPv6:
ADDRCONF(NETDEV_UP): vethBAYODL: link is not ready
Aug 26 02:15:30 aea3 kernel: [105974.710600] lxc-bridge: port
4(vethBAYODL) entered blocking state
Aug 26 02:15:30 aea3 kernel: [105974.710601] lxc-bridge: port
4(vethBAYODL) entered forwarding state
Aug 26 02:16:35 aea3 kernel: [106039.674089] BTRFS: device fsid
5b844c7a-0cbd-40a7-a8e3-6bc636aba033 devid 1 transid 984 /dev/dm-3
Aug 26 02:17:21 aea3 kernel: [106085.352453] ata4.00: failed command: READ DMA
Aug 26 02:17:21 aea3 kernel: [106085.352901] ata4.00: status: { DRDY ERR }
Aug 26 02:18:56 aea3 kernel: [106180.648062] ata4.00: exception Emask
0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 26 02:18:56 aea3 kernel: [106180.648333] ata4.00: BMDMA stat 0x25
Aug 26 02:18:56 aea3 kernel: [106180.648515] ata4.00: failed command: READ DMA
Aug 26 02:18:56 aea3 kernel: [106180.648706] ata4.00: cmd
c8/00:08:80:9c:bb/00:00:00:00:00/e3 tag 0 dma 4096 in
Aug 26 02:18:56 aea3 kernel: [106180.648706]  res
51/40:00:80:9c:bb/00:00:00:00:00/03 Emask 0x9 (media error)
Aug 26 02:18:56 aea3 kernel: [106180.649380] ata4.00: status: { DRDY ERR }
Aug 26 02:18:56 aea3 kernel: [106180.649743] ata4.00: error: { UNC }
Aug 26 02:18:56 aea3 kernel: [106180.779311] ata4.00: configured for UDMA/133
Aug 26 02:18:56 aea3 kernel: [106180.779331] sd 3:0:0:0: [sda] tag#0
FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 26 02:18:56 aea3 kernel: [106180.779335] sd 3:0:0:0: [sda] tag#0
Sense Key : Medium Error [current]
Aug 26 02:18:56 aea3 kernel: [106180.779339] sd 3:0:0:0: [sda] tag#0
Add. Sense: Unrecovered read error - auto reallocate failed
Aug 26 02:18:56 aea3 kernel: [106180.779343] sd 3:0:0:0: [sda] tag#0
CDB: Read(10) 28 00 03 bb 9c 80 00 00 08 00
Aug 26 02:18:56 aea3 kernel: [106180.779346] blk_update_request: I/O
error, dev sda, sector 62626944
Aug 26 02:18:56 aea3 kernel: [106180.779703] BTRFS error (device
dm-2): bdev /dev/mapper/master-root errs: wr 0, rd 40, flush 0,
corrupt 0, gen 0
Aug 26 02:18:56 aea3 kernel: [106180.779936] ata4: EH complete


I always saw these DRDY errors whenever I experience physical hard
drive errors, so I expect `btrfs scrub` show some kind of similar
errors but it doesn't:

btrfs scrub status /mnt/peynir/
scrub status for 8827cb0e-52d7-4f99-90fd-a975cafbfa46
scrub started at Tue Aug 28 00:43:55 2018 and finished after 00:02:07
total bytes scrubbed: 12.45GiB with 0 errors

I took new snapshots for both root and the LXC containers and nothing
gone wrong. To be confident, I reformat the swap partition (which I
saw some messages about swap partition in the crash screen).

I'm not sure how to proceed at the moment. Taking succesfull backups
made me think that everything might be okay but I'm not sure if I
should continue trusting the drive or not. What additional checks
should I perform?

How to ensure that a snapshot is not corrupted?

[Cerem Cem ASLAN]

Original question is here: https://superuser.com/questions/1347843

How can we sure that a readonly snapshot is not corrupted due to a disk failure?

Is the only way calculating the checksums one on another and store it
for further examination, or does BTRFS handle that on its own?

Re: Unmountable root partition

[Cerem Cem ASLAN]

FYI: According to ddrescue, there were read errors in +45% of the
partition. There were no chances to save anything from the disk.

2018-08-01 15:07 GMT+03:00 Cerem Cem ASLAN :

> Yes, command output is as is, because I just copied and pasted into the
> mail. When I omit the `-t btrfs` part, result is the same.
>
> I'm now trying to rescue what I can, so getting a image dump with
> `ddrescue`. It's read about 25% without any errors but it will be expected
> to finish in 6 hours. Then I'll try btrfscue to see what happens.
>
> I know it's totally my fault because I must be ready for a total
> disk/pc/building burn out. Lessons learned.
>
> 2018-08-01 8:32 GMT+03:00 Chris Murphy :
>
>> On Tue, Jul 31, 2018 at 12:03 PM, Cerem Cem ASLAN 
>> wrote:
>>
>> > 3. mount -t btrfs /dev/mapper/foo--vg-root /mnt/foo
>> > Gives the following error:
>> >
>> > mount: wrong fs type, bad option, bad superblock on ...
>> >
>> > 4. dmesg | tail
>> > Outputs the following:
>> >
>> >
>> > [17755.840916] sd 3:0:0:0: [sda] tag#0 FAILED Result:
>> > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
>> > [17755.840919] sd 3:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 07 c0 02
>> 00 00
>> > 02 00
>> > [17755.840921] blk_update_request: I/O error, dev sda, sector 507906
>> > [17755.840941] EXT4-fs (dm-4): unable to read superblock
>>
>>
>> Are you sure this is the output for the command? Because you're
>> explicitly asking for type btrfs, which fails, and then the kernel
>> reports EXT4 superblock unreadable. What do you get if you omit -t
>> btrfs and just let it autodetect?
>>
>> But yeah, this is an IO error from the device and there's nothing
>> Btrfs can do about that unless there is DUP or raid1+ metadata
>> available.
>>
>> Is it possible this LV was accidentally reformatted ext4?
>>
>>
>> --
>> Chris Murphy
>>
>
>

Re: Unmountable root partition

[Cerem Cem ASLAN]

Yes, command output is as is, because I just copied and pasted into the
mail. When I omit the `-t btrfs` part, result is the same.

I'm now trying to rescue what I can, so getting a image dump with
`ddrescue`. It's read about 25% without any errors but it will be expected
to finish in 6 hours. Then I'll try btrfscue to see what happens.

I know it's totally my fault because I must be ready for a total
disk/pc/building burn out. Lessons learned.

2018-08-01 8:32 GMT+03:00 Chris Murphy :

> On Tue, Jul 31, 2018 at 12:03 PM, Cerem Cem ASLAN 
> wrote:
>
> > 3. mount -t btrfs /dev/mapper/foo--vg-root /mnt/foo
> > Gives the following error:
> >
> > mount: wrong fs type, bad option, bad superblock on ...
> >
> > 4. dmesg | tail
> > Outputs the following:
> >
> >
> > [17755.840916] sd 3:0:0:0: [sda] tag#0 FAILED Result:
> > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
> > [17755.840919] sd 3:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 07 c0 02
> 00 00
> > 02 00
> > [17755.840921] blk_update_request: I/O error, dev sda, sector 507906
> > [17755.840941] EXT4-fs (dm-4): unable to read superblock
>
>
> Are you sure this is the output for the command? Because you're
> explicitly asking for type btrfs, which fails, and then the kernel
> reports EXT4 superblock unreadable. What do you get if you omit -t
> btrfs and just let it autodetect?
>
> But yeah, this is an IO error from the device and there's nothing
> Btrfs can do about that unless there is DUP or raid1+ metadata
> available.
>
> Is it possible this LV was accidentally reformatted ext4?
>
>
> --
> Chris Murphy
>

Re: Unmountable root partition

[Cerem Cem ASLAN]

That might be the case. Can't we recover anything with some data loss?

2018-08-01 2:04 GMT+03:00 Hans van Kranenburg <
hans.van.kranenb...@mendix.com>:

> Hi,
>
> On 07/31/2018 08:03 PM, Cerem Cem ASLAN wrote:
> > Hi,
> >
> > I'm having trouble with my server setup, which contains a BTRFS root
> > partition on top of LVM on top of LUKS partition.
> >
> > Yesterday server was shut down unexpectedly. I booted the system with a
> > pendrive which contains Debian 4.9.18 and tried to mount the BTRFS root
> > partition manually.
> >
> > 1. cryptsetup luksOpen /dev/sda5
> >
> > Seems to decrypt the partition (there are no errors)
> >
> > 2. /dev/mapper/foo--vg-root and /dev/mapper/foo--vg-swap_1 is created
> > automatically, so I suppose LVM works correctly.
> >
> > 3. mount -t btrfs /dev/mapper/foo--vg-root /mnt/foo
> > Gives the following error:
> >
> > mount: wrong fs type, bad option, bad superblock on ...
> >
> > 4. dmesg | tail
> > Outputs the following:
> >
> >
> > [17755.840916] sd 3:0:0:0: [sda] tag#0 FAILED Result:
> > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
> > [17755.840919] sd 3:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 07 c0
> > 02 00 00 02 00
> > [17755.840921] blk_update_request: I/O error, dev sda, sector 507906
> > [17755.840941] EXT4-fs (dm-4): unable to read superblock
> > [18140.052300] sd 3:0:0:0: [sda] tag#0 FAILED Result:
> > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
> > [18140.052305] sd 3:0:0:0: [sda] tag#0 CDB: ATA command pass
> > through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
> > [18142.991851] sd 3:0:0:0: [sda] tag#0 FAILED Result:
> > hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
> > [18142.991856] sd 3:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 07 c0
> > 80 00 00 08 00
> > [18142.991860] blk_update_request: I/O error, dev sda, sector 508032
> > [18142.991869] Buffer I/O error on dev dm-4, logical block 16, async
> > page read
>
> This points at hardware level failure, regardless if the disk would hold
> a btrfs or ext4 or whatever other kind of filesystem. The disk itself
> cannot read data back when we ask for it.
>
> > 4.
> >
> > # btrfs restore -i -D /dev/mapper/foo--vg-root /dev/null
> > No valid Btrfs found on /dev/mapper/foo--vg-root
> > Could not open root, trying backup super
> > No valid Btrfs found on /dev/mapper/foo--vg-root
> > Could not open root, trying backup super
> > No valid Btrfs found on /dev/mapper/foo--vg-root
> > Could not open root, trying backup super
> >
> > We are pretty sure that no unexpected electric cuts has been happened.
> >
> > At this point I don't know what information I should supply.
> >
>
>
> --
> Hans van Kranenburg
>

Re: Expected Received UUID

[Cerem Cem ASLAN]

Not an answer, but exactly same case:
https://unix.stackexchange.com/questions/377914/how-to-test-if-two-btrfs-snapshots-are-identical

2018-07-31 20:42 GMT+03:00 Gaurav Sanghavi :

> Hello everyone,
>
> While researching BTRFS for a project that involves backing up snapshots
> from device A to device B
> before consolidating backups from device B to device C, I noticed that
> received UUID on snapshot on
> device C is not the same as received UUID for the same snapshot on device
> B. Here are my steps:
>
> 1. Device A
> BTRFS version: v3.17
>
> btrfs su sn -r /home/test/snapshot1 /home/test/snaps/snapshot1
> btrfs su show /home/test/snaps/snapshot1
> Name: snapshot1
> uuid: b00e8ba1-5aaa-3641-9c4c-e168eee5c296
> Parent uuid: cb570dec-e9fd-1f40-99d2-2070c8ee2466
> Received UUID:  ---
> Creation time: 2018-07-30 18:32:37
> Flags: readonly
>
> 2. Send to Device B
> btrfs send /home/test/snaps/snapshot1 | ssh  'btrfs receive
> /home/backups/'
>
> After send completes, on Device B
> BTRFS version: v4.7.3
> btrfs su show /home/backups/snapshot1
> Name: snapshot1
> UUID: 7c13d189-7fee-584e-ac90-e68cb0012f5c
> Parent UUID: a2314f7c-4b11-ed40-901f-f1acb5ebf802
> Received UUID: b00e8ba1-5aaa-3641-9c4c-e168eee5c296
> Creation time: 2018-07-30 18:42:37 -0700
> Flags: readonly
>
>
> 3. Send to Device C
> btrfs send /home/backups/snapshot1 | ssh  'btrfs receive
> /home/backups2/'
>
> After send completes, on Device C
> BTRFS version: v4.7.3
> btrfs su show /home/backups2/snapshot1
> Name: snapshot1
> UUID: 8a13aab5-8e44-2541-9082-bc583933b964
> Parent UUID: 54e9b4ff-46dc-534e-b70f-69eb7bb21028
> Received UUID: 7c13d189-7fee-584e-ac90-e68cb0012f5c
> Creation time: 2018-07-30 18:58:32 -0700
> Flags: readonly
>
> 1. I have gone through some of the archived emails and have noticed people
> mentioning that
> if received UUID is set, btrfs send propogates this 'received UUID'. But
> in above case,
> it's different for the same snapshot on all three devices. Is this the
> expected behavior ?
>
> 2. We want to be able to start backing up from Device A to C, in case B
> goes down or needs
> to be replaced. If received UUID is expected to differ for the snapshot on
> device B and C, incremental
> backups will not work from A to C without setting received UUID. I have
> seen python-btrfs
> mentioned in a couple of emails; but have anyone of you used it in a
> production environment ?
>
> This is my first post to this email. Please let me know if I am missing
> any details.
>
> Thanks,
> Gaurav
>
>

Unmountable root partition

[Cerem Cem ASLAN]

Hi,

I'm having trouble with my server setup, which contains a BTRFS root
partition on top of LVM on top of LUKS partition.

Yesterday server was shut down unexpectedly. I booted the system with a
pendrive which contains Debian 4.9.18 and tried to mount the BTRFS root
partition manually.

1. cryptsetup luksOpen /dev/sda5

Seems to decrypt the partition (there are no errors)

2. /dev/mapper/foo--vg-root and /dev/mapper/foo--vg-swap_1 is created
automatically, so I suppose LVM works correctly.

3. mount -t btrfs /dev/mapper/foo--vg-root /mnt/foo
Gives the following error:

mount: wrong fs type, bad option, bad superblock on ...

4. dmesg | tail
Outputs the following:


[17755.840916] sd 3:0:0:0: [sda] tag#0 FAILED Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[17755.840919] sd 3:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 07 c0 02 00
00 02 00
[17755.840921] blk_update_request: I/O error, dev sda, sector 507906
[17755.840941] EXT4-fs (dm-4): unable to read superblock
[18140.052300] sd 3:0:0:0: [sda] tag#0 FAILED Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[18140.052305] sd 3:0:0:0: [sda] tag#0 CDB: ATA command pass through(16) 85
06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[18142.991851] sd 3:0:0:0: [sda] tag#0 FAILED Result:
hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[18142.991856] sd 3:0:0:0: [sda] tag#0 CDB: Read(10) 28 00 00 07 c0 80 00
00 08 00
[18142.991860] blk_update_request: I/O error, dev sda, sector 508032
[18142.991869] Buffer I/O error on dev dm-4, logical block 16, async page
read


4.

# btrfs restore -i -D /dev/mapper/foo--vg-root /dev/null
No valid Btrfs found on /dev/mapper/foo--vg-root
Could not open root, trying backup super
No valid Btrfs found on /dev/mapper/foo--vg-root
Could not open root, trying backup super
No valid Btrfs found on /dev/mapper/foo--vg-root
Could not open root, trying backup super

We are pretty sure that no unexpected electric cuts has been happened.

At this point I don't know what information I should supply.

Why isnt NOCOW attributes propogated on snapshot transfers?

[Cerem Cem ASLAN]

`btrfs send | btrfs receive` removes NOCOW attributes. Is it a bug or
a feature? If it's a feature, how can we keep these attributes if we
need to?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Building a BTRFS test machine

[Cerem Cem ASLAN]

Would that be useful to build a BTRFS test machine, which will perform
both software tests (btrfs send | btrfs receive, read/write random
data etc.) and hardware tests, such as abrupt power off test, abruptly
removing a raid-X disk physically, etc.

If it would be useful, what tests should it cover?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Btrfs incremental send | receive fails with Error: File not found

[Cerem Cem ASLAN]

Then following problem is directly related with that:
https://unix.stackexchange.com/questions/377914/how-to-test-if-two-btrfs-snapshots-are-identical

Is that a bug or a feature?


2017-08-01 23:33 GMT+03:00 A L <crimsoncott...@gmail.com>:
>
> On 8/1/2017 10:24 PM, Cerem Cem ASLAN wrote:
>>
>> What is that mean? Can't we replicate the same snapshot with `btrfs send |
>> btrfs receive` multiple times, because it will have a "Received UUID" at the
>> first `btrfs receive
>
>
> You will need to make a new read-write snapshot of the received volume to
> fix it. Any snapshots created from the received subvolume can't be used for
> send-receive again, afaik.
>
> # btrfs subvolume snapshot subvolume.received subvolume
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Btrfs incremental send | receive fails with Error: File not found

[Cerem Cem ASLAN]

What is that mean? Can't we replicate the same snapshot with `btrfs
send | btrfs receive` multiple times, because it will have a "Received
UUID" at the first `btrfs receive`?

2017-08-01 15:54 GMT+03:00 A L :
> OK. The problem was that the original subvolume had a "Received UUID". This
> caused all subsequent snapshots to have the same Received UUID which messes
> up Btrfs send | receive. Of course this means I must have used btrfs send |
> receive to create that subvolume and then turned it r/w at some point,
> though I cannot remember ever doing this.
>
> Perhaps a clear notice "WARNING: make sure that the source subvolume does
> not have a Received UUID" on the Wiki would be helpful? Both on
> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup and on
> https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-property
>
> Regards,
> A
>
>
> On 7/28/2017 9:32 PM, Hermann Schwärzler wrote:
>>
>> Hi
>>
>> for me it looks like those snapshots are not read-only. But as far as I
>> know for using send they have to be.
>>
>> At least
>>
>> https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Initial_Bootstrapping
>> states "We will need to create a read-only snapshot ,,,"
>>
>> I am using send/receive (with read-only snapshots) on a regular basis and
>> never had a problem like yours.
>>
>> What are the commands you use to create your snapshots?
>>
>> Greetings
>> Hermann
>>
>> On 07/28/2017 07:26 PM, A L wrote:
>>>
>>> I often hit the following error when doing incremental btrfs
>>> send-receive:
>>> Btrfs incremental send | receive fails with Error: File not found
>>>
>>> Sometimes I can do two-three incremental snapshots, but then the same
>>> error (different file) happens again. It seems that the files were
>>> changed or replaced between snapshots, which is causing the problems for
>>> send-receive. I have tried to delete all snapshots and started over but
>>> the problem comes back, so I think it must be a bug.
>>>
>>> The source volume is:   /mnt/storagePool (with RAID1 profile)
>>> with subvolume:   volume/userData
>>> Backup disk is:   /media/usb-backup (external USB disk)
>>
>> [...]
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Destination's Received UUID and source's UUID are not the same

[Cerem Cem ASLAN]

A few days ago I tried to copy my backups to another machine by using
my external hard drive. `btrfs send my-backup | btrfs receive
/path/to/backup/dir` didn't go right since destination snapshot's
"Received UUID" does not match with source's "UUID". Here are relevant
stackoverflow question:

https://unix.stackexchange.com/questions/377914/how-to-test-if-two-btrfs-snapshots-are-identical

Is it a bug?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Leveldb in google-chrome incompatible with btrfs?

[Cerem Cem ASLAN]

I was also struggling with this issue for quite some time. Today my 2
months old disk is crashed (which really surprises me). It doesn't
even being shown by `fdisk -l`.

After buying this this disk, I installed a Debian on a BTRFS partition
on an LVM partition on a LUKS partition. From that day till today,
while doing my work, there was always a video running on Youtube in
the background. Sometimes my editor (atom, it also shares a lot with
chromium) was slowing down which was making me end up restarting the
Chromium (and it was restarting barely with `killall -9 chromium`.
Another mentionable event is, 3 days ago my laptop suddenly restarted.

I suspect Chromium usage on BTRFS and disk the crash are somehow related.

2017-07-07 8:46 GMT+03:00 Omar Sandoval :
> On Thu, Jul 06, 2017 at 04:59:39PM -0700, Marc MERLIN wrote:
>> On Thu, Jul 06, 2017 at 04:44:51PM -0700, Omar Sandoval wrote:
>> > In the bug report, you commented that CURRENT contained MANIFEST-010814,
>> > is that indeed the case or was it actually something newer? If it was
>> > the newer one, then it's still tricky how we'd end up that way but not
>> > as outlandish.
>>
>> You are correct, my bad.
>> At this point, I'm going to have to assume that something bad happened with
>> rsync when I rsync'ed an old profile over the one that caused chrome to fail
>> to restart.
>> Especially because CURRENT was dated Oct 4th, which does not make sense.
>
> Okay, this makes sense.
>
>> Now that I know what to look for, I'll have a much closer look next time
>> this happens, with the understanding that it would be a while if I've
>> successfully fixed the reason why my laptop was crashing too often.
>
> Sounds good.
>
>> But you said you've also seen issues with google-chrome profile and btrfs.
>> What did you experience?
>
> I never debugged it, but I had to blow away the profile a couple of times. 
> Then
> there's this weird one which looks like a Btrfs bug:
>
> ┌[osandov@vader ~/.config]
> └$ ls -al google-chrome-busted/**
> ls: cannot access 'google-chrome-busted/Local State': No such file or 
> directory
> google-chrome-busted/Default:
> ls: cannot access 'google-chrome-busted/Default/Preferences': No such file or 
> directory
> ls: cannot access 'google-chrome-busted/Default/.com.google.Chrome.VfAUNx': 
> No such file or directory
> total 0
> drwx-- 1 osandov users 12 Feb  7 16:50 .
> drwx-- 1 osandov users 14 Feb  7 16:50 ..
> -? ? ?   ?  ?? .com.google.Chrome.VfAUNx
> -? ? ?   ?  ?? Preferences
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: No such file or directory error

[Cerem Cem ASLAN]

Sorry for above post, the error is just because of passing the parent
and current snapshots from different physical disks.



2017-06-16 9:29 GMT+03:00 Cerem Cem ASLAN <cerem...@ceremcem.net>:
> Hello everybody,
>
> I'm facing with a "No such file or directory" issue which only occurs
> if I use a `btrfs send -p myparent mysnap` line in a script and its
> second call. Detailed description and error log is here:
> https://github.com/ceremcem/smith-sync/issues/5
>
> I use `btrfs send` command via a function:
>
> btrfs_send_diff () {
> local parent=$1
> local current=$2
> if [[ -z $current ]]; then
> echo_yellow "No parent specified, sending whole snapshot"
> echo_red "parent: $parent, current: $current"
> return 255
> current=$1
> btrfs send $current
> else
> echo_green "Sending difference between $parent and $current"
> btrfs send -p $parent --no-data $current > /dev/null
> if [[ $? != 0 ]]; then
> echo_red "WE DETECT AN ERROR in ${FUNCNAME[0]}! parent:
> $parent, curr: $current"
> exit
> else
> btrfs send -p $parent $current
> fi
> fi
> }
>
>
>
> Am I using `btrfs send` in a wrong way or is that a known issue with
> `btrfs send`?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Fwd: No such file or directory error

[Cerem Cem ASLAN]

Hello everybody,

I'm facing with a "No such file or directory" issue which only occurs
if I use a `btrfs send -p myparent mysnap` line in a script and its
second call. Detailed description and error log is here:
https://github.com/ceremcem/smith-sync/issues/5

I use `btrfs send` command via a function:

btrfs_send_diff () {
local parent=$1
local current=$2
if [[ -z $current ]]; then
echo_yellow "No parent specified, sending whole snapshot"
echo_red "parent: $parent, current: $current"
return 255
current=$1
btrfs send $current
else
echo_green "Sending difference between $parent and $current"
btrfs send -p $parent --no-data $current > /dev/null
if [[ $? != 0 ]]; then
echo_red "WE DETECT AN ERROR in ${FUNCNAME[0]}! parent:
$parent, curr: $current"
exit
else
btrfs send -p $parent $current
fi
fi
}



Am I using `btrfs send` in a wrong way or is that a known issue with
`btrfs send`?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html