Re: SU+J Lost files after a power failure

2013-10-14 Thread Charles Swiger 
On Oct 14, 2013, at 12:41 PM, RW  wrote:
> On Mon, 14 Oct 2013 11:48:18 -0700 Charles Swiger wrote:
>> Yes.  Without journalling, you'd normally perform the full
>> timeconsuming fsck in the foreground.
> 
> Journalling removes the need for the background fsck which only recovers
> lost space. 

That and inode link changes (ie, adding or removing files from a directory).

>> With journalling, it should be able to do a journal replay to restore
>> the filesystem to an OK state,
> 
> My understanding is that the journal does nothing to restore the
> filesystem other than keep track of orphaned memory. In all other
> respect it's the job of soft-updates to keep the filesystem in an OK
> state.

Yes, SU is supposed to reorder filesystem operations to provide some level
of "ACID" transaction semantics-- and the journal helps that by avoiding
the need for bgfsck.

> When it doesn't you need a foreground check.
> 
>> but sometimes that doesn't restore consistency, in which case it
>> usually fires off a background fsck rather than the foreground fsck.
> 
> I think if the journal fails, you would really need to run at least a
> foreground preen, maybe a full fsck. 

Yes.

Regards,
-- 
-Chuck

___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Charles Swiger 
Hi--

On Oct 14, 2013, at 11:51 AM, Daniel Feenberg  wrote:
> This discussion skirts the critical issue - are files that are not open for 
> writing endangered? No description of the uses of journaling can be 
> considered informative if it doesn't address that explicitly. As a naive user 
> I have always assumed that once closed, a file was invulnerable to improper 
> shutdowns, but this discussion shakes that belief.

Well, it's good to be a little paranoid if the data matters.  :-)

First, unless you call fsync() before close() and your OS and/or drive hardware 
isn't being deceptive when fsync() returns about whether the bits have made it 
to permanent storage, then you might be surprised at just how long the 
unwritten buffers containing the last updates to the file data take to get 
properly flushed to disk.

> I expect the answer may be different for SSD and spinning disks.

Second, this is an excellent point: however, it also applies to anything where 
the actual hardware block size does not match the device blocksize that the 
filesystem thinks it has-- so new "4K sector" rotational disks also have some 
risk.

The basic issue with SSDs is that you (or the drive firmware, more precisely) 
need to read in an entire hardware sector, update the portion with changes in 
cache memory, do a bulk-erase of that block, and then scribble that back out.  
Good drive firmware actually writes out to a different block than the original 
for wear-leveling purposes and only updates the flash translation layer once 
the new version of that block is written.  That makes the drive mostly immune 
to major data integrity issues even if powered off in the middle of the process.

Less-than-good firmware, aka buggy firmware, can lead from power-failure to 
data loss of files which were not being modified at the time.  And may you 
possess recent working backups if the FTL somehow ever gets confused!

Regards,
-- 
-Chuck

___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread David Demelier 
On 14.10.2013 20:43, Adam Vande More wrote:
> On Mon, Oct 14, 2013 at 1:33 PM, CeDeROM  > wrote:
> 
> Thank you for explaining :-) So it looks that it would be sensible to
> force filesystem check every n-th mount..? 
> 
> 
> Please explain the logic in which this helps anything.
>  
> 
> Or to do a filesystem check
> after crash..?
> 
> 
> Already standard behavior as implicitly seen in this thread.
>  
> 
> Are there any flags like that to mark filesystem
> unclean and to force fsck after n-th mount? 
> 
> 
> No and any fs that requires such a system is broken by design.
>  
> 
> That would assume
> disabling journal and soft updates journaling I guess..?
> 
> What would be the best option for best data integrity in case of
> crash?
> 
> 
> mount -o sync or use ZFS. Both require hardware that correctly report
> success to fsync.

I personnally love ZFS and use it massively on my server, but for a
desktop I think this is a real overkill. Also I don't have so much RAM
to waste for that. I think UFS is enough, however as a modern operating
system I don't expect any data corruption by default using SU+J.

The filesystem domain is not a thing I really know deeply, so thanks for
all you explanation.

PS: the power failure is not the only way that does not shutdown cleanly
the system. There are kernel panics, crash and such of course. Those
which appears sometimes too.

Regards,

___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread David Demelier 
On 14.10.2013 18:47, Adam Vande More wrote:
> There is no *warranty* as explicitly stated in
> http://www.freebsd.org/copyright/freebsd-license.html
> 

Aha, please don't play on words ;-). I think you understood I was
speaking about the filesystem state
not a lawyer issue.
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread David Demelier 
On 14.10.2013 20:08, RW wrote:
> On Mon, 14 Oct 2013 18:34:36 +0200
> David Demelier wrote:
> 
>> On 14.10.2013 14:39, RW wrote:
> 
>>> If you are having problems with data integrity you might try
>>> gjournal or zfs instead.
>>
>> Why? SU+J is enabled by default. Isn't the purpose of a journaled file
>> system to ensure that any bad shutdown will protect data?
> 
> SU+J isn't a journalled filesytem, it's a filesystem with soft-updates
> that journals information about free space so it can be recovered
> without having to go through the whole filesystem.
> 

Okay, but why the fsck didn't run by itself to detect that the journal
didn't replayed correctly (if I understanding well) to correct the issues?

___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread RW 
On Mon, 14 Oct 2013 11:48:18 -0700
Charles Swiger wrote:

> fsck_y_enable="YES"

One of the most annoying things about SU+J is that fsck asks if you
want to use the journal. So fsck -y wont do a proper check unless the
journal replay fails. 
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Michael Powell 
Charles Swiger wrote:

[snip]
 
> Yes.  Without journalling, you'd normally perform the full timeconsuming
> fsck
> in the foreground.  With journalling, it should be able to do a journal
> replay to restore the filesystem to an OK state, but sometimes that
> doesn't restore consistency, in which case it usually fires off a
> background fsck rather than the foreground fsck.

In my case the journal replay failed, with an error to that effect. All 
partitions other than / failed to mount and after hitting enter at the 
.../bin/sh prompt performed manual fsck on all of them, which found and 
fixed some stuff. Then shutdown -r and everything came up fine (clean) 
afterwards. Net result was no data loss for me.

[snip]

-Mike


___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread RW 
On Mon, 14 Oct 2013 11:48:18 -0700
Charles Swiger wrote:



> Yes.  Without journalling, you'd normally perform the full
> timeconsuming fsck in the foreground.

Journalling removes the need for the background fsck which only recovers
lost space. 

>  With journalling, it should be
> able to do a journal replay to restore the filesystem to an OK state,

My understanding is that the journal does nothing to restore the
filesystem other than keep track of orphaned memory. In all other
respect it's the job of soft-updates to keep the filesystem in an OK
state. When it doesn't you need a foreground check.

> but sometimes that doesn't restore consistency, in which case it
> usually fires off a background fsck rather than the foreground fsck.

I think if the journal fails, you would really need to run at least a
foreground preen, maybe a full fsck. 
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Daniel Feenberg 



On Mon, 14 Oct 2013, Bruce Cran wrote:


On 10/14/2013 6:16 PM, CeDeROM wrote:

Isn't there Journal to prevent and reverse such damage?


Unlike other journaling filesystems, UFS+J only protects the metadata, not 
the data itself - i.e. I think it ensures you won't have to run a manual 
fsck, but just like plain old UFS files may be truncated as the journal is 
replayed.


This discussion skirts the critical issue - are files that are not open 
for writing endangered? No description of the uses of journaling can be 
considered informative if it doesn't address that explicitly. As a naive 
user I have always assumed that once closed, a file was invulnerable to 
improper shutdowns, but this discussion shakes that belief.


I expect the answer may be different for SSD and spinning disks.

dan feenberg
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread CeDeROM 
Thank you all for good hints! This will come handy! :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Charles Swiger 
On Oct 14, 2013, at 11:33 AM, CeDeROM  wrote:
> On Mon, Oct 14, 2013 at 7:54 PM, Bruce Cran  wrote:
>> On 10/14/2013 6:16 PM, CeDeROM wrote:
>>> Isn't there Journal to prevent and reverse such damage?
>> 
>> Unlike other journaling filesystems, UFS+J only protects the metadata, not
>> the data itself - i.e. I think it ensures you won't have to run a manual
>> fsck, but just like plain old UFS files may be truncated as the journal is
>> replayed.
> 
> Thank you for explaining :-) So it looks that it would be sensible to
> force filesystem check every n-th mount..?

You shouldn't ever need to recheck the filesystem if it was shutdown cleanly.

However, it doesn't hurt to fire off an fsck once a year or so just to look for
any unexpected issues.

> Or to do a filesystem check after crash..?

Yes.  Without journalling, you'd normally perform the full timeconsuming fsck
in the foreground.  With journalling, it should be able to do a journal replay 
to restore the filesystem to an OK state, but sometimes that doesn't restore
consistency, in which case it usually fires off a background fsck rather than
the foreground fsck.

> Are there any flags like that to mark filesystem
> unclean and to force fsck after n-th mount? That would assume
> disabling journal and soft updates journaling I guess..?

/etc/rc.conf should support something like the following to do what you ask:

fsck_y_enable="YES"
background_fsck="NO"
force_fsck="YES"

> What would be the best option for best data integrity in case of
> crash? That would be helpful for development systems I guess :-)

Well, you can use mount -o sync and disable write caching via hw.ata.wc=0 or
similar depending on what kind of drives you use.

This will cause a massive loss of write performance, but will greatly improve
reliability-- i.e. fsync() and such are not as likely to lie about whether bits
have made it to disk, even in the face of hardware which lies about 
ATA_FLUSHCACHE
(or SCSI "SYNCHRONIZE CACHE", etc).

Regards,
-- 
-Chuck

___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Adam Vande More 
On Mon, Oct 14, 2013 at 1:43 PM, Adam Vande More wrote:

>
>
> mount -o sync
>

should be

mount sync

-- 
Adam Vande More
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Adam Vande More 
On Mon, Oct 14, 2013 at 1:33 PM, CeDeROM  wrote:

> Thank you for explaining :-) So it looks that it would be sensible to
> force filesystem check every n-th mount..?


Please explain the logic in which this helps anything.


> Or to do a filesystem check
> after crash..?


Already standard behavior as implicitly seen in this thread.


> Are there any flags like that to mark filesystem
> unclean and to force fsck after n-th mount?


No and any fs that requires such a system is broken by design.


> That would assume
> disabling journal and soft updates journaling I guess..?
>
> What would be the best option for best data integrity in case of
> crash?
>

mount -o sync or use ZFS. Both require hardware that correctly report
success to fsync.

-- 
Adam Vande More
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Bruce Cran 

On 10/14/2013 7:33 PM, CeDeROM wrote:

Thank you for explaining :-) So it looks that it would be sensible to
force filesystem check every n-th mount..? Or to do a filesystem check
after crash..? Are there any flags like that to mark filesystem
unclean and to force fsck after n-th mount? That would assume
disabling journal and soft updates journaling I guess..?

What would be the best option for best data integrity in case of
crash? That would be helpful for development systems I guess :-)


As I understand it UFS+J gives the same reliability as UFS with a normal 
fsck after a crash, so on a development system the only ways to improve 
the situation would be to mount with the 'sync' option, disable write 
caching on the disk or to switch to a different filesystem like ZFS.


--
Bruce Cran
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread CeDeROM 
On Mon, Oct 14, 2013 at 7:54 PM, Bruce Cran  wrote:
> On 10/14/2013 6:16 PM, CeDeROM wrote:
>> Isn't there Journal to prevent and reverse such damage?
>
> Unlike other journaling filesystems, UFS+J only protects the metadata, not
> the data itself - i.e. I think it ensures you won't have to run a manual
> fsck, but just like plain old UFS files may be truncated as the journal is
> replayed.

Thank you for explaining :-) So it looks that it would be sensible to
force filesystem check every n-th mount..? Or to do a filesystem check
after crash..? Are there any flags like that to mark filesystem
unclean and to force fsck after n-th mount? That would assume
disabling journal and soft updates journaling I guess..?

What would be the best option for best data integrity in case of
crash? That would be helpful for development systems I guess :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread RW 
On Mon, 14 Oct 2013 18:34:36 +0200
David Demelier wrote:

> On 14.10.2013 14:39, RW wrote:

> > If you are having problems with data integrity you might try
> > gjournal or zfs instead.
> 
> Why? SU+J is enabled by default. Isn't the purpose of a journaled file
> system to ensure that any bad shutdown will protect data?

SU+J isn't a journalled filesytem, it's a filesystem with soft-updates
that journals information about free space so it can be recovered
without having to go through the whole filesystem.




___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Bruce Cran 

On 10/14/2013 6:16 PM, CeDeROM wrote:

Isn't there Journal to prevent and reverse such damage?


Unlike other journaling filesystems, UFS+J only protects the metadata, 
not the data itself - i.e. I think it ensures you won't have to run a 
manual fsck, but just like plain old UFS files may be truncated as the 
journal is replayed.
For ext3, https://www.kernel.org/doc/Documentation/filesystems/ext3.txt 
explains the different modes, with 'ordered' being default:


Data Mode
-
There are 3 different data modes:

* writeback mode
In data=writeback mode, ext3 does not journal data at all.  This mode provides
a similar level of journaling as that of XFS, JFS, and ReiserFS in its default
mode - metadata journaling.  A crash+recovery can cause incorrect data to
appear in files which were written shortly before the crash.  This mode will
typically provide the best ext3 performance.

* ordered mode
In data=ordered mode, ext3 only officially journals metadata, but it logically
groups metadata and data blocks into a single unit called a transaction.  When
it's time to write the new metadata out to disk, the associated data blocks
are written first.  In general, this mode performs slightly slower than
writeback but significantly faster than journal mode.

* journal mode
data=journal mode provides full data and metadata journaling.  All new data is
written to the journal first, and then to its final location.
In the event of a crash, the journal can be replayed, bringing both data and
metadata into a consistent state.  This mode is the slowest except when data
needs to be read from and written to disk at the same time where it
outperforms all other modes.



--
Bruce Cran
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread CeDeROM 
On Mon, Oct 14, 2013 at 7:09 PM, Brad Mettee  wrote:
> On 10/14/2013 12:50 PM, CeDeROM wrote:
>> Then why random files gets damaged as well even they are not
>> accessed/written on power loss? :-)
>
> Random files can be affected because the sectors of the hard disk containing
> the directory entries for those files, not the file data itself, may be
> damaged (ie: the directory was in the process of being written OR the
> pointer to that SECTOR was in the process of being written).
>
> It doesn't mean a file was in active use, just that a chunk of the disk with
> data relevant to that file was. Keep in mind, one sector of disk may have
> data for a dozen files in it (or more). Damage doesn't have to occur because
> a given file was in use at the time of a crash.

Isn't there Journal to prevent and reverse such damage?

> If your power grid is prone to failures or blips, I strongly suggest
> investing in a UPS.

I have UPS in my desktop and also I am working on a laptop, so the
power supply is not the only possible cause of system crash.. this may
be faulty driver, hardware failure, kernel panic, etc.

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Brad Mettee 

On 10/14/2013 12:50 PM, CeDeROM wrote:

On Mon, Oct 14, 2013 at 6:47 PM, Adam Vande More  wrote:

On Mon, Oct 14, 2013 at 11:34 AM, David Demelier
wrote:

Why? SU+J is enabled by default. Isn't the purpose of a journaled file
system to ensure that any bad shutdown will protect data?

As already stated, those measures are to preserve fs integrity eg meta data
is in sync.  It doesn't ensure that all the outstanding writes are
committed to disk in the event of a power outage.

Then why random files gets damaged as well even they are not
accessed/written on power loss? :-)
Random files can be affected because the sectors of the hard disk 
containing the directory entries for those files, not the file data 
itself, may be damaged (ie: the directory was in the process of being 
written OR the pointer to that SECTOR was in the process of being written).


It doesn't mean a file was in active use, just that a chunk of the disk 
with data relevant to that file was. Keep in mind, one sector of disk 
may have data for a dozen files in it (or more). Damage doesn't have to 
occur because a given file was in use at the time of a crash.


If your power grid is prone to failures or blips, I strongly suggest 
investing in a UPS.


Brad

___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread CeDeROM 
On Mon, Oct 14, 2013 at 6:56 PM, Adam Vande More  wrote:
> On Mon, Oct 14, 2013 at 11:50 AM, CeDeROM  wrote:
>> Then why random files gets damaged as well even they are not
>> accessed/written on power loss? :-)
> Prove they weren't.

Hmm, maybe /etc/pwd.db as David mentioned? This is updated on password
change, which does not happen all the time.. so why it was damaged
when no write occured..?

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Adam Vande More 
On Mon, Oct 14, 2013 at 11:50 AM, CeDeROM  wrote:

>
> Then why random files gets damaged as well even they are not
> accessed/written on power loss? :-)
>

Prove they weren't.

-- 
Adam Vande More
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread CeDeROM 
On Mon, Oct 14, 2013 at 6:47 PM, Adam Vande More  wrote:
> On Mon, Oct 14, 2013 at 11:34 AM, David Demelier
> wrote:
>>
>> Why? SU+J is enabled by default. Isn't the purpose of a journaled file
>> system to ensure that any bad shutdown will protect data?
>
> As already stated, those measures are to preserve fs integrity eg meta data
> is in sync.  It doesn't ensure that all the outstanding writes are
> committed to disk in the event of a power outage.

Then why random files gets damaged as well even they are not
accessed/written on power loss? :-)

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Adam Vande More 
On Mon, Oct 14, 2013 at 11:34 AM, David Demelier
wrote:

>
> Why? SU+J is enabled by default. Isn't the purpose of a journaled file
> system to ensure that any bad shutdown will protect data?
>

As already stated, those measures are to preserve fs integrity eg meta data
is in sync.  It doesn't ensure that all the outstanding writes are
committed to disk in the event of a power outage.

On GNU/Linux, on Windows you will not require anything else to recover
> your data.
>

This is complete garbage when using default settings as you imply below.
The default for ext3 on basically every distro still using ext3 is an
ordered journal and don't even get started on ext4.  NTFS by default
can/will also lose data on a power outage.


> I don't want to tweak the filesystem or use something different that the
> default, as it is the default it's the *warranty* that it is the correct
> way to protect data for new FreeBSD user's installations IMHO.
>

There is no *warranty* as explicitly stated in
http://www.freebsd.org/copyright/freebsd-license.html

The behavior you wish would slow down disk writes by an order of magnitude
and is already available to users willing to use non-default settings.

-- 
Adam Vande More
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread CeDeROM 
On Mon, Oct 14, 2013 at 6:34 PM, David Demelier
 wrote:
> Why? SU+J is enabled by default. Isn't the purpose of a journaled file
> system to ensure that any bad shutdown will protect data?
>
> On GNU/Linux, on Windows you will not require anything else to recover
> your data.
>
> I don't want to tweak the filesystem or use something different that the
> default, as it is the default it's the *warranty* that it is the correct
> way to protect data for new FreeBSD user's installations IMHO.

Agree :-) SU+J also seems to cause problems on SSD drives:

http://lists.freebsd.org/pipermail/freebsd-fs/2013-February/016420.html

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread David Demelier 
On 14.10.2013 14:39, RW wrote:
> On Mon, 14 Oct 2013 05:02:22 -0400
> Michael Powell wrote:
> 
>> David Demelier wrote:
>>
>>> Hello there,
>>>
>>> I'm writing because after a power failure I was unable to log in on
>>> my FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly
>>> but some files disappeared, including /etc/pwd.db. Thus I was
>>> unable to log in.
>>>
>>> I've been able to regenerate the password database with a live cd
>>> but I'm afraid that more files had disappeared somewhere else...
>>>
>>> I think this is a serious issue, the journal should not truncate
>>> files, so something should have gone wrong somewhere..
> 
> The journalling in  SU+J has nothing to do with data integrity.
> 
> When the system isn't shut-down cleanly, soft-updates are supposed to
> leave the filesystem in a self-consistent state, except that it may
> lose track of some freed disk space. The journal allows that space to
> be recovered without the lengthy background fsck that used to cripple
> performance.
> 
> If you are having problems with data integrity you might try gjournal or
> zfs instead.

Why? SU+J is enabled by default. Isn't the purpose of a journaled file
system to ensure that any bad shutdown will protect data?

On GNU/Linux, on Windows you will not require anything else to recover
your data.

I don't want to tweak the filesystem or use something different that the
default, as it is the default it's the *warranty* that it is the correct
way to protect data for new FreeBSD user's installations IMHO.

> If you look back at the lists before these were added
> there was a lot of suspicion about soft-updates and background checks.
> Some of the problems were explained by some (mostly desktop) drives
> incorrecty reporting what has been commited to disk - I don't know
> whether this is still the case.
> 
> 
>> This error about the replay of the journal(s) failing is somewhat 
>> disconcerting. 
> 
> I think this is probably a good thing. With background checks you would
> (if you were looking) occasionally see "unexpected soft-update
> inconsistency" during the background check, which would lead to a
> foreground check on the next boot.
> 
> 
> 
> ___
> [email protected] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "[email protected]"
> 

___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread RW 
On Mon, 14 Oct 2013 05:02:22 -0400
Michael Powell wrote:

> David Demelier wrote:
> 
> > Hello there,
> > 
> > I'm writing because after a power failure I was unable to log in on
> > my FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly
> > but some files disappeared, including /etc/pwd.db. Thus I was
> > unable to log in.
> > 
> > I've been able to regenerate the password database with a live cd
> > but I'm afraid that more files had disappeared somewhere else...
> > 
> > I think this is a serious issue, the journal should not truncate
> > files, so something should have gone wrong somewhere..

The journalling in  SU+J has nothing to do with data integrity.

When the system isn't shut-down cleanly, soft-updates are supposed to
leave the filesystem in a self-consistent state, except that it may
lose track of some freed disk space. The journal allows that space to
be recovered without the lengthy background fsck that used to cripple
performance.

If you are having problems with data integrity you might try gjournal or
zfs instead. If you look back at the lists before these were added
there was a lot of suspicion about soft-updates and background checks.
Some of the problems were explained by some (mostly desktop) drives
incorrecty reporting what has been commited to disk - I don't know
whether this is still the case.


> This error about the replay of the journal(s) failing is somewhat 
> disconcerting. 

I think this is probably a good thing. With background checks you would
(if you were looking) occasionally see "unexpected soft-update
inconsistency" during the background check, which would lead to a
foreground check on the next boot.



___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Michael Powell 
Michael Powell wrote:
[snip]
> The other box is my first foray into the land of GPT, along with SU+J. It
> was sitting at the 'couldn't mount... Press return for /bin/sh' line.
> There was an error indicating that replaying one or more journals had
> failed. I was able to successfully fsck all the other partitions (besides
> /), then rebooted and system came back up OK.

Meant to include also that I booted from a CD with wddiags and ran the Quick 
test and it found no errors on the disk.

[snip]
> 
> -Mike



___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-14 Thread Michael Powell 
David Demelier wrote:

> Hello there,
> 
> I'm writing because after a power failure I was unable to log in on my
> FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some
> files disappeared, including /etc/pwd.db. Thus I was unable to log in.
> 
> I've been able to regenerate the password database with a live cd but
> I'm afraid that more files had disappeared somewhere else...
> 
> I think this is a serious issue, the journal should not truncate files,
> so something should have gone wrong somewhere..
> 
> Any ideas? Should I open a PR?

Not sure there is enough to go on for a PR, but something is weird. 

Friday morning our power went down at home for about three hours after I had 
already left for work. When I came home I found the router/gateway box was 
OK. It is still with the old DOS mbr and disklabel scheme, with softupdates, 
and is a pair of disks gmirrored. 

The other box is my first foray into the land of GPT, along with SU+J. It 
was sitting at the 'couldn't mount... Press return for /bin/sh' line. There 
was an error indicating that replaying one or more journals had failed. I 
was able to successfully fsck all the other partitions (besides /), then 
rebooted and system came back up OK.

Both of these machines were recently updated to 9.2 Release from 9.1. It has 
been approximately 9 months, or so, since I last had a power outage like 
this one. Back then they were still 8.3 I think, did not have SU+J and 
recovered just fine on their own.  

This error about the replay of the journal(s) failing is somewhat 
disconcerting. Beyond that, however, I do not have any other details or 
data. Nothing to flesh out a PR, but thought I'd mention what I saw in 
conjunction with your experience.

-Mike



___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-13 Thread Thomas Mueller 
> On 13.10.2013 12:16, CeDeROM wrote:
> > On 13 Oct 2013 11:30, "David Demelier"  > > wrote:
> >> Hello there,
> >> I'm writing because after a power failure I was unable to log in on my
> >> FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some
> >> files disappeared, including /etc/pwd.db. Thus I was unable to log in.
> >> I've been able to regenerate the password database with a live cd but
> >> I'm afraid that more files had disappeared somewhere else...
> >> I think this is a serious issue, the journal should not truncate files,
> >> so something should have gone wrong somewhere..
> >> Any ideas? Should I open a PR

> > I had similar issues somewhere around 9.0 - although journal check was
> > fine running fsck revealed filesystem inconsistency. I have reported
> > this on the list, but it seemed unnoticed..? For me this is serious
> > issue as well, if you make PR I will give +1 :-)

> > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

> Yes, I've also ran fsck in single user mode after and lot of incorrect
> things were corrected, I wait a bit for answers (if any) before sending
> a PR.

Running fsck in single-user mode may not be sufficient.

You may need to run fsck_ffs from a USB-stick installation or live CD.

I remember reviving a FreeBSD partition that way, normal root partition not 
mounted.

I once revived a FreeBSD partition with fsck_ffs from a USB-stick installation 
of NetBSD 5.1_STABLE i386 after FreeBSD couldn't do it.

It helps to have a UPS to protect against short power failures and allow 
graceful shutdown on longer power outages.


Tom

___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-13 Thread David Demelier 
On 13.10.2013 12:16, CeDeROM wrote:
> On 13 Oct 2013 11:30, "David Demelier"  > wrote:
>> Hello there,
>> I'm writing because after a power failure I was unable to log in on my
>> FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some
>> files disappeared, including /etc/pwd.db. Thus I was unable to log in.
>> I've been able to regenerate the password database with a live cd but
>> I'm afraid that more files had disappeared somewhere else...
>> I think this is a serious issue, the journal should not truncate files,
>> so something should have gone wrong somewhere..
>> Any ideas? Should I open a PR
> 
> I had similar issues somewhere around 9.0 - although journal check was
> fine running fsck revealed filesystem inconsistency. I have reported
> this on the list, but it seemed unnoticed..? For me this is serious
> issue as well, if you make PR I will give +1 :-)
> 
> --
> CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
> 

Yes, I've also ran fsck in single user mode after and lot of incorrect
things were corrected, I wait a bit for answers (if any) before sending
a PR.

Cheers,
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

Re: SU+J Lost files after a power failure

2013-10-13 Thread CeDeROM 
On 13 Oct 2013 11:30, "David Demelier"  wrote:
> Hello there,
> I'm writing because after a power failure I was unable to log in on my
> FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some
> files disappeared, including /etc/pwd.db. Thus I was unable to log in.
> I've been able to regenerate the password database with a live cd but
> I'm afraid that more files had disappeared somewhere else...
> I think this is a serious issue, the journal should not truncate files,
> so something should have gone wrong somewhere..
> Any ideas? Should I open a PR

I had similar issues somewhere around 9.0 - although journal check was fine
running fsck revealed filesystem inconsistency. I have reported this on the
list, but it seemed unnoticed..? For me this is serious issue as well, if
you make PR I will give +1 :-)

--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"

SU+J Lost files after a power failure

2013-10-13 Thread David Demelier 
Hello there,

I'm writing because after a power failure I was unable to log in on my
FreeBSD 9.2-RELEASE. The SU+J journal were executed correctly but some
files disappeared, including /etc/pwd.db. Thus I was unable to log in.

I've been able to regenerate the password database with a live cd but
I'm afraid that more files had disappeared somewhere else...

I think this is a serious issue, the journal should not truncate files,
so something should have gone wrong somewhere..

Any ideas? Should I open a PR?

Regards,
___
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "[email protected]"