On 05/17, Raouf Rokhjavan wrote:
> On 05/12/17 04:44, Jaegeuk Kim wrote:
> > Hi,
> > 
> > On 05/10, Raouf Rokhjavan wrote:
> > ...
> > 
> > > As you told to use snapshot mechanism to prevent changing ckpt number 
> > > after
> > > each mount, I ran again generic tests of xfstests framework on top of
> > > log-writes target with f2fs file system. In order to automate reporting an
> > > inconsistency situation, I add a parameter to fsck.f2fs to return(-1) when
> > > c.bug_on condition is met. To evaluate how f2fs react in case of crash
> > > consistency, I replay each log and check the consistency of f2fs with a my
> > > own modified version of fsck.f2fs.  Accordingly, all tests passed smoothly
> > > except these tests:
> > > 
> > > [FAIL] Running generic/013 failed. (consistency_single)
> > Could you check whether any IO made by mkfs was added in the replay log?
> > If so, fsck.f2fs should be failed when replaying them.
> > 
> > > [FAIL] Running generic/070 failed. (consistency_single)
> > > [FAIL] Running generic/113 failed. (consistency_single)
> > I added a mark to replay in the beginning of generic/113, and ran the test.
> > But, I couldn't find any error given test_dev as a log_dev. (I tested this
> > in the latest f2fs/dev-test branch.)
> > 
> > > [FAIL] Running generic/241 failed. (consistency_single)
> > > 
> > > In other words, in these tests, c.bug_on() was true. Would you please
> > > describe why they become inconsistent?
> > > 
> > > Besides, I ran sysbench for database benchmark with 1 thread, 1000 
> > > records,
> > > and 100 transactions on top of log-writes target with f2fs. 
> > > Interestingly, I
> > > encountered a weird inconsistency. After replaying about 100 logs, 
> > > fsck.f2fs
> > > complains about inconsistency with the following messages:
> > Can you share the parameter for sysbench?
> Hi,
> 
> Since I want to make sure that my system, having a database app, stay
> operational after the power failure, I test database system on top of f2fs.
> Accordingly, I use sysbench and dm-log-writes to serve this purpose. I took
> advantage of lua scripting facility in sysbench to implement write only
> operations in database:
> 
> #sysbench 
> --test=/home/roraouf/Projects/CrashConsistencyTest/locals/var/lib/dbtests/sysbench-lua/tests/db/oltp_write_only.lua
> --db-driver=mysql --oltp-table-size=1000 --mysql-db=sysbench
> --mysql-user=sysbench --mysql-password=password --max-requests=100
> --num-threads=1 --mysql-socket=/mnt/crash_consistency/f2fs/mysql/mysql.sock
> run
> 
> I ran this test on 3 configurations:
> 1- ext4 (ordered, noatime) - success 15/15
> 2- ext4 (norecovery, noatime) - success 0/15
> 3- f2fs (noatime) - success 3/15
> 
> Success, here, means whether file system is operational without running fsck
> and fixing after each replay.
> As the result show, ext4 with ordered journaling could surmount this test,
> but ,as it had been expected, ext4 without journaling like ext2 needs fsck
> to recover file system after simulated power loss.
> The surprising part of this test is f2fs. As f2fs always maintains a stable
> checkpoint of file system, and based on its FAST paper, it always rolls back
> to its stable checkpoint after power loss, I didn't expect to see f2fs in
> inconsistent state after replaying logs as fsck.f2fs reports. (It's
> necessary to mention that we check consistency of f2fs after mkfs.f2fs.
> ext4's results verify this notion.)
> 
> Unfortunately, the results are not reproducible, and inconsistency occurs in
> different logs; moreover, fsck.f2fs passes this test occasionally.
> To give more accurate information, I uploaded the output of fsck.f2fs on
> Google Drive.
> 
> https://drive.google.com/open?id=0BxdqCs3G6wd3UWtDTmRGbFBiYmc

Hi,

Could you please check:
- did you use a snapshot device?
- what command was issued at #1687?
- how's result of fsck.f2fs -d 3?
- can you share your log-dev image?

Thanks,

> 
> Regards,
> > 
> > Thanks,
> > 
> > > Info: Segments per section = 1
> > > Info: Sections per zone = 1
> > > Info: sector size = 512
> > > Info: total sectors = 2097152 (1024 MB)
> > > Info: MKFS version
> > >    "Linux version 4.9.8 (rora...@desktopr.example.com) (gcc version 4.8.5
> > > 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Feb 7 08:24:57 IRST 2017"
> > > Info: FSCK version
> > >    from "Linux version 4.9.8 (rora...@desktopr.example.com) (gcc version
> > > 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Feb 7 08:24:57 IRST
> > > 2017"
> > >      to "Linux version 4.9.8 (rora...@desktopr.example.com) (gcc version
> > > 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Feb 7 08:24:57 IRST
> > > 2017"
> > > Info: superblock features = 0 :
> > > Info: superblock encrypt level = 0, salt = 
> > > 00000000000000000000000000000000
> > > Info: total FS sectors = 2097152 (1024 MB)
> > > Info: CKPT version = 2b59c128
> > > Info: checkpoint state = 44 :  compacted_summary sudden-power-off
> > > [ASSERT] (sanity_check_nid: 388)  --> nid[0x6] nat_entry->ino[0x6]
> > > footer.ino[0x0]
> > > 
> > > NID[0x6] is unreachable
> > > NID[0x7] is unreachable
> > > [FSCK] Unreachable nat entries                        [Fail] [0x2]
> > > [FSCK] SIT valid block bitmap checking                [Fail]
> > > [FSCK] Hard link checking for regular file            [Ok..] [0x0]
> > > [FSCK] valid_block_count matching with CP             [Fail] [0x6dc9]
> > > [FSCK] valid_node_count matcing with CP (de lookup)   [Fail] [0xe3]
> > > [FSCK] valid_node_count matcing with CP (nat lookup)  [Ok..] [0xe5]
> > > [FSCK] valid_inode_count matched with CP              [Fail] [0x63]
> > > [FSCK] free segment_count matched with CP             [Ok..] [0x1c6]
> > > [FSCK] next block offset is free                      [Ok..]
> > > [FSCK] fixing SIT types
> > > [FSCK] other corrupted bugs                           [Fail]
> > > 
> > > After canceling the test by using Ctrl-C without answering any YES/NO
> > > questions, on another terminal I run fsck.f2fs again, but the output is
> > > completely different:
> > > [root@localhost CrashConsistencyTest]# ./locals/usr/local/sbin/fsck.f2fs
> > > /dev/sdc
> > > Info: [/dev/sdc] Disk Model: VMware Virtual S1.0
> > > Info: Segments per section = 1
> > > Info: Sections per zone = 1
> > > Info: sector size = 512
> > > Info: total sectors = 2097152 (1024 MB)
> > > Info: MKFS version
> > >    "Linux version 4.9.8 (rora...@desktopr.example.com) (gcc version 4.8.5
> > > 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Feb 7 08:24:57 IRST 2017"
> > > Info: FSCK version
> > >    from "Linux version 4.9.8 (rora...@desktopr.example.com) (gcc version
> > > 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Feb 7 08:24:57 IRST
> > > 2017"
> > >      to "Linux version 4.9.8 (rora...@desktopr.example.com) (gcc version
> > > 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Feb 7 08:24:57 IRST
> > > 2017"
> > > Info: superblock features = 0 :
> > > Info: superblock encrypt level = 0, salt = 
> > > 00000000000000000000000000000000
> > > Info: total FS sectors = 2097152 (1024 MB)
> > > Info: CKPT version = 2b59c128
> > > Info: checkpoint state = 44 :  compacted_summary sudden-power-off
> > > 
> > > [FSCK] Unreachable nat entries                        [Ok..] [0x0]
> > > [FSCK] SIT valid block bitmap checking                [Ok..]
> > > [FSCK] Hard link checking for regular file            [Ok..] [0x0]
> > > [FSCK] valid_block_count matching with CP             [Ok..] [0x6dcf]
> > > [FSCK] valid_node_count matcing with CP (de lookup)   [Ok..] [0xe5]
> > > [FSCK] valid_node_count matcing with CP (nat lookup)  [Ok..] [0xe5]
> > > [FSCK] valid_inode_count matched with CP              [Ok..] [0x64]
> > > [FSCK] free segment_count matched with CP             [Ok..] [0x1c6]
> > > [FSCK] next block offset is free                      [Ok..]
> > > [FSCK] fixing SIT types
> > > [FSCK] other corrupted bugs                           [Ok..]
> > > 
> > > This situation raises a couple of questions:
> > > 1. How  does an inconsistent file system turn into a consistent one in 
> > > this
> > > case?
> > > 2. Why does an inconsistency occur in different log numbers; in other 
> > > words,
> > > why is it unpredictable?  Does ordering of logs have to do with disk
> > > controller and I/O scheduler?
> > > 
> > > I do appreciate for your help.
> > > Regards

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Reply via email to