[f2fs-dev] [PATCH] common/attr: adbjust acl_max of f2fs
f2fs has set inline_xattr as a default option, and introduced a new option named 'noinline_xattr' for disabling default inline_xattr option. So in _acl_get_max we need to check 'noinline_xattr' string in fs option, otherwise we may select the wrong max acl number since we always found the string 'inline_xattr' in fs option. Additionally, f2fs has changed disk layout of xattr block a bit, so will contain one more entry in both inline and noinline xattr inode, this patch will modify the max acl number to adjust it. Suggested-by: Chao Yu Signed-off-by: Sun Ke --- common/attr | 19 --- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/common/attr b/common/attr index 35682d7c..dae8a1bb 100644 --- a/common/attr +++ b/common/attr @@ -26,11 +26,24 @@ _acl_get_max() echo 8191 ;; f2fs) - _fs_options $TEST_DEV | grep "inline_xattr" >/dev/null 2>&1 + # If noinline_xattr is enabled, max xattr size should be: + # (4096 - 24) - (24 + 4) = 4044 + # then ACL_MAX_ENTRIES should be: + # (4044 - (4 + 4 * 4)) / 8 + 4 = 507 + _fs_options $TEST_DEV | grep "noinline_xattr" >/dev/null 2>&1 if [ $? -eq 0 ]; then - echo 531 + echo 507 else - echo 506 + # If inline_xattr is enabled, max xattr size should be: + # (4096 - 24 + 200) - (24 + 4) = 4244 + # then ACL_MAX_ENTRIES should be: + # (4244 - (4 + 4 * 4)) / 8 + 4 = 532 + _fs_options $TEST_DEV | grep "inline_xattr" >/dev/null 2>&1 + if [ $? -eq 0 ]; then + echo 532 + else + echo 507 + fi fi ;; bcachefs) -- 2.13.6 ___ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH] common/attr: adbjust acl_max of f2fs
Friendly ping... 在 2022/2/8 15:16, Sun Ke 写道: f2fs has set inline_xattr as a default option, and introduced a new option named 'noinline_xattr' for disabling default inline_xattr option. So in _acl_get_max we need to check 'noinline_xattr' string in fs option, otherwise we may select the wrong max acl number since we always found the string 'inline_xattr' in fs option. Additionally, f2fs has changed disk layout of xattr block a bit, so will contain one more entry in both inline and noinline xattr inode, this patch will modify the max acl number to adjust it. Suggested-by: Chao Yu Signed-off-by: Sun Ke --- common/attr | 19 --- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/common/attr b/common/attr index 35682d7c..dae8a1bb 100644 --- a/common/attr +++ b/common/attr @@ -26,11 +26,24 @@ _acl_get_max() echo 8191 ;; f2fs) - _fs_options $TEST_DEV | grep "inline_xattr" >/dev/null 2>&1 + # If noinline_xattr is enabled, max xattr size should be: + # (4096 - 24) - (24 + 4) = 4044 + # then ACL_MAX_ENTRIES should be: + # (4044 - (4 + 4 * 4)) / 8 + 4 = 507 + _fs_options $TEST_DEV | grep "noinline_xattr" >/dev/null 2>&1 if [ $? -eq 0 ]; then - echo 531 + echo 507 else - echo 506 + # If inline_xattr is enabled, max xattr size should be: + # (4096 - 24 + 200) - (24 + 4) = 4244 + # then ACL_MAX_ENTRIES should be: + # (4244 - (4 + 4 * 4)) / 8 + 4 = 532 + _fs_options $TEST_DEV | grep "inline_xattr" >/dev/null 2>&1 + if [ $? -eq 0 ]; then + echo 532 + else + echo 507 + fi fi ;; bcachefs) ___ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
[f2fs-dev] [PATCH] generic/066: attr1 is still there after log replay on f2fs
The test fail on f2fs:
xattr names and values after second fsync log replay:
# file: SCRATCH_MNT/foobar
+user.attr1="val1"
user.attr3="val3"
attr1 is still there after log replay.
I guess it is f2fs's special feature to improve the performance.
Signed-off-by: Sun Ke
---
Is it a BUG on f2fs?
.gitignore | 1 +
tests/generic/066 | 3 +++
tests/generic/066.cfg | 1 +
tests/generic/{066.out => 066.out.default} | 0
tests/generic/066.out.f2fs | 11 +++
5 files changed, 16 insertions(+)
create mode 100644 tests/generic/066.cfg
rename tests/generic/{066.out => 066.out.default} (100%)
create mode 100644 tests/generic/066.out.f2fs
diff --git a/.gitignore b/.gitignore
index 65b93307..135742f5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -194,6 +194,7 @@ tags
/src/perf/*.pyc
# Symlinked files
+/tests/generic/066.out
/tests/generic/035.out
/tests/generic/050.out
/tests/xfs/033.out
diff --git a/tests/generic/066 b/tests/generic/066
index 105a7acd..524dc82d 100755
--- a/tests/generic/066
+++ b/tests/generic/066
@@ -17,6 +17,7 @@
#
# Btrfs: remove deleted xattrs on fsync log replay
#
+seqfull=$0
. ./common/preamble
_begin_fstest auto quick attr metadata log
@@ -32,6 +33,8 @@ _cleanup()
. ./common/dmflakey
. ./common/attr
+# Select appropriate golden output based on fstype
+_link_out_file
# real QA test starts here
_supported_fs generic
_require_scratch
diff --git a/tests/generic/066.cfg b/tests/generic/066.cfg
new file mode 100644
index ..c25641be
--- /dev/null
+++ b/tests/generic/066.cfg
@@ -0,0 +1 @@
+f2fs: f2fs
diff --git a/tests/generic/066.out b/tests/generic/066.out.default
similarity index 100%
rename from tests/generic/066.out
rename to tests/generic/066.out.default
diff --git a/tests/generic/066.out.f2fs b/tests/generic/066.out.f2fs
new file mode 100644
index ..8fc58693
--- /dev/null
+++ b/tests/generic/066.out.f2fs
@@ -0,0 +1,11 @@
+QA output created by 066
+xattr names and values after first fsync log replay:
+# file: SCRATCH_MNT/foobar
+user.attr1="val1"
+user.attr3="val3"
+
+xattr names and values after second fsync log replay:
+# file: SCRATCH_MNT/foobar
+user.attr1="val1"
+user.attr3="val3"
+
--
2.13.6
___
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH] generic/066: attr1 is still there after log replay on f2fs
Friendly ping...
在 2022/2/28 11:57, Sun Ke 写道:
The test fail on f2fs:
xattr names and values after second fsync log replay:
# file: SCRATCH_MNT/foobar
+user.attr1="val1"
user.attr3="val3"
attr1 is still there after log replay.
I guess it is f2fs's special feature to improve the performance.
Signed-off-by: Sun Ke
---
Is it a BUG on f2fs?
.gitignore | 1 +
tests/generic/066 | 3 +++
tests/generic/066.cfg | 1 +
tests/generic/{066.out => 066.out.default} | 0
tests/generic/066.out.f2fs | 11 +++
5 files changed, 16 insertions(+)
create mode 100644 tests/generic/066.cfg
rename tests/generic/{066.out => 066.out.default} (100%)
create mode 100644 tests/generic/066.out.f2fs
diff --git a/.gitignore b/.gitignore
index 65b93307..135742f5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -194,6 +194,7 @@ tags
/src/perf/*.pyc
# Symlinked files
+/tests/generic/066.out
/tests/generic/035.out
/tests/generic/050.out
/tests/xfs/033.out
diff --git a/tests/generic/066 b/tests/generic/066
index 105a7acd..524dc82d 100755
--- a/tests/generic/066
+++ b/tests/generic/066
@@ -17,6 +17,7 @@
#
# Btrfs: remove deleted xattrs on fsync log replay
#
+seqfull=$0
. ./common/preamble
_begin_fstest auto quick attr metadata log
@@ -32,6 +33,8 @@ _cleanup()
. ./common/dmflakey
. ./common/attr
+# Select appropriate golden output based on fstype
+_link_out_file
# real QA test starts here
_supported_fs generic
_require_scratch
diff --git a/tests/generic/066.cfg b/tests/generic/066.cfg
new file mode 100644
index ..c25641be
--- /dev/null
+++ b/tests/generic/066.cfg
@@ -0,0 +1 @@
+f2fs: f2fs
diff --git a/tests/generic/066.out b/tests/generic/066.out.default
similarity index 100%
rename from tests/generic/066.out
rename to tests/generic/066.out.default
diff --git a/tests/generic/066.out.f2fs b/tests/generic/066.out.f2fs
new file mode 100644
index ..8fc58693
--- /dev/null
+++ b/tests/generic/066.out.f2fs
@@ -0,0 +1,11 @@
+QA output created by 066
+xattr names and values after first fsync log replay:
+# file: SCRATCH_MNT/foobar
+user.attr1="val1"
+user.attr3="val3"
+
+xattr names and values after second fsync log replay:
+# file: SCRATCH_MNT/foobar
+user.attr1="val1"
+user.attr3="val3"
+
___
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH] generic/066: attr1 is still there after log replay on f2fs
在 2022/3/10 15:33, Chao Yu 写道:
On 2022/3/10 9:41, Dave Chinner wrote:
On Wed, Mar 09, 2022 at 03:34:27PM +0800, Chao Yu wrote:
On 2022/3/9 14:22, Dave Chinner wrote:
On Wed, Mar 09, 2022 at 12:31:17PM +0800, Chao Yu wrote:
On 2022/2/28 11:57, Sun Ke via Linux-f2fs-devel wrote:
The test fail on f2fs:
xattr names and values after second fsync log replay:
# file: SCRATCH_MNT/foobar
+user.attr1="val1"
user.attr3="val3"
attr1 is still there after log replay.
I guess it is f2fs's special feature to improve the performance.
Signed-off-by: Sun Ke
---
Is it a BUG on f2fs?
I don't think so, it fails due to f2fs doesn't follow recovery rule
which
btrfs/ext4/xfs does, but it doesn't mean f2fs has break posix
semantics of
fsync().
I disagree. A failure in this test is indicative of non-conformance
with the Linux definition of fsync() behaviour.
You are right in that it does not break POSIX fsync semantics, but
POSIX allows "do nothing" as a valid implementation. However,
because of this loophole, the POSIX definition is complete garbage
and we do not use it.
That behaviour that Linux filesytsems are supposed to implement is
defined in the Linux fsync() man page, and it goes way beyond what
POSIX requires:
$ man fsync
DESCRIPTION
fsync() transfers ("flushes") all modified in-core data of
(i.e., modified buffer cache pages for) the file referred to by
the file descriptor fd to the disk device (or other permanent
storage device) so that all changed information can be retrieved
even if the system crashes or is rebooted. This includes
writing through or flushing a disk cache if present. The call
blocks until the device reports that the transfer has
completed.
As well as flushing the file data, fsync() also flushes the
metadata information associated with the file (see inode(7)).
IOWs, fsync() on Linux is supposed to persist all data and
metadata associated with the inode to stable storage such that it
can be retreived after a crash or reboot. "metadata information"
includes xattrs attached to the inode that is being fsync()d.
Quoted from g/066:
echo "hello world!" >> $SCRATCH_MNT/foobar
$XFS_IO_PROG -c "fsync" $SCRATCH_MNT/foobar
$SETFATTR_PROG -x user.attr1 $SCRATCH_MNT/foobar
ln $SCRATCH_MNT/foobar $SCRATCH_MNT/foobar_link
touch $SCRATCH_MNT/qwerty
$XFS_IO_PROG -c "fsync" $SCRATCH_MNT/qwerty
IIUC, to match what Linux fsync() manual restricts, if we want to
persist the
xattr removal, we should call fsync() on $SCRATCH_MNT/foobar after
"$SETFATTR_PROG -x user.attr1 $SCRATCH_MNT/foobar"? rather than
calling fsync()
on unrelated $SCRATCH_MNT/qwerty.
It might look that way, but it's not that straight forward: there's
a carefully constructed object dependency chain in this test that
defines what the correct behaviour should be here.
What's the link count of $SCRATCH_MNT/foobar when
$SCRATCH_MNT/qwerty is present after recovery? Is it 1 or 2? Does
$SCRATCH_MNT/foobar_link exist? And if $SCRATCH_MNT/foobar_link
exists, and the link count is 2. The test doesn't even look at these
things, but if user.attr1 is not present, it means that foobar_link
and qwerty are present, $SCRATCH_MNT has a link count of 5 and
foobar has a link count of 2 because that's the dependency chain
that leads to the user.attr1 removal being recovered correctly.
So what does SCRATCH_MNT actually contain when f2fs fails this test?
After f2fs recovery,
SCRATCH_MNT contains two files: foobar and qwerty, link count of both
files is 1, and foobar has two xattr entries: user.attr1 and user.attr3.
So it means, f2fs only recover file/directory which has been fsync()ed
before
SPO... since f2fs doesn't support fs-op level transaction functionality,
so it
have no way to persist all metadata updates in one transaction.
There is one alternative method to pass this case, as I suggested, we can
use "fastboot" mountoption for this case, so during last fsync on qwerty,
f2fs can trigger a checkpoint which will persist all metadata updates
before
fsync()...
Thanks,
The test can pass by using "fastboot" mountoption. I will send v2.
Thanks,
Sun Ke
These depedencies exist because we can't just randomly re-order
recovery of modifications to individual inodes and certain
operations create atomic change dependencies between inodes. It's
those atomic change dependencies that this test is actually
exercising. i.e. the link count changes tie directory modifications
to inode modifications and this creates cross-object ordering
dependencies down the line that fsync then exposes. That's what the
second part of this test is actually exercising
Cheers,
Dave.
.
___
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
[f2fs-dev] [PATCH v2] generic/066: attr1 is still there after log replay on f2fs
The test fail on f2fs: xattr names and values after second fsync log replay: # file: SCRATCH_MNT/foobar +user.attr1="val1" user.attr3="val3" attr1 is still there after log replay. f2fs doesn't support fs-op level transaction functionality. so it have no way to persist all metadata updates in one transaction. We can use "fastboot" mountoption for this case, so during last fsync on qwerty, f2fs can trigger a checkpoint which will persist all metadata updates before fsync(). Suggested-by: Chao Yu Signed-off-by: Sun Ke --- tests/generic/066 | 8 1 file changed, 8 insertions(+) diff --git a/tests/generic/066 b/tests/generic/066 index 105a7acd..b8465101 100755 --- a/tests/generic/066 +++ b/tests/generic/066 @@ -56,6 +56,14 @@ sync $SETFATTR_PROG -x user.attr2 $SCRATCH_MNT/foobar $XFS_IO_PROG -c "fsync" $SCRATCH_MNT/foobar +# f2fs doesn't support fs-op level transaction functionality. so it have no way +# to persist all metadata updates in one transaction. We can use "fastboot" +# mountoption for this case, so during last fsync on qwerty, f2fs can trigger a +# checkpoint which will persist all metadata updates before fsync(). +if [ $FSTYP = "f2fs" ]; then + export MOUNT_OPTIONS="-o fastboot" +fi + _flakey_drop_and_remount # After the fsync log is replayed, the file should have only 2 xattrs, the ones -- 2.13.6 ___ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH v2] generic/066: attr1 is still there after log replay on f2fs
在 2022/3/15 12:01, Chao Yu 写道: On 2022/3/11 17:43, Sun Ke via Linux-f2fs-devel wrote: The test fail on f2fs: xattr names and values after second fsync log replay: # file: SCRATCH_MNT/foobar +user.attr1="val1" user.attr3="val3" attr1 is still there after log replay. f2fs doesn't support fs-op level transaction functionality. so it have no way to persist all metadata updates in one transaction. We can use "fastboot" mountoption for this case, so during last fsync on qwerty, f2fs can trigger a checkpoint which will persist all metadata updates before fsync(). Suggested-by: Chao Yu Signed-off-by: Sun Ke --- tests/generic/066 | 8 1 file changed, 8 insertions(+) diff --git a/tests/generic/066 b/tests/generic/066 index 105a7acd..b8465101 100755 --- a/tests/generic/066 +++ b/tests/generic/066 @@ -56,6 +56,14 @@ sync $SETFATTR_PROG -x user.attr2 $SCRATCH_MNT/foobar $XFS_IO_PROG -c "fsync" $SCRATCH_MNT/foobar +# f2fs doesn't support fs-op level transaction functionality. so it have no way +# to persist all metadata updates in one transaction. We can use "fastboot" +# mountoption for this case, so during last fsync on qwerty, f2fs can trigger a +# checkpoint which will persist all metadata updates before fsync(). +if [ $FSTYP = "f2fs" ]; then + export MOUNT_OPTIONS="-o fastboot" Can we merge "fastboot" with original mount options? and after the test, how about recovering to original mount options? Thanks, Like this: if [ $FSTYP = "f2fs" ]; then - export MOUNT_OPTIONS="-o fastboot" + export MOUNT_OPTIONS="-o fastboot $MOUNT_OPTIONS" fi right? And I have tested that there is no need to recover to original mount options after the test. It will be the original one in the next test. Thanks, Sun Ke +fi + _flakey_drop_and_remount # After the fsync log is replayed, the file should have only 2 xattrs, the ones . ___ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
[f2fs-dev] [PATCH v3] generic/066: attr1 is still there after log replay on f2fs
The test fail on f2fs: xattr names and values after second fsync log replay: # file: SCRATCH_MNT/foobar +user.attr1="val1" user.attr3="val3" attr1 is still there after log replay. f2fs doesn't support fs-op level transaction functionality. so it have no way to persist all metadata updates in one transaction. We can use "fastboot" mountoption for this case, so during last fsync on qwerty, f2fs can trigger a checkpoint which will persist all metadata updates before fsync(). Suggested-by: Chao Yu Signed-off-by: Sun Ke --- v3: merge "fastboot" with original mount options tests/generic/066 | 8 1 file changed, 8 insertions(+) diff --git a/tests/generic/066 b/tests/generic/066 index 105a7acd..d8a38655 100755 --- a/tests/generic/066 +++ b/tests/generic/066 @@ -56,6 +56,14 @@ sync $SETFATTR_PROG -x user.attr2 $SCRATCH_MNT/foobar $XFS_IO_PROG -c "fsync" $SCRATCH_MNT/foobar +# f2fs doesn't support fs-op level transaction functionality. so it have no way +# to persist all metadata updates in one transaction. We can use "fastboot" +# mountoption for this case, so during last fsync on qwerty, f2fs can trigger a +# checkpoint which will persist all metadata updates before fsync(). +if [ $FSTYP = "f2fs" ]; then + export MOUNT_OPTIONS="-o fastboot $MOUNT_OPTIONS" +fi + _flakey_drop_and_remount # After the fsync log is replayed, the file should have only 2 xattrs, the ones -- 2.13.6 ___ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
