Re: [f2fs-dev] [PATCH v2 10/12] ext4: add basic fs-verity support
On Tuesday, November 6, 2018 6:55:03 AM IST Eric Biggers wrote:
> Hi Chandan,
>
> On Fri, Nov 02, 2018 at 03:13:14PM +0530, Chandan Rajendra wrote:
> > On Friday, November 2, 2018 4:22:28 AM IST Eric Biggers wrote:
> > > From: Eric Biggers
> > >
> > > Add basic fs-verity support to ext4. fs-verity is a filesystem feature
> > > that enables transparent integrity protection and authentication of
> > > read-only files. It uses a dm-verity like mechanism at the file level:
> > > a Merkle tree is used to verify any block in the file in log(filesize)
> > > time. It is implemented mainly by helper functions in fs/verity/.
> > > See Documentation/filesystems/fsverity.rst for details.
> > >
> > > This patch adds everything except the data verification hooks that will
> > > needed in ->readpages().
> > >
> > > On ext4, enabling fs-verity on a file requires that the filesystem has
> > > the 'verity' feature, e.g. that it was formatted with
> > > 'mkfs.ext4 -O verity' or had 'tune2fs -O verity' run on it.
> > > This requires e2fsprogs 1.44.4-2 or later.
> > >
> > > In ext4, we choose to retain the fs-verity metadata past the end of the
> > > file rather than trying to move it into an external inode xattr, since
> > > in practice keeping the metadata in-line actually results in the
> > > simplest and most efficient implementation. One non-obvious advantage
> > > of keeping the verity metadata in-line is that when fs-verity is
> > > combined with fscrypt, the verity metadata naturally gets encrypted too;
> > > this is actually necessary because it contains hashes of the plaintext.
> > >
> > > We also choose to keep the on-disk i_size equal to the original file
> > > size, in order to make the 'verity' feature a RO_COMPAT feature. Thus,
> > > ext4 has to find the fsverity_footer by looking in the last extent.
> > >
> > > Co-developed-by: Theodore Ts'o
> > > Signed-off-by: Theodore Ts'o
> > > Signed-off-by: Eric Biggers
> > > ---
> > > fs/ext4/Kconfig | 20 +++
> > > fs/ext4/ext4.h | 20 ++-
> > > fs/ext4/file.c | 6
> > > fs/ext4/inode.c | 8 +
> > > fs/ext4/ioctl.c | 12 +++
> > > fs/ext4/super.c | 91 +
> > > fs/ext4/sysfs.c | 6
> > > 7 files changed, 162 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/fs/ext4/Kconfig b/fs/ext4/Kconfig
> > > index a453cc87082b5..5a76125ac0f8a 100644
> > > --- a/fs/ext4/Kconfig
> > > +++ b/fs/ext4/Kconfig
> > > @@ -111,6 +111,26 @@ config EXT4_FS_ENCRYPTION
> > > default y
> > > depends on EXT4_ENCRYPTION
> > >
> > > +config EXT4_FS_VERITY
> > > + bool "Ext4 Verity"
> > > + depends on EXT4_FS
> > > + select FS_VERITY
> > > + help
> > > + This option enables fs-verity for ext4. fs-verity is the
> > > + dm-verity mechanism implemented at the file level. Userspace
> > > + can append a Merkle tree (hash tree) to a file, then enable
> > > + fs-verity on the file. ext4 will then transparently verify
> > > + any data read from the file against the Merkle tree. The file
> > > + is also made read-only.
> > > +
> > > + This serves as an integrity check, but the availability of the
> > > + Merkle tree root hash also allows efficiently supporting
> > > + various use cases where normally the whole file would need to
> > > + be hashed at once, such as auditing and authenticity
> > > + verification (appraisal).
> > > +
> > > + If unsure, say N.
> > > +
> > > config EXT4_DEBUG
> > > bool "EXT4 debugging support"
> > > depends on EXT4_FS
> > > diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
> > > index 12f90d48ba613..e5475a629ed80 100644
> > > --- a/fs/ext4/ext4.h
> > > +++ b/fs/ext4/ext4.h
> > > @@ -43,6 +43,9 @@
> > > #define __FS_HAS_ENCRYPTION IS_ENABLED(CONFIG_EXT4_FS_ENCRYPTION)
> > > #include
> > >
> > > +#define __FS_HAS_VERITY IS_ENABLED(CONFIG_EXT4_FS_VERITY)
> > > +#include
> > > +
> > > #include
> > >
> > > /* Until this gets included into linux/compiler-gcc.h */
> > > @@ -405,6 +408,7 @@ struct flex_groups {
> > > #define EXT4_TOPDIR_FL 0x0002 /* Top of directory
> > > hierarchies*/
> > > #define EXT4_HUGE_FILE_FL 0x0004 /* Set to each huge
> > > file */
> > > #define EXT4_EXTENTS_FL 0x0008 /* Inode uses
> > > extents */
> > > +#define EXT4_VERITY_FL 0x0010 /* Verity protected
> > > inode */
> > > #define EXT4_EA_INODE_FL 0x0020 /* Inode used for large EA */
> > > #define EXT4_EOFBLOCKS_FL0x0040 /* Blocks allocated
> > > beyond EOF */
> > > #define EXT4_INLINE_DATA_FL 0x1000 /* Inode has inline
> > > data. */
> > > @@ -472,6 +476,7 @@ enum {
> > > EXT4_INODE_TOPDIR = 17, /* Top of directory hierarchies*/
> > > EXT4_INODE_HUGE_FILE= 18, /* Set to each huge file */
> > > EXT4_INODE_EXTENTS = 19, /* Inode uses extents */
> > > + EXT4_INODE_VERITY = 20, /* Verity
[f2fs-dev] [PATCH] f2fs: stop discard thread during fstrim
Now, discard thread will issue discards in parallel with fstrim,
which will cause problem as below:
i. maybe too many discards commands at some point
ii. disturb the order of block address, which is continuous in fstrim
Signed-off-by: Yunlei He
---
fs/f2fs/f2fs.h| 1 +
fs/f2fs/segment.c | 7 ++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 56204a8..fec596d2 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -330,6 +330,7 @@ struct discard_cmd_control {
atomic_t discard_cmd_cnt; /* # of cached cmd count */
struct rb_root_cached root; /* root of discard rb-tree */
bool rbtree_check; /* config for consistence check
*/
+ atomic_t in_fstrim_cnt; /* # of fstrim in progress */
};
/* for the list of fsync inodes, used only during recovery */
diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index 6edcf83..cea1d27 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -1655,7 +1655,8 @@ static int issue_discard_thread(void *data)
continue;
if (kthread_should_stop())
return 0;
- if (is_sbi_flag_set(sbi, SBI_NEED_FSCK)) {
+ if (is_sbi_flag_set(sbi, SBI_NEED_FSCK)
+ || atomic_read(>in_fstrim_cnt)) {
wait_ms = dpolicy.max_interval;
continue;
}
@@ -1996,6 +1997,7 @@ static int create_discard_cmd_control(struct f2fs_sb_info
*sbi)
atomic_set(>issued_discard, 0);
atomic_set(>issing_discard, 0);
atomic_set(>discard_cmd_cnt, 0);
+ atomic_set(>in_fstrim_cnt, 0);
dcc->nr_discards = 0;
dcc->max_discards = MAIN_SEGS(sbi) << sbi->log_blocks_per_seg;
dcc->undiscard_blks = 0;
@@ -2720,6 +2722,7 @@ int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct
fstrim_range *range)
block_t start_block, end_block;
struct cp_control cpc;
struct discard_policy dpolicy;
+ struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info;
unsigned long long trimmed = 0;
int err = 0;
bool need_align = test_opt(sbi, LFS) && sbi->segs_per_sec > 1;
@@ -2772,11 +2775,13 @@ int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct
fstrim_range *range)
end_block = START_BLOCK(sbi, end_segno + 1);
__init_discard_policy(sbi, , DPOLICY_FSTRIM, cpc.trim_minlen);
+ atomic_inc(>in_fstrim_cnt);
trimmed = __issue_discard_cmd_range(sbi, ,
start_block, end_block);
trimmed += __wait_discard_cmd_range(sbi, ,
start_block, end_block);
+ atomic_dec(>in_fstrim_cnt);
out:
if (!err)
range->len = F2FS_BLK_TO_BYTES(trimmed);
--
1.9.1
___
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
[f2fs-dev] [PATCH] f2fs: move dir data flush to write checkpoint process
This patch move dir data flush to write checkpoint process, by doing this, it may reduce some time for dir fsync. pre: -f2fs_do_sync_file enter -file_write_and_wait_range <- flush & wait -write_checkpoint -do_checkpoint <- wait all -f2fs_do_sync_file exit now: -f2fs_do_sync_file enter -write_checkpoint -block_operations <- flush dir & no wait -do_checkpoint <- wait all -f2fs_do_sync_file exit Signed-off-by: Yunlei He --- fs/f2fs/file.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index 88b1246..9eaf07f 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -216,6 +216,9 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end, trace_f2fs_sync_file_enter(inode); + if (S_ISDIR(inode->i_mode)) + goto go_write; + /* if fdatasync is triggered, let's do in-place-update */ if (datasync || get_dirty_pages(inode) <= SM_I(sbi)->min_fsync_blocks) set_inode_flag(inode, FI_NEED_IPU); -- 1.9.1 ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Re: [f2fs-dev] [PATCH v2 10/12] ext4: add basic fs-verity support
Hi Chandan,
On Fri, Nov 02, 2018 at 03:13:14PM +0530, Chandan Rajendra wrote:
> On Friday, November 2, 2018 4:22:28 AM IST Eric Biggers wrote:
> > From: Eric Biggers
> >
> > Add basic fs-verity support to ext4. fs-verity is a filesystem feature
> > that enables transparent integrity protection and authentication of
> > read-only files. It uses a dm-verity like mechanism at the file level:
> > a Merkle tree is used to verify any block in the file in log(filesize)
> > time. It is implemented mainly by helper functions in fs/verity/.
> > See Documentation/filesystems/fsverity.rst for details.
> >
> > This patch adds everything except the data verification hooks that will
> > needed in ->readpages().
> >
> > On ext4, enabling fs-verity on a file requires that the filesystem has
> > the 'verity' feature, e.g. that it was formatted with
> > 'mkfs.ext4 -O verity' or had 'tune2fs -O verity' run on it.
> > This requires e2fsprogs 1.44.4-2 or later.
> >
> > In ext4, we choose to retain the fs-verity metadata past the end of the
> > file rather than trying to move it into an external inode xattr, since
> > in practice keeping the metadata in-line actually results in the
> > simplest and most efficient implementation. One non-obvious advantage
> > of keeping the verity metadata in-line is that when fs-verity is
> > combined with fscrypt, the verity metadata naturally gets encrypted too;
> > this is actually necessary because it contains hashes of the plaintext.
> >
> > We also choose to keep the on-disk i_size equal to the original file
> > size, in order to make the 'verity' feature a RO_COMPAT feature. Thus,
> > ext4 has to find the fsverity_footer by looking in the last extent.
> >
> > Co-developed-by: Theodore Ts'o
> > Signed-off-by: Theodore Ts'o
> > Signed-off-by: Eric Biggers
> > ---
> > fs/ext4/Kconfig | 20 +++
> > fs/ext4/ext4.h | 20 ++-
> > fs/ext4/file.c | 6
> > fs/ext4/inode.c | 8 +
> > fs/ext4/ioctl.c | 12 +++
> > fs/ext4/super.c | 91 +
> > fs/ext4/sysfs.c | 6
> > 7 files changed, 162 insertions(+), 1 deletion(-)
> >
> > diff --git a/fs/ext4/Kconfig b/fs/ext4/Kconfig
> > index a453cc87082b5..5a76125ac0f8a 100644
> > --- a/fs/ext4/Kconfig
> > +++ b/fs/ext4/Kconfig
> > @@ -111,6 +111,26 @@ config EXT4_FS_ENCRYPTION
> > default y
> > depends on EXT4_ENCRYPTION
> >
> > +config EXT4_FS_VERITY
> > + bool "Ext4 Verity"
> > + depends on EXT4_FS
> > + select FS_VERITY
> > + help
> > + This option enables fs-verity for ext4. fs-verity is the
> > + dm-verity mechanism implemented at the file level. Userspace
> > + can append a Merkle tree (hash tree) to a file, then enable
> > + fs-verity on the file. ext4 will then transparently verify
> > + any data read from the file against the Merkle tree. The file
> > + is also made read-only.
> > +
> > + This serves as an integrity check, but the availability of the
> > + Merkle tree root hash also allows efficiently supporting
> > + various use cases where normally the whole file would need to
> > + be hashed at once, such as auditing and authenticity
> > + verification (appraisal).
> > +
> > + If unsure, say N.
> > +
> > config EXT4_DEBUG
> > bool "EXT4 debugging support"
> > depends on EXT4_FS
> > diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
> > index 12f90d48ba613..e5475a629ed80 100644
> > --- a/fs/ext4/ext4.h
> > +++ b/fs/ext4/ext4.h
> > @@ -43,6 +43,9 @@
> > #define __FS_HAS_ENCRYPTION IS_ENABLED(CONFIG_EXT4_FS_ENCRYPTION)
> > #include
> >
> > +#define __FS_HAS_VERITY IS_ENABLED(CONFIG_EXT4_FS_VERITY)
> > +#include
> > +
> > #include
> >
> > /* Until this gets included into linux/compiler-gcc.h */
> > @@ -405,6 +408,7 @@ struct flex_groups {
> > #define EXT4_TOPDIR_FL 0x0002 /* Top of directory
> > hierarchies*/
> > #define EXT4_HUGE_FILE_FL 0x0004 /* Set to each huge
> > file */
> > #define EXT4_EXTENTS_FL0x0008 /* Inode uses
> > extents */
> > +#define EXT4_VERITY_FL 0x0010 /* Verity protected
> > inode */
> > #define EXT4_EA_INODE_FL 0x0020 /* Inode used for large EA */
> > #define EXT4_EOFBLOCKS_FL 0x0040 /* Blocks allocated beyond
> > EOF */
> > #define EXT4_INLINE_DATA_FL0x1000 /* Inode has inline
> > data. */
> > @@ -472,6 +476,7 @@ enum {
> > EXT4_INODE_TOPDIR = 17, /* Top of directory hierarchies*/
> > EXT4_INODE_HUGE_FILE= 18, /* Set to each huge file */
> > EXT4_INODE_EXTENTS = 19, /* Inode uses extents */
> > + EXT4_INODE_VERITY = 20, /* Verity protected inode */
> > EXT4_INODE_EA_INODE = 21, /* Inode used for large EA */
> > EXT4_INODE_EOFBLOCKS= 22, /* Blocks allocated beyond EOF */
> > EXT4_INODE_INLINE_DATA = 28, /* Data in inode. */
> > @@
Re: [f2fs-dev] [PATCH v2 10/12] ext4: add basic fs-verity support
Hi Andreas, On Mon, Nov 05, 2018 at 02:05:24PM -0700, Andreas Dilger wrote: > On Nov 1, 2018, at 4:52 PM, Eric Biggers wrote: > > > > From: Eric Biggers > > > > Add basic fs-verity support to ext4. fs-verity is a filesystem feature > > that enables transparent integrity protection and authentication of > > read-only files. It uses a dm-verity like mechanism at the file level: > > a Merkle tree is used to verify any block in the file in log(filesize) > > time. It is implemented mainly by helper functions in fs/verity/. > > See Documentation/filesystems/fsverity.rst for details. > > > > This patch adds everything except the data verification hooks that will > > needed in ->readpages(). > > > > On ext4, enabling fs-verity on a file requires that the filesystem has > > the 'verity' feature, e.g. that it was formatted with > > 'mkfs.ext4 -O verity' or had 'tune2fs -O verity' run on it. > > This requires e2fsprogs 1.44.4-2 or later. > > > > In ext4, we choose to retain the fs-verity metadata past the end of the > > file rather than trying to move it into an external inode xattr, since > > in practice keeping the metadata in-line actually results in the > > simplest and most efficient implementation. One non-obvious advantage > > of keeping the verity metadata in-line is that when fs-verity is > > combined with fscrypt, the verity metadata naturally gets encrypted too; > > this is actually necessary because it contains hashes of the plaintext. > > On the plus side, this means that the verity data will automatically be > invalidated if the file is truncated or extended, but on the negative side > it means that the verity Merkle tree needs to be recalculated for the > entire file if e.g. the file is appended to. > > I guess the current implementation will generate the Merkle tree in > userspace, but at some point it might be useful to generate it on-the-fly > to have proper data integrity from the time of write (e.g. like ZFS) > rather than only allowing it to be stored after the entire file is written? > > Storing the Merkle tree in a large xattr inode would allow this to change > in the future rather than being stuck with the current implementation. We > could encrypt the xattr data just as easily as the file data (which should > be done anyway even for non-verity files to avoid leaking data), and having > the verity attr keyed to the inode version/size/mime(?) would ensure the > kernel knows it is stale if the inode is modified. > > I'm not going to stand on my head and block this implementation, I just > thought it is worthwhile to raise these issues now rather than after it > is a fait accompli. > That would actually be the least of the problems for adding write support. Adding write support would require at least: - A way to maintain consistency between the data and hashes, including all levels of hashes, since corruption after a crash (especially of potentially the entire file!) is unacceptable. The main options for solving this are data journalling, copy-on-write, and log-structured volume. But it's very hard to retrofit existing filesystems with new consistency mechanisms. Data journalling can always be used, but is very slow. - An on-disk format that allows dynamically growing/shrinking each level of the Merkle tree; or, using a different authenticated dictionary structure, such as an authenticated skiplist rather than a Merkle tree. This would drastically increase the complexity over a regular Merkle tree. Compare it to dm-verity vs. dm-integrity. dm-verity is read-only and very simple; the kernel just uses a Merkle tree that is generated by userspace. On the other hand, dm-integrity supports writes but is slow, much more complex, and doesn't even actually do full-device authentication since it authenticates each sector independently, i.e. there is no Merkle tree. I don't think it would make sense for the same device-mapper target to support these quite different use cases. And the same general concepts apply at the filesystem level; for these reasons and others (note that per-block checksums like btrfs and ZFS wouldn't need a Merkle tree), write support is very intentionally outside the scope of fs-verity. So I think any arguments for doing things differently in fs-verity need to be made in the context of read-only files. Thanks, Eric ___ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
[f2fs-dev] [PATCH] f2fs: Change to use DEFINE_SHOW_ATTRIBUTE macro
Use DEFINE_SHOW_ATTRIBUTE macro to simplify the code.
Signed-off-by: Yangtao Li
---
fs/f2fs/debug.c | 13 +
1 file changed, 1 insertion(+), 12 deletions(-)
diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c
index 214a968962a1..ef827c0d0379 100644
--- a/fs/f2fs/debug.c
+++ b/fs/f2fs/debug.c
@@ -428,18 +428,7 @@ static int stat_show(struct seq_file *s, void *v)
return 0;
}
-static int stat_open(struct inode *inode, struct file *file)
-{
- return single_open(file, stat_show, inode->i_private);
-}
-
-static const struct file_operations stat_fops = {
- .owner = THIS_MODULE,
- .open = stat_open,
- .read = seq_read,
- .llseek = seq_lseek,
- .release = single_release,
-};
+DEFINE_SHOW_ATTRIBUTE(stat);
int f2fs_build_stats(struct f2fs_sb_info *sbi)
{
--
2.17.0
___
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
