On Mon, Sep 11, 2023 at 06:38:36PM +0200, mwi...@suse.com wrote:
> From: Martin Wilck
>
> Since "libmultipath: keep bindings in memory", we don't re-read the
> bindings file after every modification. Add a notification mechanism
> that makes multipathd aware of changes to the bindings file.
From: Matthew Sakai
Add various support utilities for the vdo target and deduplication
index, including logging utilities, string and time management, and
index-specific error codes.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
From: Matthew Sakai
MurmurHash3 is a fast, non-cryptographic, 128-bit hash. It was
originally written by Austin Appleby and placed in the public
domain. This version has been modified to produce the same result on
both big endian and little endian processors, making it suitable for
use in
Hi,
This patchset reflects a baseline of the VDO DM target. It is still
very much under active review and development, and will likley
continue to be until the end of this year. So while the code isn't
yet ready for upstream inclusion it will evolve so that one day it
hopefully is, and moving
From: Matthew Sakai
Add data and methods for marshalling and unmarshalling the persistent
metadata.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
Signed-off-by: Michael Sclafani
Co-developed-by: Sweet Tea Dorminy
Signed-off-by: Sweet
From: Matthew Sakai
Add data and methods to report statisics.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
Signed-off-by: Michael Sclafani
Co-developed-by: Sweet Tea Dorminy
Signed-off-by: Sweet Tea Dorminy
Signed-off-by: Matthew
From: Matthew Sakai
Add the data and methods that manage the dm-vdo target itself. This
includes the overall state of the target and its threads, the state of
the logical volumes, startup, shutdown, and statistics.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
From: Matthew Sakai
The dm-vdo target provides inline deduplication, compression,
zero-block elimination, and thin provisioning. By layering it with
other device mapper targets, it can add these features to any storage
stack. It can also provide a common deduplication pool for groups of
targets.
From: Matthew Sakai
Add support for dumping detailed vdo state to the kernel log via a
dmsetup message. The dump code is not thread-safe and is generally
intended for use only when a vdo is hung.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael
From: Matthew Sakai
This adds the admin-guide documentation for dm-vdo.
vdo.rst is the guide to using dm-vdo. vdo-design.rst is an overview of
the design of dm-vdo.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Signed-off-by: Matthew Sakai
Signed-off-by: Mike Snitzer
From: Matthew Sakai
Add data and methods setting run time parameters via sysfs, and to
make state and statistics information available through sysfs.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
Signed-off-by: Michael Sclafani
From: Matthew Sakai
This includes crash recovery and read-only rebuild of damaged vdos
volumes.
When a vdo is restarted after a crash, it will automatically attempt
to recover from its journals.
If a vdo encounters an unrecoverable error, it will enter read-only
mode. This mode indicates that
From: Matthew Sakai
When blocks do not deduplicate, vdo will attempt to compress them. Up
to 14 compressed blocks may be packed into a single data block (this
limitation is imposed by the block map). The packer implements a
simple best-fit packing algorithm and also manages the formatting and
From: Matthew Sakai
The set of leaf pages of the block map tree is too large to fit in
memory, so each block map zone maintains a cache of leaf pages. This
commit adds the implementation of that cache.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael
From: Matthew Sakai
The recovery journal is used to amortize updates across the block map
and slab depot. Each write request causes an entry to be made in the
journal. Entries are either "data remappings" or "block map
remappings." For a data remapping, the journal records the logical
address
From: Matthew Sakai
The dm-vdo target is not supported for 32-bit configurations. A vdo
target typically requires 1 to 1.5 GB of memory at any given time,
which is likely a large fraction of the addressable memory of a 32-bit
system. At the same time, the amount of addressable storage attached
From: Matthew Sakai
Add two hash maps, one keyed by integers, the other by pointers, and
also a priority heap. The integer map is used for locking of logical
and physical addresses. The pointer map is used for managing
concurrent writes of the same data, ensuring that those writes are
From: Matthew Sakai
Add definitions of constants defining the fixed parameters of a VDO
volume, and the default and maximum values of configurable or dynamic
parameters.
Add definitions of internal status codes used for internal
communication within the module and for logging.
Add definitions
From: Matthew Sakai
Add standardized allocation macros and memory tracking tools to track
and report any allocated memory that is not freed. This makes it
easier to ensure that the vdo target does not leak memory.
Also adds utilities for controlling whether certain threads are
allowed to
From: Matthew Sakai
The top-level deduplication index brings all the earlier components
together. The top-level index creates the separate zone structures
that enable the index to handle several requests in parallel, handles
dispatching requests to the right zones and components, and
coordinates
From: Matthew Sakai
The volume store structures manage the reading and writing of chapter
pages. When a chapter is closed, it is packed into a read-only
structure, split across several pages, and written to storage.
The volume store also contains a cache and specialized queues that
sort and
From: Matthew Sakai
Add infrastructure for managing reads and writes to the underlying
storage layer for the deduplication index. The deduplication index
uses dm-bufio for all of its reads and writes, so part of this
infrastructure is managing the various dm-bufio clients required. It
also adds
From: Matthew Sakai
Deduplication records are stored in groups called chapters. New
records are collected in a structure called the open chapter, which is
optimized for adding, removing, and sorting records.
When a chapter fills, it is packed into a read-only structure called a
closed chapter,
From: Matthew Sakai
The volume index is a large delta index that maps each record name to
the chapter which contains the newest record for that name. The volume
index can contain several million records and is stored entirely in
memory while the index is operating, accounting for the majority of
From: Matthew Sakai
The delta index is a space and memory efficient alternative to a
hashtable. Instead of storing the entire key for each entry, the
entries are sorted by key and only the difference between adjacent
keys (the delta) is stored. If the keys are evenly distributed, the
size of the
From: Matthew Sakai
Adds funnel_queue, a mostly lock-free multi-producer, single-consumer
queue. It also adds the request queue used by the dm-vdo deduplication
index, and the work_queue used by the dm-vdo data store. Both of these
are built on top of funnel queue and are intended to support the
From: Matthew Sakai
The deduplication index interface for index clients includes the
deduplication request and index session structures. This is the
interface that the rest of the vdo target uses to make requests,
receive responses, and collect statistics.
Also adds sysfs nodes for inspecting
From: Matthew Sakai
Add structures which record the configuration of various deduplication
index parameters. This also includes facilities for saving and loading
the configuration and validating its integrity.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by:
From: Matthew Sakai
Adds utilities for managing and using named threads, as well as
several locking and synchronization utilities. These utilities help
dm-vdo minimize thread transitions and manage interactions between
threads.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
From: Matthew Sakai
The slab depot maintains an additional small data structure, the "slab
summary," which is used to reduce the amount of work needed to come
back online after a crash. The slab summary maintains an entry for
each slab indicating whether or not the slab has ever been used,
From: Matthew Sakai
Add the data and methods that implement the slab_depot that manages
the allocation of slabs of blocks added by the preceding commits.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
Signed-off-by: Michael Sclafani
From: Matthew Sakai
Most of the vdo volume belongs to the slab depot. The depot contains a
collection of slabs. The slabs can be up to 32GB, and are divided into
three sections. Most of a slab consists of a linear sequence of 4K
blocks. These blocks are used either to store data, or to hold
From: Matthew Sakai
Each slab is independent of every other. They are assigned to
"physical zones" in round-robin fashion. If there are P physical
zones, then slab n is assigned to zone n mod P. The set of slabs in
each physical zone is managed by a block allocator.
Co-developed-by: J. corwin
From: Matthew Sakai
In order to deduplicate concurrent writes of the same data (to
different locations), data_vios which are writing the same data are
grouped together in a "hash lock," named for and keyed by the hash of
the data being written. Each hash lock is assigned to a hash zone
based on
From: Matthew Sakai
Add support for handling incoming flush and/or FUA bios. Each such bio
is assigned to a struct vdo_flush. These are allocated as needed, but
there is always one kept in reserve in case allocations fail. In the
event of an allocation failure, bios may need to wait for an
From: Matthew Sakai
Add the data and methods that implement the data_vio object that
handles user data bios as they are processed.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
Signed-off-by: Michael Sclafani
Co-developed-by: Sweet Tea
From: Matthew Sakai
The block map contains the logical to physical mapping. It can be
thought of as an array with one entry per logical address. Each entry
is 5 bytes: 36 bits contain the physical block number which holds the
data for the given logical address, and the remaining 4 bits are used
From: Matthew Sakai
Add the admin_state structures which are used to track the states of
individual vdo components for handling of operations like suspend and
resume. It also adds the action manager which is used to schedule and
manage cross-thread administrative and internal operations.
From: Matthew Sakai
Add the data and methods that implement the vio object that is the
basic unit of I/O in vdo.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
Signed-off-by: Michael Sclafani
Co-developed-by: Sweet Tea Dorminy
From: Matthew Sakai
The io_submitter handles bio submission from vdo data store to the
storage below. It will merge bios when possible.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
Signed-off-by: Michael Sclafani
Co-developed-by: Sweet
From: Matthew Sakai
Add the data and methods that manage queries to the deduplication
index and the responses from the index.
Co-developed-by: J. corwin Coburn
Signed-off-by: J. corwin Coburn
Co-developed-by: Michael Sclafani
Signed-off-by: Michael Sclafani
Co-developed-by: Sweet Tea
On Thu, Sep 14, 2023 at 04:51:29PM +0200, mwi...@suse.com wrote:
> From: Martin Wilck
>
> As the assignment of free aliases now relies on the bindings being
> properly sorted, add some unit tests to make sure the sorting algorithm
> works.
>
Reviewed-by: Benjamin Marzinski
> Signed-off-by:
On Thu, Sep 14, 2023 at 04:51:31PM +0200, mwi...@suse.com wrote:
> From: Martin Wilck
>
> The use of uatomic_xchg() in alias.c causes a -Warray-bounds error
> on distributions using gcc 12, such as Fedora 37. This is a similar
> error to 2534c4f ("libmultipath: avoid -Warray-bounds error with
From: Martin Wilck
As the assignment of free aliases now relies on the bindings being
properly sorted, add some unit tests to make sure the sorting algorithm
works.
Signed-off-by: Martin Wilck
---
tests/alias.c | 212 +-
1 file changed, 209
From: Martin Wilck
This patch set contains a two-step rework of the user-friendly
names code. Patch 2-5 change the current code such that it (well, almost)
never attempts to use an alias that is currently in use by another
map. We already have some checks for this, but they don't cover all
From: Martin Wilck
The use of uatomic_xchg() in alias.c causes a -Warray-bounds error
on distributions using gcc 12, such as Fedora 37. This is a similar
error to 2534c4f ("libmultipath: avoid -Warray-bounds error with gcc
12 and musl libc"). This happens only with liburcu 0.13 and earlier,
and
tree:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git
for-next
head: edcabec81f12dfc9b86b05084bba3232efdc5e4e
commit: 59d814674dd66fc64ba229605174bfe0b8e566d5 [2/4] dm persistent data:
Introduce extent allocator
config: x86_64-rhel-8.3-rust
On Thu, Sep 14 2023 at 10:54P -0400,
kernel test robot wrote:
> tree:
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git
> for-next
> head: edcabec81f12dfc9b86b05084bba3232efdc5e4e
> commit: 59d814674dd66fc64ba229605174bfe0b8e566d5 [2/4] dm persistent data:
>
tree:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git
for-next
head: 805d736ee48c10d8770f998127887795ffb0106f
commit: 064fc5e0e09b49033693b07003c142d0be27a009 [2/4] dm persistent data:
Introduce extent allocator
config: riscv-defconfig
49 matches
Mail list logo
