On 7/27/26 9:43 AM, June Park wrote:
Hello! Thank you for the thoughtful comments.
Thank you for your patience. This took me longer to get to than I had hoped.
I have to ask, what is the motivation for this change? Do you have a workload
that shows some kind of improvement from this proposal?
My initial testing in QEMU, storing files from the kernel source tree, showed
noticeable improvements in compression ratio (around 2.4 to 3.5), even though
vdo uses small block sizes. However, I did notice that certain types of data
(such as long synthetic streams of repeating bytes) showed far less improvement
with zstd, at the expense of more CPU cycles. I had concluded that it could
outperform LZ4 in some workloads in terms of compression ratio, for more CPU, a
trade-off that could be made on a case-by-case basis.
If you have data handy from these experiments, it would be interesting
to see it. This is the sort of information that would go well in a cover
letter explaining what you want to do, and why. At any rate, if you have
cases that seem to benefit from this, we can certainly revisit whether
this is worth adding.
Another thing to consider is that we originally chose the LZ4 algorithm
because it is fairly cheap to compute. If you can, it's worth trying to
quantify what the extra computational load does to vdo throughput,
especially with fast storage. The throughput for a vdo volume will often
lag the raw storage speed significantly (due to the deduplication
machinery) and it's worth knowing if changing the algorithm will make
that worse.
The discussion you linked to proposes the ability to swap the algorithm without
reformatting. My current implementation tries to reduce breaking changes as
much as possible, so I had decided on storing the compression algorithm
directly in the volume geometry, instead of for every block. This has the
downside of requiring reformatting, but since the focus of vdo is
deduplication, I don't think the large amounts of additional machinery needed
to support live changes is justified.
I appreciate that you're attempting to minimize disruption. I admit that
it is simpler, in terms of pure implementation, to make this a
format-time choice. However, imagine what happen next: Long-time vdo
users will inquire whether they can use this new feature, and we will
have to tell them no. For new users, I think they may not know all the
data they will store on a volume up front, but they will be locked into
their first choice. Given the case-by-case variability of the tradeoff,
I expect users will appreciate being able to change this setting to fit
their current needs.
In short, doing this as a format-only option looks like implementing
half a feature to me, and I think we would be better off starting with
full flexibility. Also remember that every version of this feature that
we expose to users is a feature we will have to maintain for the
lifetime of the dm-vdo driver, and I would rather not have to support
both versions.
(I believe the difference in complexity is also not that large, but
that's a bit more subjective. Setting the algorithm as a run-time option
means extending the compressed block format, but adding a format-time
option involves more work updating the user space tools, including the
formatter. Both options also require updating the table line and the
super block format, so there's also considerable overlap.)
So anyway. If you can show there is utility in doing this, we can look
at adding it. I will probably want to do it by building on what we did
last year, though. You can look at what I've already done on the branch
feature/allow-compression-configuration in the vdo-devel project. I
haven't rebased the branch in a while, but you can get an idea of how I
was planning the table line and super block changes, at least.
June
Matt
