This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
The following commit(s) were added to refs/heads/master by this push:
new 4130a60 JAMES-3150 Document blob garbage collection (#631)
4130a60 is described below
commit 4130a60e5f429860b8673ebbc351cd6015ebfb5d
Author: Benoit TELLIER <[email protected]>
AuthorDate: Mon Sep 6 11:30:46 2021 +0700
JAMES-3150 Document blob garbage collection (#631)
Co-authored-by: Rene Cordier <[email protected]>
---
.../pages/distributed/architecture/index.adoc | 6 ++--
.../pages/distributed/configure/blobstore.adoc | 15 ++++++--
.../pages/distributed/operate/webadmin.adoc | 39 ++++++++++++++++++++
src/site/markdown/server/manage-webadmin.md | 41 ++++++++++++++++++++++
src/site/xdoc/server/config-blobstore.xml | 14 ++++++--
5 files changed, 106 insertions(+), 9 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/architecture/index.adoc
b/docs/modules/servers/pages/distributed/architecture/index.adoc
index fc0ae77..3eeea53 100644
--- a/docs/modules/servers/pages/distributed/architecture/index.adoc
+++ b/docs/modules/servers/pages/distributed/architecture/index.adoc
@@ -277,11 +277,11 @@ Supported backends include S3 compatible ObjectStorage
(link:https://wiki.openst
Encryption can be configured on top of ObjectStorage.
-Blobs are currently deduplicated in order to reduce storage space. This means
that two blobs with
+Blobs can currently be deduplicated in order to reduce storage space. This
means that two blobs with
the same content will be stored one once.
-The downside is that deletion is more complicated, and a garbage collection
needs to be run. This is a work
-in progress. See
link:https://issues.apache.org/jira/browse/JAMES-3150[JAMES-3150].
+The downside is that deletion is more complicated, and a garbage collection
needs to be run. A first implementation
+based on bloom filters can be used and triggered using the WebAdmin REST API.
=== Task Manager
diff --git a/docs/modules/servers/pages/distributed/configure/blobstore.adoc
b/docs/modules/servers/pages/distributed/configure/blobstore.adoc
index 73f3d48..e9cd4a0 100644
--- a/docs/modules/servers/pages/distributed/configure/blobstore.adoc
+++ b/docs/modules/servers/pages/distributed/configure/blobstore.adoc
@@ -35,11 +35,20 @@ If you choose to enable deduplication, the mails with the
same content will be s
WARNING: Once this feature is enabled, there is no turning back as turning it
off will lead to the deletion of all
the mails sharing the same content once one is deleted.
-This feature also requires a garbage collector mechanism to effectively drop
blobs, which is not implemented yet.
+NOTE: If you are upgrading from James 3.5 or older, the deduplication was
enabled.
-Consequently, all the requested deletions will not be performed, meaning that
blobstore will only grow.
+Deduplication requires a garbage collector mechanism to effectively drop
blobs. A first implementation
+based on bloom filters can be used and triggered using the WebAdmin REST API.
See
+xref:distributed/operate/webadmin.adoc#_running_blob_garbage_collection[Running
blob garbage collection].
-NOTE: If you are upgrading from James 3.5 or older, the deduplication was
enabled.
+In order to avoid concurrency issues upon garbage collection, we slice the
blobs in generation, the two more recent
+generations are not garbage collected.
+
+*deduplication.gc.generation.duration*: Allow controlling the duration of one
generation. Longer implies better deduplication
+but deleted blobs will live longer. Duration, defaults on 30 days, the default
unit is in days.
+
+*deduplication.gc.generation.family*: Every time the duration is changed, this
integer counter must be incremented to avoid
+conflicts. Defaults to 1.
=== Encryption choice
diff --git a/docs/modules/servers/pages/distributed/operate/webadmin.adoc
b/docs/modules/servers/pages/distributed/operate/webadmin.adoc
index 282bf08..ff6b52f 100644
--- a/docs/modules/servers/pages/distributed/operate/webadmin.adoc
+++ b/docs/modules/servers/pages/distributed/operate/webadmin.adoc
@@ -2306,6 +2306,45 @@ the following `additionalInformation`:
}
....
+== Running blob garbage collection
+
+When deduplication is enabled one needs to explicitly run a garbage collection
in order to delete no longer referenced
+blobs.
+
+To do so:
+
+....
+curl -XDELETE http:ip:port/blobs?scope=unreferenced
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning a
task].
+
+Additional parameters include Bloom filter tuning parameters:
+
+ - *associatedProbability*: Allow to define the targeted false positive rate.
Note that subsequent runs do not have the
+same false-positives.
+ - *expectedBlobCount*: Expected count of blobs used to size the bloom filters.
+
+The created task has the following additional information:
+
+....
+{
+ "referenceSourceCount": 3456,
+ "blobCount": 5678,
+ "gcedBlobCount": 1234,
+ "bloomFilterExpectedBlobCount": 10000,
+ "bloomFilterAssociatedProbability": 0.01
+}
+....
+
+Where:
+
+ - *bloomFilterExpectedBlobCount* correspond to the supplied
*expectedBlobCount* query parameter.
+ - *bloomFilterAssociatedProbability* correspond to the supplied
*associatedProbability* query parameter.
+ - *referenceSourceCount* is the count of distinct blob references encountered
while populating the bloom filter.
+ - *blobCount* is the count of blobs tried against the bloom filter. This
value can be used to better size the bloom
+filter in later runs.
+ - *gcedBlobCount* is the count of blobs that were garbage collected.
== Administrating Recipient rewriting
diff --git a/src/site/markdown/server/manage-webadmin.md
b/src/site/markdown/server/manage-webadmin.md
index 571260c..25bd7a6 100644
--- a/src/site/markdown/server/manage-webadmin.md
+++ b/src/site/markdown/server/manage-webadmin.md
@@ -51,6 +51,7 @@ Finally, please note that in case of a malformed URL the 400
bad request respons
- [Sending email over webAdmin](#Sending_email_over_webAdmin)
- [Administrating DLP Configuration](#Administrating_DLP_Configuration)
- [Administrating Sieve quotas](#Administrating_Sieve_quotas)
+ - [Running blob garbage collection](#Running_blob_garbage_collection)
- [Administrating jmap uploads](#Administrating_jmap_uploads)
- [Deleted Messages Vault](#Deleted_Messages_Vault)
- [Task management](#Task_management)
@@ -3517,6 +3518,46 @@ Response codes:
- 204: Operation succeeded
+## Running blob garbage collection
+
+When deduplication is enabled one needs to explicitly run a garbage collection
in order to delete no longer referenced
+blobs.
+
+To do so:
+
+```
+curl -XDELETE http:ip:port/blobs?scope=unreferenced
+```
+
+[More details about endpoints returning a task](#Endpoints_returning_a_task).
+
+Additional parameters include Bloom filter tuning parameters:
+
+ - **associatedProbability**: Allow to define the targeted false positive
rate. Note that subsequent runs do not have the
+same false-positives.
+ - **expectedBlobCount**: Expected count of blobs used to size the bloom
filters.
+
+The created task has the following additional information:
+
+```json
+{
+ "referenceSourceCount": 3456,
+ "blobCount": 5678,
+ "gcedBlobCount": 1234,
+ "bloomFilterExpectedBlobCount": 10000,
+ "bloomFilterAssociatedProbability": 0.01
+}
+```
+
+Where:
+
+ - **bloomFilterExpectedBlobCount** correspond to the supplied
**expectedBlobCount** query parameter.
+ - **bloomFilterAssociatedProbability** correspond to the supplied
**associatedProbability** query parameter.
+ - **referenceSourceCount** is the count of distinct blob references
encountered while populating the bloom filter.
+ - **blobCount** is the count of blobs tried against the bloom filter. This
value can be used to better size the bloom
+filter in later runs.
+ - **gcedBlobCount** is the count of blobs that were garbage collected.
+
## Administrating Jmap Uploads
- [Cleaning upload repository](#Cleaning_upload_repository)
diff --git a/src/site/xdoc/server/config-blobstore.xml
b/src/site/xdoc/server/config-blobstore.xml
index 519701d..97e830e 100644
--- a/src/site/xdoc/server/config-blobstore.xml
+++ b/src/site/xdoc/server/config-blobstore.xml
@@ -53,14 +53,22 @@
The generated startup warning log can be deactivated via
the <code>cassandra.blob.store.disable.startup.warning</code> environment
variable being positioned to <code>false</code>.
</dd>
-
<dt><strong>deduplication/enable</strong></dt>
<dd>Mandatory. Supported value: true and false.</dd>
<dd>If you choose to enable deduplication, the mails with the
same content will be stored only once.</dd>
<dd>Warning: Once this feature is enabled, there is no turning
back as turning it off will lead to the deletion of all</dd>
<dd>the mails sharing the same content once one is
deleted.</dd>
- <dd>This feature also requires a garbage collector mechanism
to effectively drop blobs, which is not implemented yet.</dd>
- <dd>Consequently, all the requested deletions will not be
performed, meaning that blobstore will only grow.</dd>
+ <dd>This feature also requires a garbage collector mechanism
to effectively drop blobs. A first implementation
+ based on bloom filters can be used and triggered using the
WebAdmin REST API. See
+ <a
href="manage-webadmin.html#Running_blob_garbage_collection">Running blob
garbage collection</a>.
+ In order to avoid concurrency issues upon garbage
collection, we slice the blobs in generation, the two more recent
+ generations are not garbage collected.</dd>
+ <dd><strong>deduplication.gc.generation.duration</strong></dd>
+ <dd>Allow controlling the duration of one generation. Longer
implies better deduplication
+ but deleted blobs will live longer. Duration, defaults on
30 days, the default unit is in days.</dd>
+ <dd><strong>deduplication.gc.generation.family</strong></dd>
+ <dd>Every time the duration is changed, this integer counter
must be incremented to avoid
+ conflicts. Defaults to 1.</dd>
<dd>Upgrade note: If you are upgrading from James 3.5 or
older, the deduplication was enabled.</dd>
</dl>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
