chibenwa commented on code in PR #1004: URL: https://github.com/apache/james-project/pull/1004#discussion_r879988100
########## mailet/standard/src/main/java/org/apache/james/transport/mailets/Expires.java: ########## @@ -0,0 +1,144 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.transport.mailets; + +import java.io.StringReader; +import java.time.Duration; +import java.time.ZoneOffset; +import java.time.ZonedDateTime; +import java.time.temporal.ChronoUnit; +import java.util.Optional; +import java.util.function.Supplier; + +import javax.mail.MessagingException; +import javax.mail.internet.MimeMessage; + +import org.apache.james.mime4j.dom.datetime.DateTime; +import org.apache.james.mime4j.field.datetime.parser.DateTimeParser; +import org.apache.james.mime4j.field.datetime.parser.ParseException; +import org.apache.james.util.DurationParser; +import org.apache.mailet.Mail; +import org.apache.mailet.base.DateFormats; +import org.apache.mailet.base.GenericMailet; + + +/** + * <p>Adds an Expires header to the message, or enforces the period of an existing one.</p> + * + * <p>Sample configuration:</p> + * + * <pre><code> + * <mailet match="All" class="Expires"> + * <minAge>1d</minAge> + * <maxAge>1w</maxAge> + * <defaultAge>4w</defaultAge> + * </mailet> + * </code></pre> + * + * @version 1.0.0, 2021-12-14 Review Comment: IMO we can remove the @version orut it to the upcoming 3.8.0 release. Also we should explain that to remove the expired mails one need to call the associated webadmin endpoints. ########## server/apps/distributed-app/docs/modules/ROOT/pages/operate/webadmin.adoc: ########## @@ -1206,6 +1206,51 @@ This task could be run safely online and can be scheduled on a recurring basis outside of peak traffic by an admin to ensure Cassandra message consistency. +=== Deleting old messages of all users + +*Note:* +Consider enabling the xref:configure/vault.adoc[Deleted Messages Vault] +if you use this feature. + +Old messages tend to pile up in user INBOXes. An admin might want to delete +these on behalf of the users, e.g. all messages older than 30 days: +.... +curl -XDELETE http://ip:port/messages?olderThan=30d +.... + +link:#_endpoints_returning_a_task[More details about endpoints returning a task]. + +The `olderThan` parameter should be expressed in the following format: `Nunit`. +`N` should be strictly positive. `unit` could be either in the short form +(`d`, `w`, `y` etc.), or in the long form (`days`, `weeks`, `months`, `years`). +The default unit is `days`. + +Response codes: + +* 201: Success. Corresponding task id is returned. +* 400: Error in the request. Details can be found in the reported error. + +To delete old mails from a different mailbox than INBOX, e.g. a mailbox +named "Archived" : +.... +curl -XDELETE http://ip:port/messages?mailbox=Archived&olderThan=30d +.... + +Since this is a somewhat expensive operation, the task is throttled to one user +per second. You may speed it up via `usersPerSecond=10` for example. But keep +in mind that a high rate might overwhelm your database or blob store. + +*Cassandra only:* Review Comment: ```suggestion *Scanning search only:* ``` ########## server/apps/distributed-app/docs/modules/ROOT/pages/operate/webadmin.adoc: ########## @@ -1206,6 +1206,51 @@ This task could be run safely online and can be scheduled on a recurring basis outside of peak traffic by an admin to ensure Cassandra message consistency. +=== Deleting old messages of all users Review Comment: Generaly we also document tasks details here in webadmin documentation ########## server/protocols/webadmin/webadmin-mailbox/src/main/java/org/apache/james/webadmin/service/ExpireMailboxService.java: ########## @@ -0,0 +1,249 @@ +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ + +package org.apache.james.webadmin.service; + +import java.time.Duration; +import java.time.temporal.ChronoUnit; +import java.util.Date; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicLong; + +import javax.inject.Inject; + +import org.apache.james.core.Username; +import org.apache.james.mailbox.MailboxManager; +import org.apache.james.mailbox.MailboxSession; +import org.apache.james.mailbox.MessageManager; +import org.apache.james.mailbox.MessageUid; +import org.apache.james.mailbox.exception.MailboxException; +import org.apache.james.mailbox.exception.MailboxNotFoundException; +import org.apache.james.mailbox.model.MailboxConstants; +import org.apache.james.mailbox.model.MailboxPath; +import org.apache.james.mailbox.model.SearchQuery; +import org.apache.james.mailbox.model.SearchQuery.DateResolution; +import org.apache.james.task.Task; +import org.apache.james.task.Task.Result; +import org.apache.james.user.api.UsersRepository; +import org.apache.james.user.api.UsersRepositoryException; +import org.apache.james.util.DurationParser; +import org.apache.james.util.ReactorUtils; +import org.apache.james.util.streams.Iterators; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.annotation.JsonCreator; +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonProperty; +import com.github.fge.lambdas.Throwing; +import com.google.common.base.Preconditions; + +import reactor.core.publisher.Flux; +import reactor.core.publisher.Mono; +import reactor.core.scheduler.Schedulers; + + +public class ExpireMailboxService { + + public static class RunningOptions { + public static final RunningOptions DEFAULT = new RunningOptions(1, MailboxConstants.INBOX, true, Optional.empty()); + + public static RunningOptions fromParams( + Optional<String> byExpiresHeader, Optional<String> olderThan, + Optional<String> usersPerSecond, Optional<String> mailbox) { + try { + if (byExpiresHeader.isPresent() == olderThan.isPresent()) { + throw new IllegalArgumentException("Must specify either 'olderThan' or 'byExpiresHeader' parameter"); + } + return new RunningOptions( + usersPerSecond.map(Integer::parseInt).orElse(DEFAULT.getUsersPerSecond()), + mailbox.orElse(DEFAULT.getMailbox()), byExpiresHeader.isPresent(), olderThan); + } catch (NumberFormatException ex) { + throw new IllegalArgumentException("'usersPerSecond' must be numeric"); + } + } + + private final int usersPerSecond; + + private final String mailbox; + + private final boolean byExpiresHeader; + + private final Optional<String> olderThan; + + @JsonIgnore + private final Optional<Duration> maxAgeDuration; + + @JsonCreator + public RunningOptions(@JsonProperty("usersPerSecond") int usersPerSecond, + @JsonProperty("mailbox") String mailbox, + @JsonProperty("byExpiresHeader") boolean byExpiresHeader, + @JsonProperty("olderThan") Optional<String> olderThan) { + Preconditions.checkArgument(usersPerSecond > 0, "'usersPerSecond' must be strictly positive"); + this.usersPerSecond = usersPerSecond; + this.mailbox = mailbox; + this.byExpiresHeader = byExpiresHeader; + this.olderThan = olderThan; + this.maxAgeDuration = olderThan.map(v -> DurationParser.parse(olderThan.get(), ChronoUnit.DAYS)); + } + + public int getUsersPerSecond() { + return usersPerSecond; + } + + public String getMailbox() { + return mailbox; + } + + public boolean getByExpiresHeader() { + return byExpiresHeader; + } + + public Optional<String> getOlderThan() { + return olderThan; + } + } + + public static class Context { + private final AtomicLong inboxesExpired; + private final AtomicLong inboxesFailed; + private final AtomicLong inboxesProcessed; + private final AtomicLong messagesDeleted; + + public Context() { + this.inboxesExpired = new AtomicLong(0L); + this.inboxesFailed = new AtomicLong(0L); + this.inboxesProcessed = new AtomicLong(0L); + this.messagesDeleted = new AtomicLong(0L); + } + + public long getInboxesExpired() { + return inboxesExpired.get(); + } + + public long getInboxesFailed() { + return inboxesFailed.get(); + } + + public long getInboxesProcessed() { + return inboxesProcessed.get(); + } + + public long getMessagesDeleted() { + return messagesDeleted.get(); + } + + public void incrementExpiredCount() { + inboxesExpired.incrementAndGet(); + } + + public void incrementFailedCount() { + inboxesFailed.incrementAndGet(); + } + + public void incrementProcessedCount() { + inboxesProcessed.incrementAndGet(); + } + + public void incrementMessagesDeleted(long count) { + messagesDeleted.addAndGet(count); + } + } + + private static final Logger LOGGER = LoggerFactory.getLogger(ExpireMailboxService.class); + + private final UsersRepository usersRepository; + private final MailboxManager mailboxManager; + + @Inject + public ExpireMailboxService(UsersRepository usersRepository, MailboxManager mailboxManager) { + this.usersRepository = usersRepository; + this.mailboxManager = mailboxManager; + } + + public Mono<Result> expireMailboxes(Context context, RunningOptions runningOptions, Date now) { + try { + SearchQuery expiration = SearchQuery.of( + runningOptions.maxAgeDuration.map(maxAge -> { + Date limit = Date.from(now.toInstant().minus(maxAge)); + return SearchQuery.internalDateBefore(limit, DateResolution.Second); + }) + .orElse( + SearchQuery.headerDateBefore("Expires", now, DateResolution.Second) + ) + ); + return Iterators.toFlux(usersRepository.list()) + .transform(ReactorUtils.<Username, Task.Result>throttle() + .elements(runningOptions.getUsersPerSecond()) + .per(Duration.ofSeconds(1)) + .forOperation(username -> + expireUserMailbox(context, username, runningOptions.getMailbox(), expiration))) + .reduce(Task.Result.COMPLETED, Task::combine); + } catch (UsersRepositoryException e) { + LOGGER.error("Error while accessing users from repository", e); + return Mono.just(Task.Result.PARTIAL); + } + } + + private Mono<Result> expireUserMailbox(Context context, Username username, String mailbox, SearchQuery expiration) { + MailboxSession session = mailboxManager.createSystemSession(username); + MailboxPath mailboxPath = MailboxPath.forUser(username, mailbox); + return Mono.from(mailboxManager.getMailboxReactive(mailboxPath, session)) + // newly created users do not have mailboxes yet, just skip them + .onErrorResume(MailboxNotFoundException.class, ignore -> Mono.empty()) + .flatMap(mgr -> searchMessagesReactive(mgr, session, expiration) + .flatMap(list -> deleteMessagesReactive(mgr, session, list))) + .doOnNext(expired -> { + if (expired > 0) { + context.incrementExpiredCount(); + context.incrementMessagesDeleted(expired); + } + context.incrementProcessedCount(); + }) + .then(Mono.just(Task.Result.COMPLETED)) + .onErrorResume(e -> { + LOGGER.warn("Failed to expire user mailbox {}", username, e); + context.incrementFailedCount(); + context.incrementProcessedCount(); + return Mono.just(Task.Result.PARTIAL); + }); + } + + private Mono<List<MessageUid>> searchMessagesReactive(MessageManager mgr, MailboxSession session, SearchQuery expiration) { + try { + return Flux.from(mgr.search(expiration, session)).collectList(); + } catch (MailboxException e) { + return Mono.error(e); + } + } + + private Mono<Integer> deleteMessagesReactive(MessageManager mgr, MailboxSession session, List<MessageUid> uids) { + if (uids.isEmpty()) { + return Mono.just(0); + } else { + return Mono.fromSupplier( Review Comment: Using fromCallable should allow getting read of the Throwing...sneakyThrow boiler plate ########## server/apps/distributed-app/docs/modules/ROOT/partials/Expires.adoc: ########## @@ -0,0 +1,28 @@ +=== Expires + +Sanitizes or adds an expiration date to a message, in the form of an `Expires` +header (RFC 4021). By itself this header is informational only, but may be +useful in combination with the Web Admin operation to +xref:operate/webadmin.adoc#_administrating_messages[delete old messages]. + +The mailet can force an existing expiration date to be within the bounds +given by `minAge`, `maxAge`, or both. `minAge` specifies the minimum time +the date must lie in the future, while `maxAge` specifies a maximum. + +If a message has no expiration date, the mailet can add one according to +the optional `defaultAge` parameter. + +All parameter values should be expressed in the following format: `Nunit`. +`N` should be positive. `unit` could be either in the short form +(`h`, `d`, `w`, `y` etc.), or in the long form (`hours`, days`, `weeks`, +`months`, `years`). The default unit is `days`. + +Sample configuration: + +.... +<mailet match="All" class="Expires"> + <minAge>12h</minAge> + <defaultAge>7d</defaultAge> + <maxAge>8w</maxAge> +</mailet> +.... Review Comment: Point to the route here too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
