[jira] [Created] (JAMES-2560) ICS sanitizing
Tellier Benoit created JAMES-2560: - Summary: ICS sanitizing Key: JAMES-2560 URL: https://issues.apache.org/jira/browse/JAMES-2560 Project: James Server Issue Type: Improvement Components: Mailet Contributions Reporter: Tellier Benoit Some ICS sender sends text/calendar as headers of an empty body part, causing the ICS to be ignored by the ICS pipeline. We should sanitize such ICS in order to make them available as regular ICS attachment and processed by the ICS workflow. This can, for instance, be done by a specific mailet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[08/10] james-project git commit: JAMES-2525 adds createContainer operation to ObjectStorageBlobsDAO
JAMES-2525 adds createContainer operation to ObjectStorageBlobsDAO Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/a2c7e989 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/a2c7e989 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/a2c7e989 Branch: refs/heads/master Commit: a2c7e989f662c70eb5d0f46031eba2e2433ed381 Parents: 7c57ae1 Author: Jean Helou Authored: Tue Sep 11 13:19:38 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:27:00 2018 +0700 -- .../objectstorage/ObjectStorageBlobsDAO.java| 13 .../ObjectStorageBlobsDAOContract.java | 9 ++ .../ObjectStorageBlobsDAOTest.java | 34 3 files changed, 42 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/a2c7e989/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java index 176e5a4..1e0d12a 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAO.java @@ -33,6 +33,7 @@ import org.apache.james.blob.objectstorage.swift.SwiftKeystone3ObjectStorage; import org.apache.james.blob.objectstorage.swift.SwiftTempAuthObjectStorage; import org.jclouds.blobstore.domain.Blob; import org.jclouds.blobstore.options.CopyOptions; +import org.jclouds.domain.Location; import com.github.fge.lambdas.Throwing; import com.google.common.base.Preconditions; @@ -41,6 +42,7 @@ import com.google.common.hash.HashingInputStream; public class ObjectStorageBlobsDAO implements BlobStore { private static final InputStream EMPTY_STREAM = new ByteArrayInputStream(new byte[0]); +public static final Location DEFAULT_LOCATION = null; private final BlobId.Factory blobIdFactory; @@ -67,6 +69,17 @@ public class ObjectStorageBlobsDAO implements BlobStore { return SwiftKeystone3ObjectStorage.daoBuilder(testConfig); } +public CompletableFuture createContainer(ContainerName name) { +return CompletableFuture.supplyAsync(() -> blobStore.createContainerInLocation(DEFAULT_LOCATION, +name.value())).thenApply(created -> { +if (created) { +return name; +} else { +throw new ObjectStoreException("Unable to create container " + name.value()); +} +}); +} + @Override public CompletableFuture save(byte[] data) { return save(new ByteArrayInputStream(data)); http://git-wip-us.apache.org/repos/asf/james-project/blob/a2c7e989/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAOContract.java -- diff --git a/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAOContract.java b/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAOContract.java index a6cb4b4..36501f3 100644 --- a/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAOContract.java +++ b/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/ObjectStorageBlobsDAOContract.java @@ -29,22 +29,17 @@ import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; - import org.apache.james.blob.api.BlobId; -import org.jclouds.blobstore.BlobStore; -import org.jclouds.domain.Location; -public interface ObjectStorageBlobsDAOContract { -Location DEFAULT_LOCATION = null; +public interface ObjectStorageBlobsDAOContract { ContainerName containerName(); default void assertBlobsDAOCanStoreAndRetrieve(ObjectStorageBlobsDAOBuilder builder) throws InterruptedException, ExecutionException, TimeoutException { -BlobStore blobStore = builder.getSupplier().get(); -blobStore.createContainerInLocation(DEFAULT_LOCATION, containerName().value()); ObjectStorageBlobsDAO dao = builder.build(); +dao.createContainer(containerName()); byte[] bytes = "content".getBytes(StandardCharsets.UTF_8); CompletableFuture save = dao.save(bytes); InputStream inputStream = save.thenApply(dao::read
[06/10] james-project git commit: JAMES-2525 moves common test components to guice-common
JAMES-2525 moves common test components to guice-common Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/5dff052d Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/5dff052d Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/5dff052d Branch: refs/heads/master Commit: 5dff052d10b0dbaa689e60614c5d5d54259ae51f Parents: 8ecd8bb Author: Jean Helou Authored: Mon Sep 10 14:53:55 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:26:15 2018 +0700 -- .../james/AggregateGuiceModuleTestRule.java | 82 .../org/apache/james/GuiceModuleTestRule.java | 30 --- .../apache/james/TempFilesystemTestRule.java| 54 - .../james/AggregateGuiceModuleTestRule.java | 82 .../org/apache/james/GuiceModuleTestRule.java | 30 +++ .../apache/james/TempFilesystemTestRule.java| 54 + 6 files changed, 166 insertions(+), 166 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/5dff052d/server/container/guice/cassandra-guice/src/test/java/org/apache/james/AggregateGuiceModuleTestRule.java -- diff --git a/server/container/guice/cassandra-guice/src/test/java/org/apache/james/AggregateGuiceModuleTestRule.java b/server/container/guice/cassandra-guice/src/test/java/org/apache/james/AggregateGuiceModuleTestRule.java deleted file mode 100644 index b3c4f71..000 --- a/server/container/guice/cassandra-guice/src/test/java/org/apache/james/AggregateGuiceModuleTestRule.java +++ /dev/null @@ -1,82 +0,0 @@ -/ - * Licensed to the Apache Software Foundation (ASF) under one * - * or more contributor license agreements. See the NOTICE file * - * distributed with this work for additional information* - * regarding copyright ownership. The ASF licenses this file * - * to you under the Apache License, Version 2.0 (the* - * "License"); you may not use this file except in compliance * - * with the License. You may obtain a copy of the License at * - * * - * http://www.apache.org/licenses/LICENSE-2.0 * - * * - * Unless required by applicable law or agreed to in writing, * - * software distributed under the License is distributed on an * - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * - * KIND, either express or implied. See the License for the* - * specific language governing permissions and limitations * - * under the License. * - / - -package org.apache.james; - -import java.util.List; - -import org.junit.rules.RuleChain; -import org.junit.runner.Description; -import org.junit.runners.model.Statement; - -import com.github.steveash.guavate.Guavate; -import com.google.common.collect.ImmutableList; -import com.google.inject.Module; -import com.google.inject.util.Modules; - - -public class AggregateGuiceModuleTestRule implements GuiceModuleTestRule { - -public static AggregateGuiceModuleTestRule of(GuiceModuleTestRule... subrule) { -return new AggregateGuiceModuleTestRule(ImmutableList.copyOf(subrule)); -} - -private final List subrule; -private final RuleChain chain; - -private AggregateGuiceModuleTestRule(List subrule) { -this.subrule = subrule; -this.chain = subrule -.stream() -.reduce(RuleChain.emptyRuleChain(), -RuleChain::around, -RuleChain::around); -} - -public AggregateGuiceModuleTestRule aggregate(GuiceModuleTestRule... subrule) { -List guiceModules = ImmutableList.builder() -.addAll(this.subrule) -.add(subrule) -.build(); - -return new AggregateGuiceModuleTestRule(guiceModules); -} - -@Override -public Module getModule() { -List modules = subrule -.stream() -.map(GuiceModuleTestRule::getModule) -.collect(Guavate.toImmutableList()); - -return Modules.combine(modules); -} - -@Override -public Statement apply(Statement base, Description description) { -return chain.apply(base, description); -} - -@Override -public void await() { -subrule -.parallelStream() -.forEach(GuiceModuleTestRule::await); -} -} http://git-wip-us.apache.org/repos/asf/james-project/blob/5dff052d/server/container/guice/cassan
[07/10] james-project git commit: JAMES-2525 enforce singleton swift container for test
JAMES-2525 enforce singleton swift container for test Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/7c57ae17 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/7c57ae17 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/7c57ae17 Branch: refs/heads/master Commit: 7c57ae178a75ccbb63e12df59ed431f31ee644d6 Parents: 5dff052 Author: Jean Helou Authored: Mon Sep 10 16:08:07 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:26:49 2018 +0700 -- .../objectstorage/DockerSwiftContainer.java | 134 +++ .../objectstorage/DockerSwiftExtension.java | 52 +-- .../blob/objectstorage/DockerSwiftRule.java | 49 +++ .../objectstorage/DockerSwiftSingleton.java | 31 + 4 files changed, 217 insertions(+), 49 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/7c57ae17/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/DockerSwiftContainer.java -- diff --git a/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/DockerSwiftContainer.java b/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/DockerSwiftContainer.java new file mode 100644 index 000..b073eed --- /dev/null +++ b/server/blob/blob-objectstorage/src/test/java/org/apache/james/blob/objectstorage/DockerSwiftContainer.java @@ -0,0 +1,134 @@ +/ + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information* + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the* + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the* + * specific language governing permissions and limitations * + * under the License. * + / + +package org.apache.james.blob.objectstorage; + +import static org.testcontainers.containers.wait.strategy.Wait.forHttp; + +import java.net.URI; + +import org.apache.james.util.Host; +import org.apache.james.util.docker.RateLimiters; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.testcontainers.containers.Container; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.output.OutputFrame; +import org.testcontainers.containers.wait.strategy.WaitAllStrategy; + +import com.github.fge.lambdas.Throwing; + +public class DockerSwiftContainer { +private static final Logger LOGGER = LoggerFactory.getLogger(DockerSwiftContainer.class); +private static final String SWIFT_DOCKER_IMAGE = "jeantil/openstack-keystone-swift:pike"; +private static final int SWIFT_PORT = 8080; +private static final int KEYSTONE_ADMIN_PORT = 35357; +private final GenericContainer swiftContainer; +private DockerSwift dockerSwift; + +public DockerSwiftContainer() { +this.swiftContainer = new GenericContainer<>(SWIFT_DOCKER_IMAGE) +.withExposedPorts(KEYSTONE_ADMIN_PORT) +.withExposedPorts(SWIFT_PORT) +.withLogConsumer(DockerSwiftContainer::displayDockerLog) +.waitingFor( +new WaitAllStrategy() +.withStrategy( +forHttp("/v3") +.forPort(KEYSTONE_ADMIN_PORT) +.forStatusCode(200) +.withRateLimiter(RateLimiters.DEFAULT) +).withStrategy( +forHttp("/info") +.forPort(SWIFT_PORT) +.forStatusCode(200) +.withRateLimiter(RateLimiters.DEFAULT) +) +); + +} + +public void start() { +swiftContainer.start(); +Integer swiftPort = swiftContainer.getMappedPort(SWIFT_PORT); +String containerIpAddress = swiftContainer.getContainerIpAddress(); +Container
[04/10] james-project git commit: JAMES-2525 adds blob-objectstorage-guice module to maven
JAMES-2525 adds blob-objectstorage-guice module to maven Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/7bb6e802 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/7bb6e802 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/7bb6e802 Branch: refs/heads/master Commit: 7bb6e80223d8ab504cd1c5360e4876baf99dc816 Parents: 53eeb3f Author: Jean Helou Authored: Wed Sep 5 11:59:28 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:26:14 2018 +0700 -- pom.xml | 22 +++ .../guice/blob-objectstorage-guice/pom.xml | 62 server/container/guice/pom.xml | 1 + 3 files changed, 85 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/7bb6e802/pom.xml -- diff --git a/pom.xml b/pom.xml index 0b43776..c93c920 100644 --- a/pom.xml +++ b/pom.xml @@ -1061,6 +1061,28 @@ ${james.groupId} +blob-objectstorage +${project.version} + + +${james.groupId} +blob-objectstorage +${project.version} +test-jar + + +${james.groupId} +blob-objectstorage-guice +${project.version} + + +${james.groupId} +blob-objectstorage-guice +${project.version} +test-jar + + +${james.groupId} blob-cassandra ${project.version} http://git-wip-us.apache.org/repos/asf/james-project/blob/7bb6e802/server/container/guice/blob-objectstorage-guice/pom.xml -- diff --git a/server/container/guice/blob-objectstorage-guice/pom.xml b/server/container/guice/blob-objectstorage-guice/pom.xml new file mode 100644 index 000..24e96d4 --- /dev/null +++ b/server/container/guice/blob-objectstorage-guice/pom.xml @@ -0,0 +1,62 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> + +4.0.0 + + +org.apache.james +james-server-guice +3.2.0-SNAPSHOT +../pom.xml + + +blob-objectstorage-guice +jar + +Apache James :: Server :: Blob Store in object storage - guice injection +An advanced email server - Object storage based Blob Store bindings + + + +${james.groupId} +blob-objectstorage + + +${james.groupId} +james-server-guice-common + + +org.assertj +assertj-core +test + + +org.junit.jupiter +junit-jupiter-engine +test + + +org.junit.platform +junit-platform-launcher +test + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/james-project/blob/7bb6e802/server/container/guice/pom.xml -- diff --git a/server/container/guice/pom.xml b/server/container/guice/pom.xml index 73a6a93..e57a43f 100644 --- a/server/container/guice/pom.xml +++ b/server/container/guice/pom.xml @@ -34,6 +34,7 @@ blob-api-guice +blob-objectstorage-guice cassandra-guice cassandra-ldap-guice configuration - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Commented] (JAMES-2525) Implement swift Object Store
[ https://issues.apache.org/jira/browse/JAMES-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645866#comment-16645866 ] Tellier Benoit commented on JAMES-2525: --- https://github.com/linagora/james-project/pull/1782 Provides Guice bindings for object stores > Implement swift Object Store > > > Key: JAMES-2525 > URL: https://issues.apache.org/jira/browse/JAMES-2525 > Project: James Server > Issue Type: New Feature > Components: James Core >Reporter: Jean Helou >Priority: Major > Fix For: 3.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[09/10] james-project git commit: JAMES-2525 creates guice rule and cucumber singleton for swift docker container
JAMES-2525 creates guice rule and cucumber singleton for swift docker container Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/7554b2e0 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/7554b2e0 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/7554b2e0 Branch: refs/heads/master Commit: 7554b2e0caaed7f3cfed15a49a2e9b6ee52fe3a0 Parents: a2c7e98 Author: Jean Helou Authored: Tue Sep 11 13:20:53 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:27:08 2018 +0700 -- .../guice/blob-objectstorage-guice/pom.xml | 18 .../guice/DockerSwiftTestRule.java | 87 .../cassandra-jmap-integration-testing/pom.xml | 17 .../cucumber/CucumberSwiftSingleton.java| 27 ++ 4 files changed, 149 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/7554b2e0/server/container/guice/blob-objectstorage-guice/pom.xml -- diff --git a/server/container/guice/blob-objectstorage-guice/pom.xml b/server/container/guice/blob-objectstorage-guice/pom.xml index 24e96d4..20e74f0 100644 --- a/server/container/guice/blob-objectstorage-guice/pom.xml +++ b/server/container/guice/blob-objectstorage-guice/pom.xml @@ -41,9 +41,22 @@ ${james.groupId} +blob-objectstorage +${project.version} +test-jar +test + + +${james.groupId} james-server-guice-common +${james.groupId} +james-server-guice-common +test-jar +test + + org.assertj assertj-core test @@ -58,5 +71,10 @@ junit-platform-launcher test + +org.testcontainers +testcontainers +test + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/james-project/blob/7554b2e0/server/container/guice/blob-objectstorage-guice/src/test/java/org/apache/james/modules/objectstorage/guice/DockerSwiftTestRule.java -- diff --git a/server/container/guice/blob-objectstorage-guice/src/test/java/org/apache/james/modules/objectstorage/guice/DockerSwiftTestRule.java b/server/container/guice/blob-objectstorage-guice/src/test/java/org/apache/james/modules/objectstorage/guice/DockerSwiftTestRule.java new file mode 100644 index 000..ac5c459 --- /dev/null +++ b/server/container/guice/blob-objectstorage-guice/src/test/java/org/apache/james/modules/objectstorage/guice/DockerSwiftTestRule.java @@ -0,0 +1,87 @@ +/ + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information* + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the* + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the* + * specific language governing permissions and limitations * + * under the License. * + / + +package org.apache.james.modules.objectstorage.guice; + +import java.util.UUID; + +import org.apache.james.GuiceModuleTestRule; +import org.apache.james.blob.api.BlobStore; +import org.apache.james.blob.api.HashBlobId; +import org.apache.james.blob.objectstorage.ContainerName; +import org.apache.james.blob.objectstorage.ObjectStorageBlobsDAO; +import org.apache.james.blob.objectstorage.swift.Credentials; +import org.apache.james.blob.objectstorage.swift.SwiftKeystone2ObjectStorage; +import org.apache.james.blob.objectstorage.swift.TenantName; +import org.apache.james.blob.objectstorage.swift.UserName; +import org.junit.runner.Description; +import org.junit.runners.model.Statement; +import org.testcontainers.containers.GenericContainer; + +import com.github.fge.lambdas.Throwing; +import com.google.inject.Mod
[03/10] james-project git commit: JAMES-2525 Provider and bindings for ObjectStorageDAO
JAMES-2525 Provider and bindings for ObjectStorageDAO Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/8ecd8bbf Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/8ecd8bbf Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/8ecd8bbf Branch: refs/heads/master Commit: 8ecd8bbf967970bd0c19b3eeea61283d5ea008a3 Parents: a4872d6 Author: Jean Helou Authored: Wed Sep 5 15:34:36 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:26:14 2018 +0700 -- .../swift/SwiftKeystone2ObjectStorage.java | 2 + .../swift/SwiftKeystone3ObjectStorage.java | 2 + .../swift/SwiftTempAuthObjectStorage.java | 2 + .../ObjectStorageBlobStoreModule.java | 35 ++ .../ObjectStorageBlobsDAOProvider.java | 116 +++ 5 files changed, 157 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/8ecd8bbf/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java index bf75bcd..fcb1833 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java @@ -38,6 +38,8 @@ import com.google.common.collect.ImmutableSet; import com.google.inject.Module; public class SwiftKeystone2ObjectStorage { +public static final String AUTH_API_NAME = "keystone2"; + private static final Iterable JCLOUDS_MODULES = ImmutableSet.of(new SLF4JLoggingModule()); http://git-wip-us.apache.org/repos/asf/james-project/blob/8ecd8bbf/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java index c0dcee3..13350ea 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java @@ -40,6 +40,8 @@ import com.google.common.collect.ImmutableSet; import com.google.inject.Module; public class SwiftKeystone3ObjectStorage { +public static final String AUTH_API_NAME = "keystone2"; + private static final Iterable JCLOUDS_MODULES = ImmutableSet.of(new SLF4JLoggingModule()); http://git-wip-us.apache.org/repos/asf/james-project/blob/8ecd8bbf/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java index b5b1c22..631d63d 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java @@ -39,6 +39,8 @@ import com.google.common.collect.ImmutableSet; import com.google.inject.Module; public class SwiftTempAuthObjectStorage { +public static final String AUTH_API_NAME = "tmpauth"; + private static final Iterable JCLOUDS_MODULES = ImmutableSet.of(new SLF4JLoggingModule()); http://git-wip-us.apache.org/repos/asf/james-project/blob/8ecd8bbf/server/container/guice/blob-objectstorage-guice/src/main/java/org/apache/james/modules/objectstorage/ObjectStorageBlobStoreModule.java -- diff --git a/server/container/guice/blob-objectstorage-guice/src/main/java/org/apache/james/modules/objectstorage/ObjectStorageBlobStoreModule.java b/server/container/guice/blob-objectstorage-guice/src/main/java/org/apache/james/modules/objectstorage/ObjectStorageBlobStoreModule.java new file mode 100644 index 000..85ceb
[05/10] james-project git commit: JAMES-2525 extracts common blob store guice module
JAMES-2525 extracts common blob store guice module Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/53eeb3fb Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/53eeb3fb Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/53eeb3fb Branch: refs/heads/master Commit: 53eeb3fbb0f4c5a392b03d6c1a9f3e8d1c8c26d2 Parents: 27ef335 Author: Jean Helou Authored: Tue Sep 4 10:25:21 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:26:14 2018 +0700 -- pom.xml | 11 server/container/guice/blob-api-guice/pom.xml | 58 .../modules/mailbox/BlobStoreAPIModule.java | 34 server/container/guice/cassandra-guice/pom.xml | 5 ++ .../apache/james/CassandraJamesServerMain.java | 2 + .../mailbox/CassandraObjectStoreModule.java | 4 -- server/container/guice/pom.xml | 1 + 7 files changed, 111 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/53eeb3fb/pom.xml -- diff --git a/pom.xml b/pom.xml index 6a3dda1..0b43776 100644 --- a/pom.xml +++ b/pom.xml @@ -1050,6 +1050,17 @@ ${james.groupId} +blob-api-guice +${project.version} + + +${james.groupId} +blob-api-guice +${project.version} +test-jar + + +${james.groupId} blob-cassandra ${project.version} http://git-wip-us.apache.org/repos/asf/james-project/blob/53eeb3fb/server/container/guice/blob-api-guice/pom.xml -- diff --git a/server/container/guice/blob-api-guice/pom.xml b/server/container/guice/blob-api-guice/pom.xml new file mode 100644 index 000..41f5881 --- /dev/null +++ b/server/container/guice/blob-api-guice/pom.xml @@ -0,0 +1,58 @@ + + +http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> + +4.0.0 + + +org.apache.james +james-server-guice +3.2.0-SNAPSHOT +../pom.xml + + +blob-api-guice +jar + +Apache James :: Server :: Blob Store API - guice injection +An advanced email server - Common object storage bindings + + + +${james.groupId} +blob-api + + +${james.groupId} +james-server-guice-common + + + + + +org.apache.maven.plugins +maven-surefire-plugin + +true + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/james-project/blob/53eeb3fb/server/container/guice/blob-api-guice/src/main/java/org/apache/james/modules/mailbox/BlobStoreAPIModule.java -- diff --git a/server/container/guice/blob-api-guice/src/main/java/org/apache/james/modules/mailbox/BlobStoreAPIModule.java b/server/container/guice/blob-api-guice/src/main/java/org/apache/james/modules/mailbox/BlobStoreAPIModule.java new file mode 100644 index 000..4887488 --- /dev/null +++ b/server/container/guice/blob-api-guice/src/main/java/org/apache/james/modules/mailbox/BlobStoreAPIModule.java @@ -0,0 +1,34 @@ +/ + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information* + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the* + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the* + * specific language governing permissions and limitations * + * under the License.
[10/10] james-project git commit: JAMES-2558 Upgrade to Apache Camel 2.22.1
JAMES-2558 Upgrade to Apache Camel 2.22.1 Fixes CVE-2018-8041 Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/8f5df497 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/8f5df497 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/8f5df497 Branch: refs/heads/master Commit: 8f5df4978450810394c2d420cd33a04262379920 Parents: 7554b2e Author: Benoit Tellier Authored: Wed Oct 10 15:31:22 2018 +0700 Committer: Benoit Tellier Committed: Thu Oct 11 09:27:29 2018 +0700 -- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/8f5df497/pom.xml -- diff --git a/pom.xml b/pom.xml index c93c920..80d1b71 100644 --- a/pom.xml +++ b/pom.xml @@ -588,7 +588,7 @@ 5.15.5 0.8.2 3.0.0 -2.22.0 +2.22.1 10.14.2.0 1.1.1 0.94.27 - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[01/10] james-project git commit: JAMES-2525 fixes broken equality introduced in previous commit
Repository: james-project Updated Branches: refs/heads/master 34f102990 -> 8f5df4978 JAMES-2525 fixes broken equality introduced in previous commit Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/27ef3356 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/27ef3356 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/27ef3356 Branch: refs/heads/master Commit: 27ef335611e3d162755f4c5ce7237ab3d30a9158 Parents: 34f1029 Author: Jean Helou Authored: Fri Oct 5 17:02:42 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:25:13 2018 +0700 -- .../blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java | 2 +- .../james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/27ef3356/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java index 2e1986d..f5e7272 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java @@ -209,7 +209,7 @@ public class SwiftKeystone3ObjectStorage { @Override public boolean equals(Object o) { -if (o instanceof SwiftKeystone2ObjectStorage.Configuration) { +if (o instanceof SwiftKeystone3ObjectStorage.Configuration) { Configuration that = (Configuration) o; return Objects.equal(endpoint, that.endpoint) && Objects.equal(identity, that.identity) && http://git-wip-us.apache.org/repos/asf/james-project/blob/27ef3356/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java index 48eeeb2..e8f1129 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java @@ -195,7 +195,7 @@ public class SwiftTempAuthObjectStorage { @Override public boolean equals(Object o) { -if (o instanceof SwiftKeystone2ObjectStorage.Configuration) { +if (o instanceof SwiftTempAuthObjectStorage.Configuration) { Configuration that = (Configuration) o; return Objects.equal(endpoint, that.endpoint) && Objects.equal(identity, that.identity) && - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[02/10] james-project git commit: JAMES-2525 implements swift auth protocol config readers
JAMES-2525 implements swift auth protocol config readers Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/a4872d69 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/a4872d69 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/a4872d69 Branch: refs/heads/master Commit: a4872d69a9d62aad11fb3331e7daa192536ebd04 Parents: 7bb6e80 Author: Jean Helou Authored: Wed Sep 5 12:00:01 2018 +0200 Committer: Benoit Tellier Committed: Thu Oct 11 09:26:14 2018 +0700 -- .../swift/SwiftKeystone2ObjectStorage.java | 4 +- .../swift/SwiftKeystone3ObjectStorage.java | 14 +- .../swift/SwiftTempAuthObjectStorage.java | 15 ++ .../objectstorage/SwiftConfigurationReader.java | 26 ++ .../SwiftKeystone2ConfigurationReader.java | 74 ++ .../SwiftKeystone3ConfigurationReader.java | 134 ++ .../SwiftTmpAuthConfigurationReader.java| 90 +++ .../SwiftKeystone2ConfigurationReaderTest.java | 150 +++ .../SwiftKeystone3ConfigurationReaderTest.java | 254 +++ .../SwiftTmpAuthConfigurationReaderTest.java| 183 + 10 files changed, 940 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/a4872d69/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java index 5ace285..bf75bcd 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone2ObjectStorage.java @@ -114,8 +114,8 @@ public class SwiftKeystone2ObjectStorage { return this; } -public Builder region(Region region) { -this.region = Optional.of(region); +public Builder region(Optional region) { +this.region = region; return this; } http://git-wip-us.apache.org/repos/asf/james-project/blob/a4872d69/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java index f5e7272..c0dcee3 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftKeystone3ObjectStorage.java @@ -110,8 +110,13 @@ public class SwiftKeystone3ObjectStorage { return this; } -public Builder region(Region region) { -this.region = Optional.of(region); +public Builder region(Optional region) { +this.region = region; +return this; +} + +public Builder domainId(Optional domainId) { +this.domainId = domainId; return this; } @@ -120,6 +125,11 @@ public class SwiftKeystone3ObjectStorage { return this; } +public Builder project(Optional project) { +this.project = project; +return this; +} + public Builder project(Project project) { this.project = Optional.of(project); return this; http://git-wip-us.apache.org/repos/asf/james-project/blob/a4872d69/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java -- diff --git a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java index e8f1129..b5b1c22 100644 --- a/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/objectstorage/swift/SwiftTempAuthObjectStorage.java +++ b/server/blob/blob-objectstorage/src/main/java/org/apache/james/blob/
[jira] [Created] (JAMES-2559) Branch merge should be on a specific original branch
Antoine Duprat created JAMES-2559: - Summary: Branch merge should be on a specific original branch Key: JAMES-2559 URL: https://issues.apache.org/jira/browse/JAMES-2559 Project: James Server Issue Type: New Feature Components: docker Reporter: Antoine Duprat Actually, the branch merging tool is merging one branch into `master`. In order to be more flexible, we have to introduce a new parameter and merge the branch into the given original branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[13/14] james-project git commit: JAMES-2554 Add test on Cassandra MailRepository storing implementation The aim of those tests is to prouve the order of storing data: MimeMessage -> message blob part
JAMES-2554 Add test on Cassandra MailRepository storing implementation The aim of those tests is to prouve the order of storing data: MimeMessage -> message blob parts -> keys -> mail repository size Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/e536e62f Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/e536e62f Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/e536e62f Branch: refs/heads/master Commit: e536e62fabe75a5463f345a4b72db5f8dcadc162 Parents: 33bfd0d Author: Antoine Duprat Authored: Mon Oct 8 15:20:36 2018 +0200 Committer: Benoit Tellier Committed: Wed Oct 10 15:47:38 2018 +0700 -- ...ilRepositoryWithFakeImplementationsTest.java | 291 +++ 1 file changed, 291 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/e536e62f/server/mailrepository/mailrepository-cassandra/src/test/java/org/apache/james/mailrepository/cassandra/CassandraMailRepositoryWithFakeImplementationsTest.java -- diff --git a/server/mailrepository/mailrepository-cassandra/src/test/java/org/apache/james/mailrepository/cassandra/CassandraMailRepositoryWithFakeImplementationsTest.java b/server/mailrepository/mailrepository-cassandra/src/test/java/org/apache/james/mailrepository/cassandra/CassandraMailRepositoryWithFakeImplementationsTest.java new file mode 100644 index 000..75aa16a --- /dev/null +++ b/server/mailrepository/mailrepository-cassandra/src/test/java/org/apache/james/mailrepository/cassandra/CassandraMailRepositoryWithFakeImplementationsTest.java @@ -0,0 +1,291 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.james.mailrepository.cassandra; + +import com.datastax.driver.core.ResultSet; +import com.datastax.driver.core.Session; +import com.google.common.collect.ImmutableList; +import org.apache.james.backends.cassandra.CassandraCluster; +import org.apache.james.backends.cassandra.CassandraClusterExtension; +import org.apache.james.backends.cassandra.components.CassandraModule; +import org.apache.james.backends.cassandra.init.CassandraTypesProvider; +import org.apache.james.backends.cassandra.utils.CassandraUtils; +import org.apache.james.blob.api.BlobId; +import org.apache.james.blob.api.HashBlobId; +import org.apache.james.blob.api.Store; +import org.apache.james.blob.cassandra.BlobTable; +import org.apache.james.blob.cassandra.CassandraBlobModule; +import org.apache.james.blob.cassandra.CassandraBlobsDAO; +import org.apache.james.blob.mail.MimeMessagePartsId; +import org.apache.james.blob.mail.MimeMessageStore; +import org.apache.james.core.MailAddress; +import org.apache.james.core.builder.MimeMessageBuilder; +import org.apache.james.mailrepository.api.MailKey; +import org.apache.james.mailrepository.api.MailRepositoryUrl; +import org.apache.james.server.core.MailImpl; +import org.apache.mailet.Mail; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.api.extension.ExtensionContext; + +import javax.mail.MessagingException; +import javax.mail.internet.MimeMessage; +import java.util.List; +import java.util.concurrent.CompletableFuture; + +import static com.datastax.driver.core.querybuilder.QueryBuilder.select; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +@ExtendWith(CassandraMailRepositoryWithFakeImplementationsTest.MailRepositoryCassandraClusterExtension.class) +class CassandraMailRepositoryWithFakeImplementationsTest { +static final MailRepositoryUrl URL = MailRepositoryUrl.from("proto://url"); +static final HashBlobId.Factory BLOB_ID_FACTORY = new HashBlobId.Factory(); + +static class MailRepositoryCassandraClusterExtension extends CassandraClusterExtension { +public MailRepositoryCassandraClusterExtension() { +
[04/14] james-project git commit: JAMES-2556 Fix reprocessing when several repositories with same path
JAMES-2556 Fix reprocessing when several repositories with same path We need to aggregate repositories to check for mail existence before reprocessing Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/9d52d569 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/9d52d569 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/9d52d569 Branch: refs/heads/master Commit: 9d52d569f749a28163719ea0ee709a55153ebbc4 Parents: f15655b Author: Benoit Tellier Authored: Wed Oct 10 10:43:41 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:19 2018 +0700 -- .../james/webadmin/service/ReprocessingService.java | 15 +-- .../webadmin/routes/MailRepositoriesRoutesTest.java | 2 -- 2 files changed, 9 insertions(+), 8 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/9d52d569/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java b/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java index c316b95..6ccb045 100644 --- a/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java +++ b/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java @@ -25,6 +25,7 @@ import java.util.function.Consumer; import javax.inject.Inject; import javax.mail.MessagingException; +import org.apache.commons.lang3.tuple.Pair; import org.apache.james.mailrepository.api.MailKey; import org.apache.james.mailrepository.api.MailRepository; import org.apache.james.mailrepository.api.MailRepositoryPath; @@ -90,13 +91,15 @@ public class ReprocessingService { public void reprocess(MailRepositoryPath path, MailKey key, Optional targetProcessor, String targetQueue) throws MailRepositoryStore.MailRepositoryStoreException, MessagingException { Reprocessor reprocessor = new Reprocessor(getMailQueue(targetQueue), targetProcessor); -mailRepositoryStoreService +Pair mailPair = mailRepositoryStoreService .getRepositories(path) -.forEach(Throwing.consumer((MailRepository repository) -> -reprocessor.reprocess(repository, -Optional.ofNullable(repository.retrieve(key)) -.orElseThrow(() -> new MissingKeyException(key -.sneakyThrow()); +.map(Throwing.function(repository -> Pair.of(repository, Optional.ofNullable(repository.retrieve(key) +.filter(pair -> pair.getRight().isPresent()) +.map(pair -> Pair.of(pair.getLeft(), pair.getRight().get())) +.findFirst() +.orElseThrow(() -> new MissingKeyException(key)); + +reprocessor.reprocess(mailPair.getKey(), mailPair.getValue()); } private MailQueue getMailQueue(String targetQueue) { http://git-wip-us.apache.org/repos/asf/james-project/blob/9d52d569/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java index ea78e4c..7d98451 100644 --- a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java +++ b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java @@ -83,7 +83,6 @@ import org.apache.mailet.base.test.FakeMail; import org.eclipse.jetty.http.HttpStatus; import org.junit.After; import org.junit.Before; -import org.junit.Ignore; import org.junit.Test; import com.google.common.collect.Sets; @@ -1506,7 +1505,6 @@ public class MailRepositoriesRoutesTest { .body("completedDate", is(notNullValue())); } -@Ignore @Test public void reprocessingOneTaskShouldNotFailWhenSeveralRepositoryWithSamePath() throws Exception { MailRepository mailRepository = mailRepositoryStore.create(URL_MY_REPO); - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additi
[jira] [Commented] (JAMES-2554) Coherence issues between mail repository list and mails
[ https://issues.apache.org/jira/browse/JAMES-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644660#comment-16644660 ] Tellier Benoit commented on JAMES-2554: --- https://github.com/linagora/james-project/pull/1786 contributed some consistency related tests for CassandraRepositories > Coherence issues between mail repository list and mails > --- > > Key: JAMES-2554 > URL: https://issues.apache.org/jira/browse/JAMES-2554 > Project: James Server > Issue Type: New Feature > Components: cassandra, MailStore & MailRepository >Reporter: Antoine Duprat >Priority: Major > > When getting the list of mails from a repository, some of them does not exist. > I don't know what happened with them, but GETting them fails with not found. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-2556) Impossible to reprocess a mailqueue when a key is not found
[ https://issues.apache.org/jira/browse/JAMES-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tellier Benoit closed JAMES-2556. - > Impossible to reprocess a mailqueue when a key is not found > --- > > Key: JAMES-2556 > URL: https://issues.apache.org/jira/browse/JAMES-2556 > Project: James Server > Issue Type: Bug > Components: MailStore & MailRepository, webadmin >Affects Versions: master, 3.1.0 >Reporter: Tellier Benoit >Priority: Minor > Fix For: 3.2.0 > > > We miss a null check on mailRepository.retrieve(key) > This can happen for instance in the face of concurrent deletes > {code:java} > 11:56:43.579 [ERROR] o.a.j.t.MemoryTaskManager - Error while running task > java.lang.NullPointerException: null > at > org.apache.james.webadmin.service.ReprocessingService.reprocess(ReprocessingService.java:72) > at > org.apache.james.webadmin.service.ReprocessingService.lambda$null$0(ReprocessingService.java:59) > at > com.github.fge.lambdas.consumers.ConsumerChainer.doAccept(ConsumerChainer.java:20) > at > com.github.fge.lambdas.consumers.ThrowingConsumer.accept(ThrowingConsumer.java:22) > at > java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) > at > java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373) > at java.util.Iterator.forEachRemaining(Iterator.java:116) > at > java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) > at > java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) > at > java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) > at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at > java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) > at > org.apache.james.webadmin.service.ReprocessingService.lambda$reprocessAll$1(ReprocessingService.java:59) > at > com.github.fge.lambdas.consumers.ConsumerChainer.lambda$sneakyThrow$9(ConsumerChainer.java:73) > at > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > at > java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) > at > org.apache.james.webadmin.service.ReprocessingService.reprocessAll(ReprocessingService.java:56) > at > org.apache.james.webadmin.service.ReprocessingAllMailsTask.run(ReprocessingAllMailsTask.java:99) > at > org.apache.james.task.MemoryTaskManager.run(MemoryTaskManager.java:84) > at > org.apache.james.task.MemoryTaskManager.lambda$runWithMdc$2(MemoryTaskManager.java:77) > at org.apache.james.util.MDCBuilder.lambda$withMdc$0(MDCBuilder.java:60) > at org.apache.james.util.MDCBuilder.withMdc(MDCBuilder.java:48) > at org.apache.james.util.MDCBuilder.withMdc(MDCBuilder.java:59) > at > org.apache.james.task.MemoryTaskManager.runWithMdc(MemoryTaskManager.java:72) > at > org.apache.james.task.MemoryTaskManager.lambda$submit$1(MemoryTaskManager.java:67) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Commented] (JAMES-2545) Implement a HealthCheck for RabbitMQ component
[ https://issues.apache.org/jira/browse/JAMES-2545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644658#comment-16644658 ] Tellier Benoit commented on JAMES-2545: --- https://github.com/linagora/james-project/pull/1783 stabilises some RabbitMQ tests > Implement a HealthCheck for RabbitMQ component > -- > > Key: JAMES-2545 > URL: https://issues.apache.org/jira/browse/JAMES-2545 > Project: James Server > Issue Type: Improvement > Components: Queue >Reporter: Tellier Benoit >Priority: Major > Fix For: 3.2.0 > > > This health check will simply try to open a connection to rabbitMQ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Resolved] (JAMES-2556) Impossible to reprocess a mailqueue when a key is not found
[ https://issues.apache.org/jira/browse/JAMES-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tellier Benoit resolved JAMES-2556. --- Resolution: Fixed https://github.com/linagora/james-project/pull/1791 solved this > Impossible to reprocess a mailqueue when a key is not found > --- > > Key: JAMES-2556 > URL: https://issues.apache.org/jira/browse/JAMES-2556 > Project: James Server > Issue Type: Bug > Components: MailStore & MailRepository, webadmin >Affects Versions: master, 3.1.0 >Reporter: Tellier Benoit >Priority: Minor > Fix For: 3.2.0 > > > We miss a null check on mailRepository.retrieve(key) > This can happen for instance in the face of concurrent deletes > {code:java} > 11:56:43.579 [ERROR] o.a.j.t.MemoryTaskManager - Error while running task > java.lang.NullPointerException: null > at > org.apache.james.webadmin.service.ReprocessingService.reprocess(ReprocessingService.java:72) > at > org.apache.james.webadmin.service.ReprocessingService.lambda$null$0(ReprocessingService.java:59) > at > com.github.fge.lambdas.consumers.ConsumerChainer.doAccept(ConsumerChainer.java:20) > at > com.github.fge.lambdas.consumers.ThrowingConsumer.accept(ThrowingConsumer.java:22) > at > java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) > at > java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373) > at java.util.Iterator.forEachRemaining(Iterator.java:116) > at > java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) > at > java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) > at > java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) > at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at > java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) > at > org.apache.james.webadmin.service.ReprocessingService.lambda$reprocessAll$1(ReprocessingService.java:59) > at > com.github.fge.lambdas.consumers.ConsumerChainer.lambda$sneakyThrow$9(ConsumerChainer.java:73) > at > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > at > java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) > at > org.apache.james.webadmin.service.ReprocessingService.reprocessAll(ReprocessingService.java:56) > at > org.apache.james.webadmin.service.ReprocessingAllMailsTask.run(ReprocessingAllMailsTask.java:99) > at > org.apache.james.task.MemoryTaskManager.run(MemoryTaskManager.java:84) > at > org.apache.james.task.MemoryTaskManager.lambda$runWithMdc$2(MemoryTaskManager.java:77) > at org.apache.james.util.MDCBuilder.lambda$withMdc$0(MDCBuilder.java:60) > at org.apache.james.util.MDCBuilder.withMdc(MDCBuilder.java:48) > at org.apache.james.util.MDCBuilder.withMdc(MDCBuilder.java:59) > at > org.apache.james.task.MemoryTaskManager.runWithMdc(MemoryTaskManager.java:72) > at > org.apache.james.task.MemoryTaskManager.lambda$submit$1(MemoryTaskManager.java:67) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Closed] (JAMES-2557) NULL_SENDER throws an error in Dlp
[ https://issues.apache.org/jira/browse/JAMES-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tellier Benoit closed JAMES-2557. - > NULL_SENDER throws an error in Dlp > -- > > Key: JAMES-2557 > URL: https://issues.apache.org/jira/browse/JAMES-2557 > Project: James Server > Issue Type: Bug > Components: Mailet Contributions >Affects Versions: master, 3.1.0 >Reporter: Tellier Benoit >Priority: Major > Fix For: 3.2.0 > > > Watching Kibana logs I got this recurring error: > {code:java} > Exception calling dlp.Dlp: NULL sender '<>' do not have domain part > java.lang.IllegalStateException: NULL sender '<>' do not have domain part > at org.apache.james.core.MailAddress$1.getDomain(MailAddress.java:87) > at org.apache.james.transport.matchers.dlp.Dlp.matchingRule(Dlp.java:76) > at > org.apache.james.transport.matchers.dlp.Dlp.lambda$findFirstMatchingRule$0(Dlp.java:72) > at java.util.Optional.flatMap(Optional.java:241) > at > org.apache.james.transport.matchers.dlp.Dlp.findFirstMatchingRule(Dlp.java:72) > at org.apache.james.transport.matchers.dlp.Dlp.match(Dlp.java:55) > at > org.apache.james.mailetcontainer.impl.camel.MatcherSplitter.split(MatcherSplitter.java:109) > at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:481) > at > org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:300) > at > org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:273) > at > org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:187) > at > org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) > at > org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:70) > at > org.apache.camel.language.bean.BeanExpression.invokeBean(BeanExpression.java:200) > at > org.apache.camel.language.bean.BeanExpression.evaluate(BeanExpression.java:124) > at > org.apache.camel.language.bean.BeanExpression.evaluate(BeanExpression.java:135) > at > org.apache.camel.processor.Splitter.createProcessorExchangePairs(Splitter.java:127) > at > org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:241) > at org.apache.camel.processor.Splitter.process(Splitter.java:122) > at > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) > at > org.apache.camel.component.direct.DirectBlockingProducer.process(DirectBlockingProducer.java:53) > at > org.apache.camel.processor.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:186) > at > org.apache.camel.processor.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:86) > at > org.apache.camel.impl.ProducerCache$1.doInProducer(ProducerCache.java:541) > at > org.apache.camel.impl.ProducerCache$1.doInProducer(ProducerCache.java:506) > at > org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:369) > at > org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:506) > at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:229) > at > org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:144) > at > org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:161) > at > org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:168) > at > org.apache.james.mailetcontainer.impl.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68) > at > org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:84) > at > org.apache.james.mailetcontainer.impl.JamesMailSpooler.lambda$run$0(JamesMailSpooler.java:163) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > It cames from the Dlp part (Dlp.java): > {code:java} > pri
[jira] [Resolved] (JAMES-2557) NULL_SENDER throws an error in Dlp
[ https://issues.apache.org/jira/browse/JAMES-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tellier Benoit resolved JAMES-2557. --- Resolution: Fixed https://github.com/linagora/james-project/pull/1789 fixed this > NULL_SENDER throws an error in Dlp > -- > > Key: JAMES-2557 > URL: https://issues.apache.org/jira/browse/JAMES-2557 > Project: James Server > Issue Type: Bug > Components: Mailet Contributions >Affects Versions: master, 3.1.0 >Reporter: Tellier Benoit >Priority: Major > Fix For: 3.2.0 > > > Watching Kibana logs I got this recurring error: > {code:java} > Exception calling dlp.Dlp: NULL sender '<>' do not have domain part > java.lang.IllegalStateException: NULL sender '<>' do not have domain part > at org.apache.james.core.MailAddress$1.getDomain(MailAddress.java:87) > at org.apache.james.transport.matchers.dlp.Dlp.matchingRule(Dlp.java:76) > at > org.apache.james.transport.matchers.dlp.Dlp.lambda$findFirstMatchingRule$0(Dlp.java:72) > at java.util.Optional.flatMap(Optional.java:241) > at > org.apache.james.transport.matchers.dlp.Dlp.findFirstMatchingRule(Dlp.java:72) > at org.apache.james.transport.matchers.dlp.Dlp.match(Dlp.java:55) > at > org.apache.james.mailetcontainer.impl.camel.MatcherSplitter.split(MatcherSplitter.java:109) > at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:481) > at > org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:300) > at > org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:273) > at > org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:187) > at > org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) > at > org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:70) > at > org.apache.camel.language.bean.BeanExpression.invokeBean(BeanExpression.java:200) > at > org.apache.camel.language.bean.BeanExpression.evaluate(BeanExpression.java:124) > at > org.apache.camel.language.bean.BeanExpression.evaluate(BeanExpression.java:135) > at > org.apache.camel.processor.Splitter.createProcessorExchangePairs(Splitter.java:127) > at > org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:241) > at org.apache.camel.processor.Splitter.process(Splitter.java:122) > at > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) > at > org.apache.camel.component.direct.DirectBlockingProducer.process(DirectBlockingProducer.java:53) > at > org.apache.camel.processor.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:186) > at > org.apache.camel.processor.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:86) > at > org.apache.camel.impl.ProducerCache$1.doInProducer(ProducerCache.java:541) > at > org.apache.camel.impl.ProducerCache$1.doInProducer(ProducerCache.java:506) > at > org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:369) > at > org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:506) > at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:229) > at > org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:144) > at > org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:161) > at > org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:168) > at > org.apache.james.mailetcontainer.impl.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68) > at > org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:84) > at > org.apache.james.mailetcontainer.impl.JamesMailSpooler.lambda$run$0(JamesMailSpooler.java:163) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(
[jira] [Commented] (JAMES-2544) Manageable RabbitMQ: browse
[ https://issues.apache.org/jira/browse/JAMES-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644649#comment-16644649 ] Tellier Benoit commented on JAMES-2544: --- https://github.com/linagora/james-project/pull/1781 Added a Retrier on top of management API > Manageable RabbitMQ: browse > --- > > Key: JAMES-2544 > URL: https://issues.apache.org/jira/browse/JAMES-2544 > Project: James Server > Issue Type: Improvement > Components: Queue, rabbitmq >Reporter: Trần Tiến Đức >Priority: Major > > retrieve last message timestamp from Cassandra > read messages from cassandra with slide queries > filter messages the same way as Dequeue (noop for now) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[14/14] james-project git commit: JAMES-2554 Use logback in tests
JAMES-2554 Use logback in tests Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/34f10299 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/34f10299 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/34f10299 Branch: refs/heads/master Commit: 34f10299003853fc30a87812ba693fcc0a3c6234 Parents: e536e62 Author: Antoine Duprat Authored: Tue Oct 9 09:50:47 2018 +0200 Committer: Benoit Tellier Committed: Wed Oct 10 15:47:38 2018 +0700 -- .../mailrepository-cassandra/pom.xml| 5 +++ .../src/test/resources/logback-test.xml | 43 2 files changed, 48 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/34f10299/server/mailrepository/mailrepository-cassandra/pom.xml -- diff --git a/server/mailrepository/mailrepository-cassandra/pom.xml b/server/mailrepository/mailrepository-cassandra/pom.xml index 479a90f..506fe9e 100644 --- a/server/mailrepository/mailrepository-cassandra/pom.xml +++ b/server/mailrepository/mailrepository-cassandra/pom.xml @@ -78,6 +78,11 @@ test +ch.qos.logback +logback-classic +test + + org.assertj assertj-core test http://git-wip-us.apache.org/repos/asf/james-project/blob/34f10299/server/mailrepository/mailrepository-cassandra/src/test/resources/logback-test.xml -- diff --git a/server/mailrepository/mailrepository-cassandra/src/test/resources/logback-test.xml b/server/mailrepository/mailrepository-cassandra/src/test/resources/logback-test.xml new file mode 100644 index 000..cf76022 --- /dev/null +++ b/server/mailrepository/mailrepository-cassandra/src/test/resources/logback-test.xml @@ -0,0 +1,43 @@ + + + + + + +true + + + + +%d{HH:mm:ss.SSS} [%-5level] %logger{15} - %msg%n%rEx +false + + + + + + + + + + + + + - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[11/14] james-project git commit: JAMES-2334 Disable nodeKillingWhenConsuming
JAMES-2334 Disable nodeKillingWhenConsuming Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/86961f3f Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/86961f3f Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/86961f3f Branch: refs/heads/master Commit: 86961f3fc818ab3fdee2ddc76dcd16ff06e61dff Parents: de7d0e0 Author: Benoit Tellier Authored: Mon Oct 8 10:32:50 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:36 2018 +0700 -- .../org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java| 3 +++ 1 file changed, 3 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/86961f3f/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java -- diff --git a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java b/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java index 081f0d1..b72b45b 100644 --- a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java +++ b/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java @@ -245,6 +245,9 @@ class RabbitMQClusterTest { } } +@Disabled("JAMES-2334 For some reason, we are unable to recover topology when reconnecting" + +"See https://github.com/rabbitmq/rabbitmq-server/issues/959"; + +"This test have roughly 4% chance to fail") @Test void nodeKillingWhenConsuming(DockerRabbitMQCluster cluster) throws Exception { resilientChannel.exchangeDeclare(EXCHANGE_NAME, DIRECT, DURABLE); - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[06/14] james-project git commit: JAMES-2556 Add tests for multi-path repository reprocessing
JAMES-2556 Add tests for multi-path repository reprocessing Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/6cf97c9d Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/6cf97c9d Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/6cf97c9d Branch: refs/heads/master Commit: 6cf97c9d5bbe32c77e4923767f9cc547a6c9e784 Parents: 9d1d697 Author: Benoit Tellier Authored: Wed Oct 10 10:08:38 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:19 2018 +0700 -- .../routes/MailRepositoriesRoutesTest.java | 120 +++ .../src/test/resources/mailrepositorystore.xml | 5 + 2 files changed, 125 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/6cf97c9d/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java index 4ea84f7..7294348 100644 --- a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java +++ b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java @@ -83,6 +83,7 @@ import org.apache.mailet.base.test.FakeMail; import org.eclipse.jetty.http.HttpStatus; import org.junit.After; import org.junit.Before; +import org.junit.Ignore; import org.junit.Test; import com.google.common.collect.Sets; @@ -95,6 +96,7 @@ import io.restassured.parsing.Parser; public class MailRepositoriesRoutesTest { private static final MailRepositoryUrl URL_MY_REPO = MailRepositoryUrl.from("memory://myRepo"); +private static final MailRepositoryUrl URL_MY_REPO_OTHER = MailRepositoryUrl.from("other://myRepo"); private static final MailRepositoryPath PATH_MY_REPO = MailRepositoryPath.from("myRepo"); private static final String PATH_ESCAPED_MY_REPO = "myRepo"; private static final String MY_REPO_MAILS = "myRepo/mails"; @@ -1045,6 +1047,36 @@ public class MailRepositoriesRoutesTest { } @Test +public void reprocessingAllTaskShouldNotFailWhenSeveralRepositoriesWithSamePath() throws Exception { +MailRepository mailRepository = mailRepositoryStore.create(URL_MY_REPO); +mailRepositoryStore.create(URL_MY_REPO_OTHER); +String name1 = "name1"; +String name2 = "name2"; +mailRepository.store(FakeMail.builder() +.name(name1) +.build()); +mailRepository.store(FakeMail.builder() +.name(name2) +.build()); + +String transport = "transport"; +String taskId = with() +.param("action", "reprocess") +.param("queue", CUSTOM_QUEUE) +.param("processor", transport) +.patch(PATH_ESCAPED_MY_REPO + "/mails") +.jsonPath() +.get("taskId"); + +given() +.basePath(TasksRoutes.BASE) +.when() +.get(taskId + "/await") +.then() +.body("status", is("completed")); +} + +@Test public void reprocessingAllTaskShouldClearMailRepository() throws Exception { MailRepository mailRepository = mailRepositoryStore.create(URL_MY_REPO); String name1 = "name1"; @@ -1073,6 +1105,36 @@ public class MailRepositoriesRoutesTest { } @Test +public void reprocessingAllTaskShouldClearBothMailRepositoriesWhenSamePath() throws Exception { +MailRepository mailRepository1 = mailRepositoryStore.create(URL_MY_REPO); +MailRepository mailRepository2 = mailRepositoryStore.create(URL_MY_REPO_OTHER); +String name1 = "name1"; +String name2 = "name2"; +mailRepository1.store(FakeMail.builder() +.name(name1) +.build()); +mailRepository2.store(FakeMail.builder() +.name(name2) +.build()); + +String transport = "transport"; +String taskId = with() +.param("action", "reprocess") +.param("queue", CUSTOM_QUEUE) +.param("processor", transport) +.patch(PATH_ESCAPED_MY_REPO + "/mails") +.jsonPath() +.get("taskId"); + +with() +.basePath(TasksRoutes.BASE) +.get(taskId + "/await"); + +assertThat(mailRepository1.list()).isEmpty(); +assertThat(mailRepository2.list()).isEmpty(); +} + +
[10/14] james-project git commit: JAMES-2551 Fasting up RabbitMQ cluster startup
JAMES-2551 Fasting up RabbitMQ cluster startup - No need to start App and await node 1, which is never stopper - Doing a single big step run in parallel is more efficient than several little parallel steps (-30 s) Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/33bfd0d7 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/33bfd0d7 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/33bfd0d7 Branch: refs/heads/master Commit: 33bfd0d7aa5bb30c37decee8eee6b80218300540 Parents: 86961f3 Author: Benoit Tellier Authored: Mon Oct 8 13:23:59 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:36 2018 +0700 -- .../backend/rabbitmq/DockerClusterRabbitMQExtension.java | 10 -- .../org/apache/james/backend/rabbitmq/DockerRabbitMQ.java | 2 ++ 2 files changed, 2 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/33bfd0d7/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerClusterRabbitMQExtension.java -- diff --git a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerClusterRabbitMQExtension.java b/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerClusterRabbitMQExtension.java index 811c768..aede7c6 100644 --- a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerClusterRabbitMQExtension.java +++ b/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerClusterRabbitMQExtension.java @@ -64,17 +64,7 @@ public class DockerClusterRabbitMQExtension implements BeforeEachCallback, After Throwing.runnable(() -> rabbitMQ2.join(rabbitMQ1)), Throwing.runnable(() -> rabbitMQ3.join(rabbitMQ1))); -Runnables.runParallel( -Throwing.runnable(rabbitMQ1::startApp), -Throwing.runnable(rabbitMQ2::startApp), -Throwing.runnable(rabbitMQ3::startApp)); - cluster = new DockerRabbitMQCluster(rabbitMQ1, rabbitMQ2, rabbitMQ3); - -Runnables.runParallel( -rabbitMQ1::waitForReadyness, -rabbitMQ2::waitForReadyness, -rabbitMQ3::waitForReadyness); } @Override http://git-wip-us.apache.org/repos/asf/james-project/blob/33bfd0d7/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerRabbitMQ.java -- diff --git a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerRabbitMQ.java b/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerRabbitMQ.java index a546939..29add81 100644 --- a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerRabbitMQ.java +++ b/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/DockerRabbitMQ.java @@ -140,6 +140,8 @@ public class DockerRabbitMQ { public void join(DockerRabbitMQ rabbitMQ) throws Exception { stopApp(); joinCluster(rabbitMQ); +startApp(); +waitForReadyness(); } public void stopApp() throws java.io.IOException, InterruptedException { - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[08/14] james-project git commit: JAMES-2556 Add tests for multi-path repository management
JAMES-2556 Add tests for multi-path repository management This proves limit and offset to be failing Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/ea70abcd Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/ea70abcd Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/ea70abcd Branch: refs/heads/master Commit: ea70abcd16910b63822061f0e5858f7d8be1cbd3 Parents: 6cf97c9 Author: Benoit Tellier Authored: Wed Oct 10 10:16:52 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:19 2018 +0700 -- .../routes/MailRepositoriesRoutesTest.java | 108 +++ 1 file changed, 108 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/ea70abcd/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java index 7294348..f17b20c 100644 --- a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java +++ b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java @@ -212,6 +212,20 @@ public class MailRepositoriesRoutesTest { } @Test +public void getMailRepositoriesShouldDeduplicateAccordingToPath() throws Exception { +mailRepositoryStore.create(URL_MY_REPO); +mailRepositoryStore.create(URL_MY_REPO_OTHER); + +when() +.get() +.then() +.statusCode(HttpStatus.OK_200) +.body("", hasSize(1)) +.body("[0].repository", is(PATH_MY_REPO.asString())) +.body("[0].path", is(PATH_ESCAPED_MY_REPO)); +} + +@Test public void getMailRepositoriesShouldReturnTwoRepositoriesWhenTwo() throws Exception { mailRepositoryStore.create(URL_MY_REPO); mailRepositoryStore.create(MailRepositoryUrl.from("memory://mySecondRepo")); @@ -271,6 +285,26 @@ public class MailRepositoriesRoutesTest { } @Test +public void listingKeysShouldMergeRepositoryContentWhenSamePath() throws Exception { +MailRepository mailRepository1 = mailRepositoryStore.create(URL_MY_REPO); +MailRepository mailRepository2 = mailRepositoryStore.create(URL_MY_REPO_OTHER); + +mailRepository1.store(FakeMail.builder() +.name("name1") +.build()); +mailRepository2.store(FakeMail.builder() +.name("name2") +.build()); + +when() +.get(MY_REPO_MAILS) +.then() +.statusCode(HttpStatus.OK_200) +.body("", hasSize(2)) +.body("", containsInAnyOrder("name1", "name2")); +} + +@Test public void listingKeysShouldApplyLimitAndOffset() throws Exception { MailRepository mailRepository = mailRepositoryStore.create(URL_MY_REPO); @@ -295,6 +329,31 @@ public class MailRepositoriesRoutesTest { .body("", contains("name2")); } +@Ignore("Limit and offset are applied on a per repository bases") +@Test +public void listingKeysShouldApplyLimitWhenSeveralRepositories() throws Exception { +MailRepository mailRepository1 = mailRepositoryStore.create(URL_MY_REPO); +MailRepository mailRepository2 = mailRepositoryStore.create(URL_MY_REPO_OTHER); + +mailRepository1.store(FakeMail.builder() +.name("name1") +.build()); +mailRepository1.store(FakeMail.builder() +.name("name2") +.build()); +mailRepository2.store(FakeMail.builder() +.name("name3") +.build()); + +given() +.param("limit", "1") +.when() +.get(MY_REPO_MAILS) +.then() +.statusCode(HttpStatus.OK_200) +.body("", hasSize(1)); +} + @Test public void listingKeysShouldReturnErrorOnInvalidOffset() { given() @@ -344,6 +403,34 @@ public class MailRepositoriesRoutesTest { .body("", hasSize(0)); } +@Ignore("Offset is applied on a per mail repository basis") +@Test +public void offsetShouldBeAplliedOnTheMergedViewOfMailRepositories() throws Exception { +MailRepository mailRepository1 = mailRepositoryStore.create(URL_MY_REPO); +MailRepository mailRepository2 = mailRepositoryStore.create(URL_MY_REPO_OTHER); + +mailRepositor
[03/14] james-project git commit: JAMES-2556 Add a test proving reprocessing failure upon concurent mail deletion
JAMES-2556 Add a test proving reprocessing failure upon concurent mail deletion Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/9d1d697f Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/9d1d697f Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/9d1d697f Branch: refs/heads/master Commit: 9d1d697f6a14b1b1245a846a141787c835a09271 Parents: 0d8c68a Author: Benoit Tellier Authored: Wed Oct 10 10:32:11 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:06 2018 +0700 -- .../service/ReprocessingServiceTest.java| 196 +++ 1 file changed, 196 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/9d1d697f/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/service/ReprocessingServiceTest.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/service/ReprocessingServiceTest.java b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/service/ReprocessingServiceTest.java new file mode 100644 index 000..c1d3688 --- /dev/null +++ b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/service/ReprocessingServiceTest.java @@ -0,0 +1,196 @@ +/ + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information* + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the* + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the* + * specific language governing permissions and limitations * + * under the License. * + / + +package org.apache.james.webadmin.service; + +import static org.assertj.core.api.Assertions.assertThat; + +import java.util.Optional; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.function.Consumer; + +import org.apache.james.filesystem.api.FileSystem; +import org.apache.james.mailrepository.api.MailKey; +import org.apache.james.mailrepository.api.MailRepository; +import org.apache.james.mailrepository.api.MailRepositoryPath; +import org.apache.james.mailrepository.api.MailRepositoryUrl; +import org.apache.james.mailrepository.memory.MemoryMailRepositoryProvider; +import org.apache.james.mailrepository.memory.MemoryMailRepositoryStore; +import org.apache.james.mailrepository.memory.MemoryMailRepositoryUrlStore; +import org.apache.james.queue.api.MailQueueFactory; +import org.apache.james.queue.api.ManageableMailQueue; +import org.apache.james.queue.api.RawMailQueueItemDecoratorFactory; +import org.apache.james.queue.memory.MemoryMailQueueFactory; +import org.apache.james.server.core.configuration.Configuration; +import org.apache.james.server.core.configuration.FileConfigurationProvider; +import org.apache.james.server.core.filesystem.FileSystemImpl; +import org.apache.mailet.base.test.FakeMail; +import org.junit.Before; +import org.junit.Ignore; +import org.junit.Test; + +import com.github.fge.lambdas.Throwing; +import com.github.fge.lambdas.consumers.ConsumerChainer; +import com.google.common.collect.ImmutableList; +import com.google.common.collect.Sets; + +public class ReprocessingServiceTest { +private static final String MEMORY_PROTOCOL = "memory"; +private static final MailRepositoryPath PATH = MailRepositoryPath.from("path"); +private static final String NAME_1 = "key-1"; +private static final String NAME_2 = "key-2"; +private static final String NAME_3 = "key-3"; +private static final MailKey KEY_1 = new MailKey(NAME_1); +private static final MailKey KEY_2 = new MailKey(NAME_2); +private static final MailKey KEY_3 = new MailKey(NAME_3); +private static final String SPOOL = "spool"; +private static final Consumer NOOP_CONSUMER = key -> {}; +private static final Optional NO_TAR
[02/14] james-project git commit: JAMES-2557 DLP should ignore null sender
JAMES-2557 DLP should ignore null sender Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/0d8c68ab Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/0d8c68ab Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/0d8c68ab Branch: refs/heads/master Commit: 0d8c68ab7860f69eb1c376fc89ab04f0926e223c Parents: 0135291 Author: Benoit Tellier Authored: Tue Oct 9 11:44:27 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:45:50 2018 +0700 -- .../org/apache/james/transport/matchers/dlp/Dlp.java | 4 ++-- .../james/transport/matchers/dlp/DlpDomainRules.java | 4 +++- .../apache/james/transport/matchers/dlp/DlpTest.java | 13 + 3 files changed, 18 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/0d8c68ab/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/Dlp.java -- diff --git a/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/Dlp.java b/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/Dlp.java index 1f65082..a82dc06 100644 --- a/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/Dlp.java +++ b/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/Dlp.java @@ -23,7 +23,6 @@ import java.util.Collection; import java.util.Optional; import javax.inject.Inject; -import javax.mail.MessagingException; import org.apache.james.core.MailAddress; import org.apache.james.dlp.api.DLPConfigurationItem; @@ -51,7 +50,7 @@ public class Dlp extends GenericMatcher { } @Override -public Collection match(Mail mail) throws MessagingException { +public Collection match(Mail mail) { Optional firstMatchingRuleId = findFirstMatchingRule(mail); if (firstMatchingRuleId.isPresent()) { @@ -69,6 +68,7 @@ public class Dlp extends GenericMatcher { private Optional findFirstMatchingRule(Mail mail) { return Optional .ofNullable(mail.getSender()) +.filter(sender -> !sender.isNullSender()) .flatMap(sender -> matchingRule(sender, mail)); } http://git-wip-us.apache.org/repos/asf/james-project/blob/0d8c68ab/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/DlpDomainRules.java -- diff --git a/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/DlpDomainRules.java b/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/DlpDomainRules.java index b954801..031f306 100644 --- a/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/DlpDomainRules.java +++ b/server/mailet/mailets/src/main/java/org/apache/james/transport/matchers/dlp/DlpDomainRules.java @@ -169,7 +169,9 @@ public class DlpDomainRules { } private Stream listEnvelopSender(Mail mail) { -return StreamUtils.ofNullables(mail.getSender()).map(MailAddress::asString); +return StreamUtils.ofNullables(mail.getSender()) +.filter(sender -> !sender.isNullSender()) +.map(MailAddress::asString); } private Stream listFromHeaders(Mail mail) throws MessagingException { http://git-wip-us.apache.org/repos/asf/james-project/blob/0d8c68ab/server/mailet/mailets/src/test/java/org/apache/james/transport/matchers/dlp/DlpTest.java -- diff --git a/server/mailet/mailets/src/test/java/org/apache/james/transport/matchers/dlp/DlpTest.java b/server/mailet/mailets/src/test/java/org/apache/james/transport/matchers/dlp/DlpTest.java index 12aa394..88266d6 100644 --- a/server/mailet/mailets/src/test/java/org/apache/james/transport/matchers/dlp/DlpTest.java +++ b/server/mailet/mailets/src/test/java/org/apache/james/transport/matchers/dlp/DlpTest.java @@ -34,6 +34,7 @@ import java.util.Optional; import java.util.regex.Pattern; import org.apache.james.core.Domain; +import org.apache.james.core.MailAddress; import org.apache.james.core.builder.MimeMessageBuilder; import org.apache.james.dlp.api.DLPConfigurationItem.Id; import org.apache.mailet.base.test.FakeMail; @@ -103,6 +104,18 @@ class DlpTest { } @Test +void nullSenderShouldBeIgnored() throws Exception { +Dlp dlp = new Dlp( +asRulesLoaderFor( +JAMES_APACHE_ORG_DOMAIN, +DlpDomainRules.builder().recipientRule(Id.of("match all recipient"), Pattern.compile(".*")).build())); + +FakeMail mail = FakeMail.builde
[01/14] james-project git commit: JAMES-2544 Use Feign retry mechanism to retry calls on RabbitMQ management API
Repository: james-project Updated Branches: refs/heads/master eec12864d -> 34f102990 JAMES-2544 Use Feign retry mechanism to retry calls on RabbitMQ management API Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/01352919 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/01352919 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/01352919 Branch: refs/heads/master Commit: 01352919615cb49d70934e03759e885cb69dc7ad Parents: eec1286 Author: Benoit Tellier Authored: Wed Oct 10 09:12:16 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 09:15:24 2018 +0700 -- .../james/queue/rabbitmq/RabbitMQManagementApi.java| 13 + 1 file changed, 13 insertions(+) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/01352919/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQManagementApi.java -- diff --git a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQManagementApi.java b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQManagementApi.java index 8d39b24..cbeea3a 100644 --- a/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQManagementApi.java +++ b/server/queue/queue-rabbitmq/src/main/java/org/apache/james/queue/rabbitmq/RabbitMQManagementApi.java @@ -19,6 +19,7 @@ package org.apache.james.queue.rabbitmq; +import java.util.Date; import java.util.List; import java.util.stream.Stream; @@ -30,13 +31,23 @@ import com.fasterxml.jackson.annotation.JsonProperty; import feign.Feign; import feign.Logger; import feign.RequestLine; +import feign.RetryableException; +import feign.Retryer; import feign.auth.BasicAuthRequestInterceptor; +import feign.codec.ErrorDecoder; import feign.jackson.JacksonDecoder; import feign.jackson.JacksonEncoder; import feign.slf4j.Slf4jLogger; class RabbitMQManagementApi { +private static final ErrorDecoder RETRY_500 = (methodKey, response) -> { +if (response.status() == 500) { +throw new RetryableException("Error encountered, scheduling retry", response.request().httpMethod(), new Date()); +} +throw new RuntimeException("Non recoverable exception status: " + response.status()); +}; + public interface Api { class MessageQueue { @@ -58,6 +69,8 @@ class RabbitMQManagementApi { .logLevel(Logger.Level.FULL) .encoder(new JacksonEncoder()) .decoder(new JacksonDecoder()) +.retryer(new Retryer.Default()) +.errorDecoder(RETRY_500) .target(Api.class, configuration.getManagementUri().toString()); } - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[07/14] james-project git commit: JAMES-2556 Remove unused MailRepositoryStoreService::getRepository
JAMES-2556 Remove unused MailRepositoryStoreService::getRepository Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/95b4ead2 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/95b4ead2 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/95b4ead2 Branch: refs/heads/master Commit: 95b4ead22d4fd4d934f848c10579b258ab0d4899 Parents: d68a63e Author: Benoit Tellier Authored: Wed Oct 10 10:22:48 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:19 2018 +0700 -- .../james/webadmin/service/MailRepositoryStoreService.java | 9 - 1 file changed, 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/95b4ead2/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java b/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java index a626655..987b443 100644 --- a/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java +++ b/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java @@ -105,15 +105,6 @@ public class MailRepositoryStoreService { return new ClearMailRepositoryTask(getRepositories(path).collect(Guavate.toImmutableList()), path); } -public MailRepository getRepository(MailRepositoryUrl url) throws MailRepositoryStore.MailRepositoryStoreException { -return mailRepositoryStore.get(url) -.orElseThrow(() -> ErrorResponder.builder() -.statusCode(HttpStatus.NOT_FOUND_404) -.type(ErrorResponder.ErrorType.NOT_FOUND) -.message(url.asString() + " does not exist") -.haltError()); -} - public Stream getRepositories(MailRepositoryPath path) throws MailRepositoryStore.MailRepositoryStoreException { Stream byPath = mailRepositoryStore.getByPath(path); List repositories = byPath.collect(Collectors.toList()); - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[05/14] james-project git commit: JAMES-2556 Fix reprocessing upon concurent mail deletion
JAMES-2556 Fix reprocessing upon concurent mail deletion Single key reprocessing should fail on missing key while full reprocessing should not. Because the argument list keeps growing adding a 'Reprocessor' object helps readability. Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/f15655b2 Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/f15655b2 Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/f15655b2 Branch: refs/heads/master Commit: f15655b20e3e0e3b35707773ba70ddded2563403 Parents: 95b4ead Author: Benoit Tellier Authored: Wed Oct 10 10:41:29 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:19 2018 +0700 -- .../webadmin/service/ReprocessingService.java | 47 +++- .../service/ReprocessingServiceTest.java| 2 - 2 files changed, 36 insertions(+), 13 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/f15655b2/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java b/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java index f02bd3f..c316b95 100644 --- a/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java +++ b/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/ReprocessingService.java @@ -31,12 +31,38 @@ import org.apache.james.mailrepository.api.MailRepositoryPath; import org.apache.james.mailrepository.api.MailRepositoryStore; import org.apache.james.queue.api.MailQueue; import org.apache.james.queue.api.MailQueueFactory; +import org.apache.james.util.OptionalUtils; import org.apache.james.util.streams.Iterators; import org.apache.mailet.Mail; import com.github.fge.lambdas.Throwing; public class ReprocessingService { +public static class MissingKeyException extends RuntimeException { +MissingKeyException(MailKey key) { +super(key.asString() + " can not be found"); +} +} + +static class Reprocessor { +private final MailQueue mailQueue; +private final Optional targetProcessor; + +Reprocessor(MailQueue mailQueue, Optional targetProcessor) { +this.mailQueue = mailQueue; +this.targetProcessor = targetProcessor; +} + +private void reprocess(MailRepository repository, Mail mail) { +try { +targetProcessor.ifPresent(mail::setState); +mailQueue.enQueue(mail); +repository.remove(mail); +} catch (Exception e) { +throw new RuntimeException("Error encountered while reprocessing mail " + mail.getName(), e); +} +} +} private final MailQueueFactory mailQueueFactory; private final MailRepositoryStoreService mailRepositoryStoreService; @@ -49,29 +75,28 @@ public class ReprocessingService { } public void reprocessAll(MailRepositoryPath path, Optional targetProcessor, String targetQueue, Consumer keyListener) throws MailRepositoryStore.MailRepositoryStoreException, MessagingException { -MailQueue mailQueue = getMailQueue(targetQueue); +Reprocessor reprocessor = new Reprocessor(getMailQueue(targetQueue), targetProcessor); mailRepositoryStoreService .getRepositories(path) .forEach(Throwing.consumer((MailRepository repository) -> Iterators.toStream(repository.list()) .peek(keyListener) -.forEach(Throwing.consumer(key -> reprocess(repository, mailQueue, key, targetProcessor.sneakyThrow()); +.map(Throwing.function(key -> Optional.ofNullable(repository.retrieve(key +.flatMap(OptionalUtils::toStream) +.forEach(mail -> reprocessor.reprocess(repository, mail; } public void reprocess(MailRepositoryPath path, MailKey key, Optional targetProcessor, String targetQueue) throws MailRepositoryStore.MailRepositoryStoreException, MessagingException { -MailQueue mailQueue = getMailQueue(targetQueue); +Reprocessor reprocessor = new Reprocessor(getMailQueue(targetQueue), targetProcessor); mailRepositoryStoreService .getRepositories(path) -.forEach(Throwing.consumer((MailRepository repository) -> reprocess(repository, mailQueue, key, targetProcessor)).sneak
[09/14] james-project git commit: JAMES-2556 Fix limit and offset behaviour in multi-repository context
JAMES-2556 Fix limit and offset behaviour in multi-repository context Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/d68a63ef Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/d68a63ef Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/d68a63ef Branch: refs/heads/master Commit: d68a63efda3734eb0b34bc2c7abde327762a02c5 Parents: ea70abc Author: Benoit Tellier Authored: Wed Oct 10 10:22:00 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:19 2018 +0700 -- .../service/MailRepositoryStoreService.java | 18 +++--- .../routes/MailRepositoriesRoutesTest.java| 2 -- 2 files changed, 7 insertions(+), 13 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/d68a63ef/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java b/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java index ff4ee76..a626655 100644 --- a/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java +++ b/server/protocols/webadmin/webadmin-mailrepository/src/main/java/org/apache/james/webadmin/service/MailRepositoryStoreService.java @@ -48,8 +48,8 @@ import org.apache.mailet.Mail; import org.eclipse.jetty.http.HttpStatus; import com.github.fge.lambdas.Throwing; -import com.github.fge.lambdas.functions.ThrowingFunction; import com.github.steveash.guavate.Guavate; +import com.google.common.collect.ImmutableList; public class MailRepositoryStoreService { private final MailRepositoryStore mailRepositoryStore; @@ -70,17 +70,13 @@ public class MailRepositoryStoreService { } public Optional> listMails(MailRepositoryPath path, Offset offset, Limit limit) throws MailRepositoryStore.MailRepositoryStoreException, MessagingException { -ThrowingFunction> list = repository -> list(repository, offset, limit); -return Optional.of(getRepositories(path) -.flatMap(Throwing.function(list).sneakyThrow()) -.collect(Guavate.toImmutableList())); -} +Optional> maybeMails = Optional.of(getRepositories(path) +.flatMap(Throwing.function((MailRepository repository) -> Iterators.toStream(repository.list())).sneakyThrow()) +.map(MailKeyDTO::new) +.skip(offset.getOffset())); -private Stream list(MailRepository mailRepository, Offset offset, Limit limit) throws MessagingException { -return limit.applyOnStream( -Iterators.toStream(mailRepository.list()) -.skip(offset.getOffset())) -.map(MailKeyDTO::new); +return maybeMails.map(limit::applyOnStream) +.map(stream -> stream.collect(ImmutableList.toImmutableList())); } public Optional size(MailRepositoryPath path) throws MailRepositoryStore.MailRepositoryStoreException { http://git-wip-us.apache.org/repos/asf/james-project/blob/d68a63ef/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java -- diff --git a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java index f17b20c..ea78e4c 100644 --- a/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java +++ b/server/protocols/webadmin/webadmin-mailrepository/src/test/java/org/apache/james/webadmin/routes/MailRepositoriesRoutesTest.java @@ -329,7 +329,6 @@ public class MailRepositoriesRoutesTest { .body("", contains("name2")); } -@Ignore("Limit and offset are applied on a per repository bases") @Test public void listingKeysShouldApplyLimitWhenSeveralRepositories() throws Exception { MailRepository mailRepository1 = mailRepositoryStore.create(URL_MY_REPO); @@ -403,7 +402,6 @@ public class MailRepositoriesRoutesTest { .body("", hasSize(0)); } -@Ignore("Offset is applied on a per mail repository basis") @Test public void offsetShouldBeAplliedOnTheMergedViewOfMailRepositories() throws Exception { MailRepository mailRep
[12/14] james-project git commit: JAMES-2334 Use resilient channel for queue declaration
JAMES-2334 Use resilient channel for queue declaration Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/de7d0e0a Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/de7d0e0a Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/de7d0e0a Branch: refs/heads/master Commit: de7d0e0aaf711a6da6105788d2bbecacdbb2d967 Parents: 9d52d56 Author: Benoit Tellier Authored: Mon Oct 8 10:31:27 2018 +0700 Committer: Benoit Tellier Committed: Wed Oct 10 15:46:36 2018 +0700 -- .../org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/james-project/blob/de7d0e0a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java -- diff --git a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java b/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java index 6486a84..081f0d1 100644 --- a/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java +++ b/backends-common/rabbitmq/src/test/java/org/apache/james/backend/rabbitmq/RabbitMQClusterTest.java @@ -247,9 +247,9 @@ class RabbitMQClusterTest { @Test void nodeKillingWhenConsuming(DockerRabbitMQCluster cluster) throws Exception { -node2Channel.exchangeDeclare(EXCHANGE_NAME, DIRECT, DURABLE); -node2Channel.queueDeclare(QUEUE, DURABLE, !EXCLUSIVE, !AUTO_DELETE, ImmutableMap.of()).getQueue(); -node2Channel.queueBind(QUEUE, EXCHANGE_NAME, ROUTING_KEY); +resilientChannel.exchangeDeclare(EXCHANGE_NAME, DIRECT, DURABLE); +resilientChannel.queueDeclare(QUEUE, DURABLE, !EXCLUSIVE, !AUTO_DELETE, ImmutableMap.of()).getQueue(); +resilientChannel.queueBind(QUEUE, EXCHANGE_NAME, ROUTING_KEY); int nbMessages = 10; IntStream.range(0, nbMessages) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Updated] (JAMES-2558) Use SpamAssassin version 3.4.2 in our test suite
[ https://issues.apache.org/jira/browse/JAMES-2558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tellier Benoit updated JAMES-2558: -- Priority: Minor (was: Major) > Use SpamAssassin version 3.4.2 in our test suite > > > Key: JAMES-2558 > URL: https://issues.apache.org/jira/browse/JAMES-2558 > Project: James Server > Issue Type: Improvement > Components: mailbox, Mailet Contributions >Affects Versions: master >Reporter: Tellier Benoit >Priority: Minor > Labels: security > Fix For: 3.2.0 > > > Version 3.4.1 is vulnerable to CVE-2017-15705, CVE-2016-1238, CVE-2018-11780 > & CVE-2018-11781 > Causing denial of service. > JAMES only enforces SpamAssassin 3.4.1 use as part of its test suite, and > SpamAssassin components are currently undocumented. > Thus we just need to update SpamAssassin images used in tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
[jira] [Created] (JAMES-2558) Use SpamAssassin version 3.4.2 in our test suite
Tellier Benoit created JAMES-2558: - Summary: Use SpamAssassin version 3.4.2 in our test suite Key: JAMES-2558 URL: https://issues.apache.org/jira/browse/JAMES-2558 Project: James Server Issue Type: Improvement Components: mailbox, Mailet Contributions Affects Versions: master Reporter: Tellier Benoit Fix For: 3.2.0 Version 3.4.1 is vulnerable to CVE-2017-15705, CVE-2016-1238, CVE-2018-11780 & CVE-2018-11781 Causing denial of service. JAMES only enforces SpamAssassin 3.4.1 use as part of its test suite, and SpamAssassin components are currently undocumented. Thus we just need to update SpamAssassin images used in tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org