JAMES-2494 Remove Kafka
Project: http://git-wip-us.apache.org/repos/asf/james-project/repo Commit: http://git-wip-us.apache.org/repos/asf/james-project/commit/f9e478fc Tree: http://git-wip-us.apache.org/repos/asf/james-project/tree/f9e478fc Diff: http://git-wip-us.apache.org/repos/asf/james-project/diff/f9e478fc Branch: refs/heads/master Commit: f9e478fce6d116ac10cb31e26bec89dc770fa240 Parents: 44583eb Author: Gautier DI FOLCO <[email protected]> Authored: Mon Jul 23 15:30:06 2018 +0200 Committer: benwa <[email protected]> Committed: Wed Jul 25 09:34:51 2018 +0700 ---------------------------------------------------------------------- .../spring/destination/conf/events.properties | 42 ------ .../run/spring/destination/conf/events.xml | 48 +------ mailbox/kafka/pom.xml | 85 ------------ .../mailbox/kafka/KafkaMessageConsumer.java | 136 ------------------- .../james/mailbox/kafka/KafkaPublisher.java | 79 ----------- mailbox/pom.xml | 1 - .../james/mailbox/spring/SpringMailbox.java | 76 +++++------ .../resources/META-INF/spring/event-system.xml | 5 - .../META-INF/spring/spring-mailbox.xml | 6 + .../resources/META-INF/spring/event-alias.xml | 34 ----- .../test/resources/META-INF/spring/metrics.xml | 27 ++++ .../spring/src/test/resources/events.properties | 38 ------ .../mailbox/store/publisher/Publisher.java | 4 +- pom.xml | 5 - server/app/src/main/resources/events.properties | 42 ------ server/app/src/main/resources/events.xml | 48 +------ server/app/src/test/resources/events.xml | 48 +------ ...tsConfigurationBeanFactoryPostProcessor.java | 120 ---------------- .../META-INF/org/apache/james/spring-server.xml | 1 - src/site/xdoc/server/config-events.xml | 27 +--- 20 files changed, 77 insertions(+), 795 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/dockerfiles/run/spring/destination/conf/events.properties ---------------------------------------------------------------------- diff --git a/dockerfiles/run/spring/destination/conf/events.properties b/dockerfiles/run/spring/destination/conf/events.properties deleted file mode 100644 index 3b49533..0000000 --- a/dockerfiles/run/spring/destination/conf/events.properties +++ /dev/null @@ -1,42 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# This template file can be used as example for James Server configuration -# DO NOT USE IT AS SUCH AND ADAPT IT TO YOUR NEEDS - -# The following properties needs to be set -# if you want to use Kafka for inter node messaging for your event system. -# -# This needs to be done if you used <publisher>kafka</publisher> in event.xml - -kafka.ip=127.0.0.1 -kafka.port=9092 -zookeeper.connection.string=localhost -group.id=azerty -event.thread.count=4 - -# This property needs to be set if you want to use the Broadcast Mailbox Delegating Listener - -global.topic=JAMES_TOPIC - -distant.mailbox.path.register.max.retries=100 -distant.mailbox.path.register.refresh=900 - -# Worker threads count for asynchronous event delivery - -event.delivery.thread.count=10 http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/dockerfiles/run/spring/destination/conf/events.xml ---------------------------------------------------------------------- diff --git a/dockerfiles/run/spring/destination/conf/events.xml b/dockerfiles/run/spring/destination/conf/events.xml index 536278a..5555280 100644 --- a/dockerfiles/run/spring/destination/conf/events.xml +++ b/dockerfiles/run/spring/destination/conf/events.xml @@ -36,55 +36,9 @@ --> <delivery>synchronous</delivery> - <!-- Broadcast event system will send messages to all your James servers. - - It is the only viable options if you rely on Mailbox Listeners that do not perform well in a distributed system. - - Here follows a non exhaustive list of such Mailbox Listeners : - - Memory quota based system - - Lucene based search system - - Workload is higher because event needs to be serialized and de serialized. - - Also note that depending on the publisher you use, messages might be either lost or duplicated, implying wrong - indexing or wrong quota count on network partitions. - - Here follows a sample configuration : - --> - <!-- - <type>broadcast</type> - <serialization>json</serialization> - <publisher>kafka</publisher> - --> - - <!-- - Further notes on the choice you had here : - - serialization : you have to choices : either json or message-pack - - message-pack is a binary representation for json, and is 2 times lighter by average, but is 2 time slower to serialize and de serialize. - This choice allows you to trade compute against network. - - publisher : Publish - subscribe used. The only current implementation is Kafka. - --> - - <!-- Registered event system - - Scalable event system based on registration : each James server listen on his own topic and manage registrations on - mailbox paths in order to receive events interesting him. - - This configuration does not support distributed unfriendly Mailbox Listeners (see the non exhaustive list above) for obvious reasons. - --> - <!-- - <type>registered</type> - <serialization>json</serialization> - <registration>cassandra</registration> - <publisher>kafka</publisher> - --> - <!-- The only registration system available is Cassandra, used in an available fashion. This means there is no warranties, either at least one delivery nor at most one delivery. But you get the certitude that the system will always be available. --> -</events> \ No newline at end of file +</events> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/kafka/pom.xml ---------------------------------------------------------------------- diff --git a/mailbox/kafka/pom.xml b/mailbox/kafka/pom.xml deleted file mode 100644 index 9e035ee..0000000 --- a/mailbox/kafka/pom.xml +++ /dev/null @@ -1,85 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> -<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> - <modelVersion>4.0.0</modelVersion> - <parent> - <groupId>org.apache.james</groupId> - <artifactId>apache-james-mailbox</artifactId> - <version>3.2.0-SNAPSHOT</version> - </parent> - - <artifactId>apache-james-mailbox-kafka</artifactId> - - <name>Apache James :: Mailbox :: Kafka</name> - - <dependencies> - <dependency> - <groupId>${james.groupId}</groupId> - <artifactId>apache-james-mailbox-api</artifactId> - </dependency> - <dependency> - <groupId>${james.groupId}</groupId> - <artifactId>apache-james-mailbox-api</artifactId> - <type>test-jar</type> - <scope>test</scope> - </dependency> - <dependency> - <groupId>${james.groupId}</groupId> - <artifactId>apache-james-mailbox-store</artifactId> - </dependency> - <dependency> - <groupId>${james.groupId}</groupId> - <artifactId>apache-james-mailbox-store</artifactId> - <type>test-jar</type> - <scope>test</scope> - </dependency> - - <dependency> - <groupId>junit</groupId> - <artifactId>junit</artifactId> - <scope>test</scope> - </dependency> - <dependency> - <groupId>org.apache.kafka</groupId> - <artifactId>kafka_2.10</artifactId> - <version>0.8.2.2</version> - </dependency> - <dependency> - <groupId>org.assertj</groupId> - <artifactId>assertj-core</artifactId> - <scope>test</scope> - </dependency> - <dependency> - <groupId>org.mockito</groupId> - <artifactId>mockito-core</artifactId> - <scope>test</scope> - </dependency> - <dependency> - <groupId>org.slf4j</groupId> - <artifactId>slf4j-api</artifactId> - </dependency> - <dependency> - <groupId>org.slf4j</groupId> - <artifactId>slf4j-simple</artifactId> - <scope>test</scope> - </dependency> - </dependencies> - -</project> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/kafka/src/main/java/org/apache/james/mailbox/kafka/KafkaMessageConsumer.java ---------------------------------------------------------------------- diff --git a/mailbox/kafka/src/main/java/org/apache/james/mailbox/kafka/KafkaMessageConsumer.java b/mailbox/kafka/src/main/java/org/apache/james/mailbox/kafka/KafkaMessageConsumer.java deleted file mode 100644 index 74ce42f..0000000 --- a/mailbox/kafka/src/main/java/org/apache/james/mailbox/kafka/KafkaMessageConsumer.java +++ /dev/null @@ -1,136 +0,0 @@ -/**************************************************************** - * Licensed to the Apache Software Foundation (ASF) under one * - * or more contributor license agreements. See the NOTICE file * - * distributed with this work for additional information * - * regarding copyright ownership. The ASF licenses this file * - * to you under the Apache License, Version 2.0 (the * - * "License"); you may not use this file except in compliance * - * with the License. You may obtain a copy of the License at * - * * - * http://www.apache.org/licenses/LICENSE-2.0 * - * * - * Unless required by applicable law or agreed to in writing, * - * software distributed under the License is distributed on an * - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * - * KIND, either express or implied. See the License for the * - * specific language governing permissions and limitations * - * under the License. * - ****************************************************************/ - -package org.apache.james.mailbox.kafka; - -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Properties; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; - -import javax.annotation.PostConstruct; -import javax.annotation.PreDestroy; - -import org.apache.james.mailbox.store.publisher.MessageConsumer; -import org.apache.james.mailbox.store.publisher.MessageReceiver; -import org.apache.james.mailbox.store.publisher.Topic; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import kafka.consumer.ConsumerConfig; -import kafka.consumer.KafkaStream; -import kafka.javaapi.consumer.ConsumerConnector; -import kafka.message.MessageAndMetadata; - -public class KafkaMessageConsumer implements MessageConsumer { - - private class Consumer implements Runnable { - - private final KafkaStream<byte[], byte[]> stream; - - public Consumer(KafkaStream<byte[], byte[]> stream) { - this.stream = stream; - } - - @Override - public void run() { - for (MessageAndMetadata<byte[], byte[]> message : stream) { - messageReceiver.receiveSerializedEvent(message.message()); - } - } - } - - private static final String ZK_SESSION_TIMEOUT = "400"; - private static final String ZK_SYNC_TIME = "200"; - private static final String AUTO_COMMIT8INTERVAL_MS = "1000"; - private static final Logger LOG = LoggerFactory.getLogger(KafkaMessageConsumer.class); - - private final ConsumerConnector consumer; - private final int numberOfTread; - private MessageReceiver messageReceiver; - private ExecutorService executor; - private boolean isInitialized; - - - public KafkaMessageConsumer(String zookeeperConnectionString, - String groupId, - int numberOfThread) { - this.consumer = kafka.consumer.Consumer.createJavaConsumerConnector(createConsumerConfig(zookeeperConnectionString, groupId)); - this.numberOfTread = numberOfThread; - this.isInitialized = false; - } - - @Override - public void setMessageReceiver(MessageReceiver messageReceiver) { - if (!isInitialized) { - this.messageReceiver = messageReceiver; - } else { - throw new RuntimeException("Can not change the MessageReceiver of a running KafkaMessageConsumer"); - } - } - - @Override - @PreDestroy - public void destroy() { - if (consumer != null) { - consumer.shutdown(); - } - if (executor != null) { - executor.shutdown(); - } - this.isInitialized = false; - } - - @Override - @PostConstruct - public void init(Topic topic) { - if (!isInitialized) { - this.isInitialized = true; - List<KafkaStream<byte[], byte[]>> streams = getKafkaStreams(topic.getValue()); - executor = Executors.newFixedThreadPool(numberOfTread); - startConsuming(streams); - } else { - LOG.warn("This Kafka MailboxMessage Receiver was already launched."); - } - } - - private List<KafkaStream<byte[], byte[]>> getKafkaStreams(String topic) { - Map<String, Integer> topicCountMap = new HashMap<>(); - topicCountMap.put(topic, numberOfTread); - Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap); - return consumerMap.get(topic); - } - - private void startConsuming(List<KafkaStream<byte[], byte[]>> streams) { - streams.forEach(stream -> executor.submit(new Consumer(stream))); - } - - private ConsumerConfig createConsumerConfig(String zookeeperConnectionString, String groupId) { - Properties props = new Properties(); - props.put("zookeeper.connect", zookeeperConnectionString); - props.put("group.id", groupId); - props.put("zookeeper.session.timeout.ms", ZK_SESSION_TIMEOUT); - props.put("zookeeper.sync.time.ms", ZK_SYNC_TIME); - props.put("auto.commit.interval.ms", AUTO_COMMIT8INTERVAL_MS); - return new ConsumerConfig(props); - } - -} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/kafka/src/main/java/org/apache/james/mailbox/kafka/KafkaPublisher.java ---------------------------------------------------------------------- diff --git a/mailbox/kafka/src/main/java/org/apache/james/mailbox/kafka/KafkaPublisher.java b/mailbox/kafka/src/main/java/org/apache/james/mailbox/kafka/KafkaPublisher.java deleted file mode 100644 index f304705..0000000 --- a/mailbox/kafka/src/main/java/org/apache/james/mailbox/kafka/KafkaPublisher.java +++ /dev/null @@ -1,79 +0,0 @@ -/**************************************************************** - * Licensed to the Apache Software Foundation (ASF) under one * - * or more contributor license agreements. See the NOTICE file * - * distributed with this work for additional information * - * regarding copyright ownership. The ASF licenses this file * - * to you under the Apache License, Version 2.0 (the * - * "License"); you may not use this file except in compliance * - * with the License. You may obtain a copy of the License at * - * * - * http://www.apache.org/licenses/LICENSE-2.0 * - * * - * Unless required by applicable law or agreed to in writing, * - * software distributed under the License is distributed on an * - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * - * KIND, either express or implied. See the License for the * - * specific language governing permissions and limitations * - * under the License. * - ****************************************************************/ - -package org.apache.james.mailbox.kafka; - -import java.util.Properties; - -import javax.annotation.PostConstruct; -import javax.annotation.PreDestroy; - -import org.apache.james.mailbox.store.publisher.Publisher; -import org.apache.james.mailbox.store.publisher.Topic; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import kafka.javaapi.producer.Producer; -import kafka.producer.KeyedMessage; -import kafka.producer.ProducerConfig; - -public class KafkaPublisher implements Publisher { - - private static final Logger LOG = LoggerFactory.getLogger(KafkaPublisher.class); - - private Producer<String, byte[]> producer; - private final int kafkaPort; - private final String kafkaIp; - private boolean producerLaunched; - - public KafkaPublisher(String kafkaHostIpString, int kafkaPort) { - this.kafkaIp = kafkaHostIpString; - this.kafkaPort = kafkaPort; - producerLaunched = false; - } - - @PostConstruct - @Override - public void init() { - if (!producerLaunched) { - Properties props = new Properties(); - props.put("metadata.broker.list", kafkaIp + ":" + kafkaPort); - props.put("serializer.class", "kafka.serializer.DefaultEncoder"); - props.put("request.required.acks", "1"); - ProducerConfig config = new ProducerConfig(props); - producer = new Producer<>(config); - producerLaunched = true; - } else { - LOG.warn("Kafka producer was already instantiated"); - } - } - - - @Override - public void publish(Topic topic, byte[] message) { - producer.send(new KeyedMessage<>(topic.getValue(), message)); - } - - @PreDestroy - @Override - public void close() { - producer.close(); - } - -} http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/pom.xml ---------------------------------------------------------------------- diff --git a/mailbox/pom.xml b/mailbox/pom.xml index 6479ba4..974ef85 100644 --- a/mailbox/pom.xml +++ b/mailbox/pom.xml @@ -43,7 +43,6 @@ <module>hbase</module> <module>jcr</module> <module>jpa</module> - <module>kafka</module> <module>lucene</module> <module>maildir</module> <module>memory</module> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/spring/src/main/java/org/apache/james/mailbox/spring/SpringMailbox.java ---------------------------------------------------------------------- diff --git a/mailbox/spring/src/main/java/org/apache/james/mailbox/spring/SpringMailbox.java b/mailbox/spring/src/main/java/org/apache/james/mailbox/spring/SpringMailbox.java index db85ad8..f68871a 100644 --- a/mailbox/spring/src/main/java/org/apache/james/mailbox/spring/SpringMailbox.java +++ b/mailbox/spring/src/main/java/org/apache/james/mailbox/spring/SpringMailbox.java @@ -1,38 +1,38 @@ -/**************************************************************** - * Licensed to the Apache Software Foundation (ASF) under one * - * or more contributor license agreements. See the NOTICE file * - * distributed with this work for additional information * - * regarding copyright ownership. The ASF licenses this file * - * to you under the Apache License, Version 2.0 (the * - * "License"); you may not use this file except in compliance * - * with the License. You may obtain a copy of the License at * - * * - * http://www.apache.org/licenses/LICENSE-2.0 * - * * - * Unless required by applicable law or agreed to in writing, * - * software distributed under the License is distributed on an * - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * - * KIND, either express or implied. See the License for the * - * specific language governing permissions and limitations * - * under the License. * - ****************************************************************/ -package org.apache.james.mailbox.spring; - -import org.springframework.context.ApplicationContext; -import org.springframework.context.support.ClassPathXmlApplicationContext; - -public class SpringMailbox { - - private final ApplicationContext applicationContext; - - public SpringMailbox() { - applicationContext = new ClassPathXmlApplicationContext("META-INF/spring/spring-mailbox.xml", - "META-INF/spring/mailbox-authenticator-anonymous.xml", "META-INF/spring/mailbox-no-authorizator.xml", "META-INF/spring/quota.xml", - "META-INF/spring/quota-alias.xml", "META-INF/spring/event-system.xml", "META-INF/spring/event-alias.xml"); - } - - public Object getBean(String beanName) { - return applicationContext.getBean(beanName); - } - -} +/**************************************************************** + * Licensed to the Apache Software Foundation (ASF) under one * + * or more contributor license agreements. See the NOTICE file * + * distributed with this work for additional information * + * regarding copyright ownership. The ASF licenses this file * + * to you under the Apache License, Version 2.0 (the * + * "License"); you may not use this file except in compliance * + * with the License. You may obtain a copy of the License at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * Unless required by applicable law or agreed to in writing, * + * software distributed under the License is distributed on an * + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * + * KIND, either express or implied. See the License for the * + * specific language governing permissions and limitations * + * under the License. * + ****************************************************************/ +package org.apache.james.mailbox.spring; + +import org.springframework.context.ApplicationContext; +import org.springframework.context.support.ClassPathXmlApplicationContext; + +public class SpringMailbox { + + private final ApplicationContext applicationContext; + + public SpringMailbox() { + applicationContext = new ClassPathXmlApplicationContext("META-INF/spring/spring-mailbox.xml", + "META-INF/spring/mailbox-authenticator-anonymous.xml", "META-INF/spring/mailbox-no-authorizator.xml", "META-INF/spring/quota.xml", + "META-INF/spring/quota-alias.xml", "META-INF/spring/event-system.xml", "META-INF/spring/metrics.xml"); + } + + public Object getBean(String beanName) { + return applicationContext.getBean(beanName); + } + +} http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/spring/src/main/resources/META-INF/spring/event-system.xml ---------------------------------------------------------------------- diff --git a/mailbox/spring/src/main/resources/META-INF/spring/event-system.xml b/mailbox/spring/src/main/resources/META-INF/spring/event-system.xml index 4e04252..2647d0c 100644 --- a/mailbox/spring/src/main/resources/META-INF/spring/event-system.xml +++ b/mailbox/spring/src/main/resources/META-INF/spring/event-system.xml @@ -23,11 +23,6 @@ xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd";> - <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> - <property name="ignoreUnresolvablePlaceholders" value="true"/> - <property name ="location" value="classpath:events.properties"/> - </bean> - <bean id="default-delegating-listener" class="org.apache.james.mailbox.store.event.DefaultDelegatingMailboxListener" lazy-init="true"> <constructor-arg index="0" ref="event-delivery"/> <constructor-arg index="1" ref="event-registry"/> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/spring/src/main/resources/META-INF/spring/spring-mailbox.xml ---------------------------------------------------------------------- diff --git a/mailbox/spring/src/main/resources/META-INF/spring/spring-mailbox.xml b/mailbox/spring/src/main/resources/META-INF/spring/spring-mailbox.xml index 0b50ccb..353cdfd 100644 --- a/mailbox/spring/src/main/resources/META-INF/spring/spring-mailbox.xml +++ b/mailbox/spring/src/main/resources/META-INF/spring/spring-mailbox.xml @@ -72,5 +72,11 @@ <constructor-arg index="3" ref="dispatcher" /> </bean> + <alias name="default-delegating-listener" alias="delegating-listener"/> + <alias name="synchronous-event-delivery" alias="event-delivery"/> + + <bean id="dispatcher" class="org.apache.james.mailbox.store.event.MailboxEventDispatcher"> + <constructor-arg index="0" ref="delegating-listener"/> + </bean> </beans> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/spring/src/test/resources/META-INF/spring/event-alias.xml ---------------------------------------------------------------------- diff --git a/mailbox/spring/src/test/resources/META-INF/spring/event-alias.xml b/mailbox/spring/src/test/resources/META-INF/spring/event-alias.xml deleted file mode 100644 index 7406b84..0000000 --- a/mailbox/spring/src/test/resources/META-INF/spring/event-alias.xml +++ /dev/null @@ -1,34 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> - -<beans xmlns="http://www.springframework.org/schema/beans"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd";> - - <alias name="default-delegating-listener" alias="delegating-listener"/> - <alias name="synchronous-event-delivery" alias="event-delivery"/> - - <bean id="dispatcher" class="org.apache.james.mailbox.store.event.MailboxEventDispatcher"> - <constructor-arg index="0" ref="delegating-listener"/> - </bean> - - <bean id="metricFactory" class="org.apache.james.metrics.api.NoopMetricFactory"/> - -</beans> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/spring/src/test/resources/META-INF/spring/metrics.xml ---------------------------------------------------------------------- diff --git a/mailbox/spring/src/test/resources/META-INF/spring/metrics.xml b/mailbox/spring/src/test/resources/META-INF/spring/metrics.xml new file mode 100644 index 0000000..83c8d78 --- /dev/null +++ b/mailbox/spring/src/test/resources/META-INF/spring/metrics.xml @@ -0,0 +1,27 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +<beans xmlns="http://www.springframework.org/schema/beans"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd";> + + <bean id="metricFactory" class="org.apache.james.metrics.api.NoopMetricFactory"/> + +</beans> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/spring/src/test/resources/events.properties ---------------------------------------------------------------------- diff --git a/mailbox/spring/src/test/resources/events.properties b/mailbox/spring/src/test/resources/events.properties deleted file mode 100644 index 696bc26..0000000 --- a/mailbox/spring/src/test/resources/events.properties +++ /dev/null @@ -1,38 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# This template file can be used as example for James Server configuration -# DO NOT USE IT AS SUCH AND ADAPT IT TO YOUR NEEDS - -# The following properties needs to be set -# if you want to use Kafka for inter node messaging for your event system. -# -# This needs to be done if you used <publisher>kafka</publisher> in event.xml - -kafka.ip=127.0.0.1 -kafka.port=9092 -zookeeper.connection.string=localhost -group.id=azerty -event.thread.count=4 - -# This property needs to be set if you want to use the Broadcast Mailbox Delegating Listener - -global.topic=JAMES_TOPIC - -distant.mailbox.path.register.max.retries=100 -distant.mailbox.path.register.refresh=900 http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/mailbox/store/src/main/java/org/apache/james/mailbox/store/publisher/Publisher.java ---------------------------------------------------------------------- diff --git a/mailbox/store/src/main/java/org/apache/james/mailbox/store/publisher/Publisher.java b/mailbox/store/src/main/java/org/apache/james/mailbox/store/publisher/Publisher.java index 56e1e0f..1ee8bf7 100644 --- a/mailbox/store/src/main/java/org/apache/james/mailbox/store/publisher/Publisher.java +++ b/mailbox/store/src/main/java/org/apache/james/mailbox/store/publisher/Publisher.java @@ -24,7 +24,7 @@ import java.io.Closeable; /** * The Publisher can be used to send information outside James. * - * For instance you can send information to a message queue like Kafka or perform a POST on a restful API + * For instance you can send information to a message queue or perform a POST on a restful API */ public interface Publisher extends Closeable { @@ -32,4 +32,4 @@ public interface Publisher extends Closeable { void init(); -} \ No newline at end of file +} http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index 702e2fa..ee24151 100644 --- a/pom.xml +++ b/pom.xml @@ -767,11 +767,6 @@ </dependency> <dependency> <groupId>${james.groupId}</groupId> - <artifactId>apache-james-mailbox-kafka</artifactId> - <version>${project.version}</version> - </dependency> - <dependency> - <groupId>${james.groupId}</groupId> <artifactId>apache-james-mailbox-lucene</artifactId> <version>${project.version}</version> </dependency> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/server/app/src/main/resources/events.properties ---------------------------------------------------------------------- diff --git a/server/app/src/main/resources/events.properties b/server/app/src/main/resources/events.properties deleted file mode 100644 index 3b49533..0000000 --- a/server/app/src/main/resources/events.properties +++ /dev/null @@ -1,42 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# This template file can be used as example for James Server configuration -# DO NOT USE IT AS SUCH AND ADAPT IT TO YOUR NEEDS - -# The following properties needs to be set -# if you want to use Kafka for inter node messaging for your event system. -# -# This needs to be done if you used <publisher>kafka</publisher> in event.xml - -kafka.ip=127.0.0.1 -kafka.port=9092 -zookeeper.connection.string=localhost -group.id=azerty -event.thread.count=4 - -# This property needs to be set if you want to use the Broadcast Mailbox Delegating Listener - -global.topic=JAMES_TOPIC - -distant.mailbox.path.register.max.retries=100 -distant.mailbox.path.register.refresh=900 - -# Worker threads count for asynchronous event delivery - -event.delivery.thread.count=10 http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/server/app/src/main/resources/events.xml ---------------------------------------------------------------------- diff --git a/server/app/src/main/resources/events.xml b/server/app/src/main/resources/events.xml index 536278a..5555280 100644 --- a/server/app/src/main/resources/events.xml +++ b/server/app/src/main/resources/events.xml @@ -36,55 +36,9 @@ --> <delivery>synchronous</delivery> - <!-- Broadcast event system will send messages to all your James servers. - - It is the only viable options if you rely on Mailbox Listeners that do not perform well in a distributed system. - - Here follows a non exhaustive list of such Mailbox Listeners : - - Memory quota based system - - Lucene based search system - - Workload is higher because event needs to be serialized and de serialized. - - Also note that depending on the publisher you use, messages might be either lost or duplicated, implying wrong - indexing or wrong quota count on network partitions. - - Here follows a sample configuration : - --> - <!-- - <type>broadcast</type> - <serialization>json</serialization> - <publisher>kafka</publisher> - --> - - <!-- - Further notes on the choice you had here : - - serialization : you have to choices : either json or message-pack - - message-pack is a binary representation for json, and is 2 times lighter by average, but is 2 time slower to serialize and de serialize. - This choice allows you to trade compute against network. - - publisher : Publish - subscribe used. The only current implementation is Kafka. - --> - - <!-- Registered event system - - Scalable event system based on registration : each James server listen on his own topic and manage registrations on - mailbox paths in order to receive events interesting him. - - This configuration does not support distributed unfriendly Mailbox Listeners (see the non exhaustive list above) for obvious reasons. - --> - <!-- - <type>registered</type> - <serialization>json</serialization> - <registration>cassandra</registration> - <publisher>kafka</publisher> - --> - <!-- The only registration system available is Cassandra, used in an available fashion. This means there is no warranties, either at least one delivery nor at most one delivery. But you get the certitude that the system will always be available. --> -</events> \ No newline at end of file +</events> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/server/app/src/test/resources/events.xml ---------------------------------------------------------------------- diff --git a/server/app/src/test/resources/events.xml b/server/app/src/test/resources/events.xml index 536278a..5555280 100644 --- a/server/app/src/test/resources/events.xml +++ b/server/app/src/test/resources/events.xml @@ -36,55 +36,9 @@ --> <delivery>synchronous</delivery> - <!-- Broadcast event system will send messages to all your James servers. - - It is the only viable options if you rely on Mailbox Listeners that do not perform well in a distributed system. - - Here follows a non exhaustive list of such Mailbox Listeners : - - Memory quota based system - - Lucene based search system - - Workload is higher because event needs to be serialized and de serialized. - - Also note that depending on the publisher you use, messages might be either lost or duplicated, implying wrong - indexing or wrong quota count on network partitions. - - Here follows a sample configuration : - --> - <!-- - <type>broadcast</type> - <serialization>json</serialization> - <publisher>kafka</publisher> - --> - - <!-- - Further notes on the choice you had here : - - serialization : you have to choices : either json or message-pack - - message-pack is a binary representation for json, and is 2 times lighter by average, but is 2 time slower to serialize and de serialize. - This choice allows you to trade compute against network. - - publisher : Publish - subscribe used. The only current implementation is Kafka. - --> - - <!-- Registered event system - - Scalable event system based on registration : each James server listen on his own topic and manage registrations on - mailbox paths in order to receive events interesting him. - - This configuration does not support distributed unfriendly Mailbox Listeners (see the non exhaustive list above) for obvious reasons. - --> - <!-- - <type>registered</type> - <serialization>json</serialization> - <registration>cassandra</registration> - <publisher>kafka</publisher> - --> - <!-- The only registration system available is Cassandra, used in an available fashion. This means there is no warranties, either at least one delivery nor at most one delivery. But you get the certitude that the system will always be available. --> -</events> \ No newline at end of file +</events> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/server/container/spring/src/main/java/org/apache/james/container/spring/bean/factorypostprocessor/EventsConfigurationBeanFactoryPostProcessor.java ---------------------------------------------------------------------- diff --git a/server/container/spring/src/main/java/org/apache/james/container/spring/bean/factorypostprocessor/EventsConfigurationBeanFactoryPostProcessor.java b/server/container/spring/src/main/java/org/apache/james/container/spring/bean/factorypostprocessor/EventsConfigurationBeanFactoryPostProcessor.java deleted file mode 100644 index 8f25443..0000000 --- a/server/container/spring/src/main/java/org/apache/james/container/spring/bean/factorypostprocessor/EventsConfigurationBeanFactoryPostProcessor.java +++ /dev/null @@ -1,120 +0,0 @@ -/**************************************************************** - * Licensed to the Apache Software Foundation (ASF) under one * - * or more contributor license agreements. See the NOTICE file * - * distributed with this work for additional information * - * regarding copyright ownership. The ASF licenses this file * - * to you under the Apache License, Version 2.0 (the * - * "License"); you may not use this file except in compliance * - * with the License. You may obtain a copy of the License at * - * * - * http://www.apache.org/licenses/LICENSE-2.0 * - * * - * Unless required by applicable law or agreed to in writing, * - * software distributed under the License is distributed on an * - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * - * KIND, either express or implied. See the License for the * - * specific language governing permissions and limitations * - * under the License. * - ****************************************************************/ - -package org.apache.james.container.spring.bean.factorypostprocessor; - -import org.apache.commons.configuration.ConfigurationException; -import org.apache.commons.configuration.HierarchicalConfiguration; -import org.apache.james.container.spring.lifecycle.ConfigurationProvider; -import org.springframework.beans.BeansException; -import org.springframework.beans.FatalBeanException; -import org.springframework.beans.factory.config.BeanFactoryPostProcessor; -import org.springframework.beans.factory.config.ConfigurableListableBeanFactory; - -import com.google.common.base.Strings; - -public class EventsConfigurationBeanFactoryPostProcessor implements BeanFactoryPostProcessor { - - @Override - public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException { - ConfigurationProvider confProvider = beanFactory.getBean(ConfigurationProvider.class); - try { - HierarchicalConfiguration config = confProvider.getConfiguration("events"); - String type = config.getString("type", "default"); - String serialization = config.getString("serialization", "json"); - String publisher = config.getString("publisher", "kafka"); - String registration = config.getString("registration", "cassandra"); - String delivery = config.getString("delivery", "synchronous"); - String delegatingListenerAlias = getDelegatingListenerAlias(type); - String serializationAlias = getSerializationAlias(serialization); - String registrationAlias = getRegistrationAlias(registration); - String deliveryAlias = getDeliveryString(delivery); - String publisherAlias = null; - String consumerAlias = null; - - if (publisher.equals("kafka")) { - publisherAlias = "kafka-publisher"; - consumerAlias = "kafka-consumer"; - } - - detectInvalidValue(delegatingListenerAlias, "Delegating listener type " + type + " not supported!"); - detectInvalidValue(deliveryAlias, "Event delivery " + delivery + " not supported"); - beanFactory.registerAlias(delegatingListenerAlias, "delegating-listener"); - beanFactory.registerAlias(deliveryAlias, "event-delivery"); - if (!delegatingListenerAlias.equals("default")) { - detectInvalidValue(serializationAlias, "Serialization system type " + serialization + " not supported!"); - detectInvalidValue(publisherAlias, "Publisher system type " + publisher + " not supported!"); - beanFactory.registerAlias(serializationAlias, "event-serializer"); - beanFactory.registerAlias(publisherAlias, "publisher"); - beanFactory.registerAlias(consumerAlias, "consumer"); - if (delegatingListenerAlias.equals("registered")) { - detectInvalidValue(registrationAlias, "Registration system type " + registration + " not supported!"); - beanFactory.registerAlias(registrationAlias, "distant-mailbox-path-register-mapper"); - } - } - - } catch (ConfigurationException e) { - throw new FatalBeanException("Unable to config the mailboxmanager", e); - } - } - - private void detectInvalidValue(String registrationAlias, String message) throws ConfigurationException { - if (Strings.isNullOrEmpty(registrationAlias)) { - throw new ConfigurationException(message); - } - } - - private String getRegistrationAlias(String registration) { - if (registration.equals("cassandra")) { - return "cassandra-mailbox-path-register-mapper"; - } - return null; - } - - private String getSerializationAlias(String serialization) { - if (serialization.equals("json")) { - return "json-event-serializer"; - } else if (serialization.equals("message-pack")) { - return "message-pack-event-serializer"; - } - return null; - } - - private String getDelegatingListenerAlias(String type) { - if (type.equals("default")) { - return "default-delegating-listener"; - } else if (type.equals("broadcast")) { - return "broadcast-delegating-listener"; - } else if (type.equals("registered")) { - return "registered-delegating-listener"; - } - return null; - } - - public String getDeliveryString(String delivery) { - if (delivery.equals("synchronous")) { - return "synchronous-event-delivery"; - } else if (delivery.equals("asynchronous")) { - return "asynchronous-event-delivery"; - } else if (delivery.equals("mixed")) { - return "mixed-event-delivery"; - } - return null; - } -} http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/server/container/spring/src/main/resources/META-INF/org/apache/james/spring-server.xml ---------------------------------------------------------------------- diff --git a/server/container/spring/src/main/resources/META-INF/org/apache/james/spring-server.xml b/server/container/spring/src/main/resources/META-INF/org/apache/james/spring-server.xml index 40f6008..297b579 100644 --- a/server/container/spring/src/main/resources/META-INF/org/apache/james/spring-server.xml +++ b/server/container/spring/src/main/resources/META-INF/org/apache/james/spring-server.xml @@ -149,7 +149,6 @@ </bean> <!-- Event system --> - <bean class="org.apache.james.container.spring.bean.factorypostprocessor.EventsConfigurationBeanFactoryPostProcessor"/> <bean id="dispatcher" class="org.apache.james.mailbox.store.event.MailboxEventDispatcher"> <constructor-arg index="0" ref="delegating-listener"/> </bean> http://git-wip-us.apache.org/repos/asf/james-project/blob/f9e478fc/src/site/xdoc/server/config-events.xml ---------------------------------------------------------------------- diff --git a/src/site/xdoc/server/config-events.xml b/src/site/xdoc/server/config-events.xml index 7bc0c48..1ad0222 100644 --- a/src/site/xdoc/server/config-events.xml +++ b/src/site/xdoc/server/config-events.xml @@ -110,34 +110,9 @@ <li>The default implementation might not deliver some events on server stop.</li> </ul> - <ul> - Broadcast implementation : - <li>The broadcast implementation might not deliver some events on server stop.</li> - <li>The broadcast implementation is tight to limitation of the underlying publisher.</li> - </ul> - - <ul> - Registered implementation : - <li>The registered implementation might not deliver some events on server stop.</li> - <li>The registered implementation is tight to limitation of the underlying publisher, and underlying registration system.</li> - </ul> - - <h2>Publisher</h2> - - <p> - Available implementation is Kafka based. Kafka ensure at least one delivery. This means some messages might be - delivered two times. You need to compile and run James using Java 8 in order to use the Kafka messaging system. - </p> - <h2>Event serializer</h2> - <p>There are two types of event serialization systems : - <ul> - <li>Json : events are converted to JSON</li> - <li>Message Pack : a binary representation of JSON. 2 times smaller in average but two times longer to compute. It allows you to trade - bandwidth and data readability against CPU time.</li> - </ul> - </p> + <p>Json is the only serialization system currently supported.</p> <h2>Registration systems</h2> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
