this sounds very interesting !
Have you got some design doc about this cluster deployment feature ?
the app have to be managed only on one TomEE in the cluster
(deploy/undeploy....) ?
in GF there is the concept of DAS, a Node in the cluster which does
configuration things and coordinates deployment on the other nodes
The admin have to configure many things....
- Resources (Datasources, Queues....) have to be configured cluster-wide
- Tomcat session replication must be configured
- OpenEJB cluster features must be configured
- Quartz configuration must be the same (except from the scheduler local
instance id) and it must use a datasource
- Embedded ActiveMQ configuration will not be trivial (need to configure
as master/slaver, use a shared datasource, client url to use MUST use
failover or discovery.....) or the useer have tto use an external activemq
- JPA issues ??
- Configuration of Reverse-Proxy/Load balancers (maybe this is a Tomcat
problem)
- Enrico
Il 20/07/2012 15:52, Romain Manni-Bucau ha scritto:
just pushed some cluster deployment feature. It still needs a lot of
improvement but i pushed it more for the idea.
1) currently it needs an app deploy through our deployer
(appinfo.autodeploy = false) -> here some work to do, the autodeploy
boolean should be true when tomcat manage the app or tomee manage it from
apps/ or a deployment defines in tomee.xml
2) assembler deploy/undeploy events are observed and then a deploy/undeploy
tomcat message cluster is sent to the cluster
3) other listeneing member do the same
it doesn't loop because we test if the app is already deployed.
currently the filesystem should be the same
here the todo we can discuss on:
1) autodeploy boolean
2) do we serialize the app to be able to deploy it even on remote hosts
3) others
any comments are welcomed of course ;)
- Romain
---------- Forwarded message ----------
From: <[email protected]>
Date: 2012/7/20
Subject: svn commit: r1363775 - in /openejb/trunk/openejb:
container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/
tomee/tomee-catalina/
tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/
tomee/tomee-catalina/src/main/java/org/apac...
To: [email protected]
Author: rmannibucau
Date: Fri Jul 20 13:47:29 2012
New Revision: 1363775
URL: http://svn.apache.org/viewvc?rev=1363775&view=rev
Log:
TOMEE-334 basic clustering deployment through tomcat cluster
Added:
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/ClusterObserver.java
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/DeployMessage.java
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/TomEEClusterListener.java
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/UndeployMessage.java
Modified:
openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/AppInfo.java
openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
openejb/trunk/openejb/tomee/tomee-catalina/pom.xml
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/TomcatWebAppBuilder.java
Modified:
openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/AppInfo.java
URL:
http://svn.apache.org/viewvc/openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/AppInfo.java?rev=1363775&r1=1363774&r2=1363775&view=diff
==============================================================================
---
openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/AppInfo.java
(original)
+++
openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/AppInfo.java
Fri Jul 20 13:47:29 2012
@@ -28,6 +28,7 @@ import java.util.TreeSet;
public class AppInfo extends InfoObject {
public String appId;
public String path;
+ public boolean autoDeploy = true;
public boolean standaloneModule;
public final List<ClientInfo> clients = new ArrayList<ClientInfo>();
public final List<EjbJarInfo> ejbJars = new ArrayList<EjbJarInfo>();
Modified:
openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
URL:
http://svn.apache.org/viewvc/openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java?rev=1363775&r1=1363774&r2=1363775&view=diff
==============================================================================
---
openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
(original)
+++
openejb/trunk/openejb/container/openejb-core/src/main/java/org/apache/openejb/assembler/classic/Assembler.java
Fri Jul 20 13:47:29 2012
@@ -121,6 +121,7 @@ import org.apache.openejb.javaagent.Agen
import org.apache.openejb.jpa.integration.MakeTxLookup;
import org.apache.openejb.loader.JarLocation;
import org.apache.openejb.loader.Options;
+import org.apache.openejb.loader.ProvisioningUtil;
import org.apache.openejb.loader.SystemInstance;
import org.apache.openejb.monitoring.DynamicMBeanWrapper;
import org.apache.openejb.assembler.monitoring.JMXContainer;
@@ -419,6 +420,10 @@ public class Assembler extends Assembler
}
}
+ public boolean isDeployed(final String path) {
+ return
deployedApplications.containsKey(ProvisioningUtil.realLocation(path));
+ }
+
public Collection<AppInfo> getDeployedApplications() {
return new ArrayList<AppInfo>(deployedApplications.values());
}
Modified: openejb/trunk/openejb/tomee/tomee-catalina/pom.xml
URL:
http://svn.apache.org/viewvc/openejb/trunk/openejb/tomee/tomee-catalina/pom.xml?rev=1363775&r1=1363774&r2=1363775&view=diff
==============================================================================
--- openejb/trunk/openejb/tomee/tomee-catalina/pom.xml (original)
+++ openejb/trunk/openejb/tomee/tomee-catalina/pom.xml Fri Jul 20 13:47:29
2012
@@ -86,6 +86,12 @@
<version>${tomcat.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.tomcat</groupId>
+ <artifactId>tomcat-catalina-ha</artifactId>
+ <version>${tomcat.version}</version>
+ <scope>provided</scope>
+ </dependency>
</dependencies>
</project>
Modified:
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/TomcatWebAppBuilder.java
URL:
http://svn.apache.org/viewvc/openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/TomcatWebAppBuilder.java?rev=1363775&r1=1363774&r2=1363775&view=diff
==============================================================================
---
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/TomcatWebAppBuilder.java
(original)
+++
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/TomcatWebAppBuilder.java
Fri Jul 20 13:47:29 2012
@@ -16,9 +16,7 @@
*/
package org.apache.tomee.catalina;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.List;
+import org.apache.catalina.Cluster;
import org.apache.catalina.Container;
import org.apache.catalina.Engine;
import org.apache.catalina.Host;
@@ -43,19 +41,31 @@ import org.apache.catalina.deploy.Contex
import org.apache.catalina.deploy.ContextResourceLink;
import org.apache.catalina.deploy.ContextTransaction;
import org.apache.catalina.deploy.NamingResources;
+import org.apache.catalina.ha.CatalinaCluster;
import org.apache.catalina.loader.WebappClassLoader;
import org.apache.catalina.loader.WebappLoader;
import org.apache.catalina.startup.Constants;
import org.apache.catalina.startup.ContextConfig;
import org.apache.catalina.startup.HostConfig;
import org.apache.catalina.startup.RealmRuleSet;
+import org.apache.catalina.tribes.Member;
import org.apache.naming.ContextAccessController;
import org.apache.naming.ContextBindings;
import org.apache.openejb.AppContext;
import org.apache.openejb.BeanContext;
import org.apache.openejb.Injection;
import org.apache.openejb.OpenEJBException;
-import org.apache.openejb.assembler.classic.*;
+import org.apache.openejb.assembler.classic.AppInfo;
+import org.apache.openejb.assembler.classic.Assembler;
+import org.apache.openejb.assembler.classic.ClassListInfo;
+import org.apache.openejb.assembler.classic.ConnectorInfo;
+import org.apache.openejb.assembler.classic.DeploymentExceptionManager;
+import org.apache.openejb.assembler.classic.EjbJarInfo;
+import org.apache.openejb.assembler.classic.InjectionBuilder;
+import org.apache.openejb.assembler.classic.JndiEncBuilder;
+import org.apache.openejb.assembler.classic.ServletInfo;
+import org.apache.openejb.assembler.classic.WebAppBuilder;
+import org.apache.openejb.assembler.classic.WebAppInfo;
import org.apache.openejb.cdi.CdiBuilder;
import org.apache.openejb.config.AppModule;
import org.apache.openejb.config.ConfigurationFactory;
@@ -74,6 +84,8 @@ import org.apache.openejb.util.LogCatego
import org.apache.openejb.util.Logger;
import org.apache.tomcat.InstanceManager;
import org.apache.tomcat.util.digester.Digester;
+import org.apache.tomee.catalina.cluster.ClusterObserver;
+import org.apache.tomee.catalina.cluster.TomEEClusterListener;
import org.apache.tomee.catalina.event.AfterApplicationCreated;
import org.apache.tomee.common.LegacyAnnotationProcessor;
import org.apache.tomee.common.TomcatVersion;
@@ -99,9 +111,12 @@ import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Field;
+import java.util.ArrayList;
+import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
+import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
@@ -178,6 +193,8 @@ public class TomcatWebAppBuilder impleme
private String defaultHost = "localhost";
+ private Set<CatalinaCluster> clusters = new HashSet<CatalinaCluster>();
+
/**
* Creates a new web application builder
* instance.
@@ -195,11 +212,13 @@ public class TomcatWebAppBuilder impleme
for (final Service service : standardServer.findServices()) {
if (service.getContainer() instanceof Engine) {
final Engine engine = (Engine) service.getContainer();
+ manageCluster(engine.getCluster());
defaultHost = engine.getDefaultHost();
addTomEERealm(engine);
for (final Container engineChild : engine.findChildren()) {
if (engineChild instanceof StandardHost) {
final StandardHost host = (StandardHost)
engineChild;
+ manageCluster(host.getCluster());
addTomEERealm(host);
hosts.put(host.getName(), host);
for (final LifecycleListener listener :
host.findLifecycleListeners()) {
@@ -213,10 +232,24 @@ public class TomcatWebAppBuilder impleme
}
}
+ SystemInstance.get().addObserver(new ClusterObserver(clusters));
+
configurationFactory = new ConfigurationFactory();
deploymentLoader = new DeploymentLoader();
}
+ private void manageCluster(final Cluster cluster) {
+ if (cluster == null) {
+ return;
+ }
+
+ if (cluster instanceof CatalinaCluster) {
+ final CatalinaCluster haCluster = (CatalinaCluster) cluster;
+ haCluster.addClusterListener(new TomEEClusterListener());
+ clusters.add(haCluster);
+ }
+ }
+
private void addTomEERealm(final Engine engine) {
final Realm realm = engine.getRealm();
if (realm != null && !(realm instanceof TomEERealm)
@@ -388,6 +421,7 @@ public class TomcatWebAppBuilder impleme
// TODO: instead of storing deployers, we could just
lookup the right hostconfig for the server.
final HostConfig deployer = deployers.get(host);
+ appInfo.autoDeploy = false;
if (isReady(deployer)) { // if not ready using directly
host to avoid a NPE
// host isn't set until we call deployer.manageApp, so
pass it
// ?? host is set through an event and it can be null
here :(
Added:
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/ClusterObserver.java
URL:
http://svn.apache.org/viewvc/openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/ClusterObserver.java?rev=1363775&view=auto
==============================================================================
---
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/ClusterObserver.java
(added)
+++
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/ClusterObserver.java
Fri Jul 20 13:47:29 2012
@@ -0,0 +1,38 @@
+package org.apache.tomee.catalina.cluster;
+
+import org.apache.catalina.ha.CatalinaCluster;
+import org.apache.catalina.ha.ClusterMessage;
+import org.apache.openejb.assembler.classic.AppInfo;
+import
org.apache.openejb.assembler.classic.event.AssemblerAfterApplicationCreated;
+import
org.apache.openejb.assembler.classic.event.AssemblerBeforeApplicationDestroyed;
+import org.apache.openejb.observer.Observes;
+
+import java.io.File;
+import java.util.Set;
+
+public class ClusterObserver {
+ private final Set<CatalinaCluster> clusters;
+
+ public ClusterObserver(final Set<CatalinaCluster> clusters) {
+ this.clusters = clusters;
+ }
+
+ public void deploy(@Observes final AssemblerAfterApplicationCreated
app) {
+ final AppInfo appInfo = app.getApp();
+ send(new UndeployMessage(appInfo.path), appInfo);
+ }
+
+ public void undeploy(@Observes final
AssemblerBeforeApplicationDestroyed app) {
+ final AppInfo appInfo = app.getApp();
+ send(new DeployMessage(appInfo.path), appInfo);
+ }
+
+ private void send(final ClusterMessage message, final AppInfo app) {
+ for (CatalinaCluster cluster : clusters) {
+ final String path = app.path;
+ if (new File(path).exists() && !app.autoDeploy) {
+ cluster.send(message);
+ }
+ }
+ }
+}
Added:
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/DeployMessage.java
URL:
http://svn.apache.org/viewvc/openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/DeployMessage.java?rev=1363775&view=auto
==============================================================================
---
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/DeployMessage.java
(added)
+++
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/DeployMessage.java
Fri Jul 20 13:47:29 2012
@@ -0,0 +1,16 @@
+package org.apache.tomee.catalina.cluster;
+
+import org.apache.catalina.ha.ClusterMessageBase;
+
+// TODO: serialize file in byte[] to be able to send it over the network?
+public class DeployMessage extends ClusterMessageBase {
+ private String file;
+
+ public DeployMessage(final String path) {
+ file = path;
+ }
+
+ public String getFile() {
+ return file;
+ }
+}
Added:
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/TomEEClusterListener.java
URL:
http://svn.apache.org/viewvc/openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/TomEEClusterListener.java?rev=1363775&view=auto
==============================================================================
---
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/TomEEClusterListener.java
(added)
+++
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/TomEEClusterListener.java
Fri Jul 20 13:47:29 2012
@@ -0,0 +1,77 @@
+package org.apache.tomee.catalina.cluster;
+
+import org.apache.catalina.ha.ClusterListener;
+import org.apache.catalina.ha.ClusterMessage;
+import org.apache.openejb.NoSuchApplicationException;
+import org.apache.openejb.OpenEJBException;
+import org.apache.openejb.UndeployException;
+import org.apache.openejb.assembler.Deployer;
+import org.apache.openejb.assembler.classic.Assembler;
+import org.apache.openejb.core.LocalInitialContextFactory;
+import org.apache.openejb.loader.SystemInstance;
+import org.apache.openejb.util.LogCategory;
+import org.apache.openejb.util.Logger;
+
+import javax.naming.Context;
+import javax.naming.InitialContext;
+import javax.naming.NamingException;
+import java.io.File;
+import java.util.Properties;
+
+public class TomEEClusterListener extends ClusterListener {
+ private static final Logger LOGGER =
Logger.getInstance(LogCategory.OPENEJB, TomEEClusterListener.class);
+ private static final Properties IC_PROPS = new Properties();
+
+ static {
+ IC_PROPS.setProperty(Context.INITIAL_CONTEXT_FACTORY,
LocalInitialContextFactory.class.getName());
+ }
+
+ @Override
+ public void messageReceived(final ClusterMessage clusterMessage) {
+ final Class<?> type = clusterMessage.getClass();
+
+ if (DeployMessage.class.equals(type)) {
+ final DeployMessage msg = (DeployMessage) clusterMessage;
+ final String file = msg.getFile();
+ final boolean alreadyDeployed =
SystemInstance.get().getComponent(Assembler.class).isDeployed(file);
+ final File ioFile = new File(file);
+ if (ioFile.exists() && !alreadyDeployed) {
+ try {
+ deployer().deploy(file);
+ } catch (OpenEJBException e) {
+ LOGGER.warning("can't deploy: " + ioFile.getPath(), e);
+ } catch (NamingException e) {
+ LOGGER.warning("can't find deployer", e);
+ }
+ } else if (!alreadyDeployed) {
+ LOGGER.warning("file is remote, can't deploy it: " +
ioFile.getPath());
+ } else {
+ LOGGER.info("application already deployed: " + file);
+ }
+ } else if (UndeployMessage.class.equals(type)) {
+ final String file = ((UndeployMessage)
clusterMessage).getFile();
+ if
(SystemInstance.get().getComponent(Assembler.class).isDeployed(file)) {
+ try {
+ deployer().undeploy(file);
+ } catch (UndeployException e) {
+ LOGGER.error("can't undeploy app", e);
+ } catch (NoSuchApplicationException e) {
+ LOGGER.warning("no app toi deploy", e);
+ } catch (NamingException e) {
+ LOGGER.warning("can't find deployer", e);
+ }
+ }
+ } else {
+ LOGGER.warning("message type not supported: " + type);
+ }
+ }
+
+ private Deployer deployer() throws NamingException {
+ return (Deployer) new
InitialContext(IC_PROPS).lookup("openejb/DeployerBusinessRemote");
+ }
+
+ @Override
+ public boolean accept(final ClusterMessage clusterMessage) {
+ return DeployMessage.class.equals(clusterMessage.getClass()) ||
UndeployMessage.class.equals(clusterMessage.getClass());
+ }
+}
Added:
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/UndeployMessage.java
URL:
http://svn.apache.org/viewvc/openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/UndeployMessage.java?rev=1363775&view=auto
==============================================================================
---
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/UndeployMessage.java
(added)
+++
openejb/trunk/openejb/tomee/tomee-catalina/src/main/java/org/apache/tomee/catalina/cluster/UndeployMessage.java
Fri Jul 20 13:47:29 2012
@@ -0,0 +1,15 @@
+package org.apache.tomee.catalina.cluster;
+
+import org.apache.catalina.ha.ClusterMessageBase;
+
+public class UndeployMessage extends ClusterMessageBase {
+ private String file;
+
+ public UndeployMessage(final String path) {
+ file = path;
+ }
+
+ public String getFile() {
+ return file;
+ }
+}
