-----Original Message-----
From: Stacy Curl
Sent: 06 April 2001 16:14
To: 'marc fleury'
Subject: RE: [JBoss-dev] JMX in jBoss, Clustering, Monitoring, Failover,
better bootstrapp ing via pluggable XMLet
> What is the mechanism for proxying? RMI? What do you use to connect to the
> distant JMX instances?
I'm using RMI for proxying, I connect using RMIConnector which is obtained
from JNDI.
> So you ship the MAIL module as a critical module of each agent?
All I'm using for email is JavaMail + an external SMTP server.
> I would publish on a Queue frankly. It is not your domain to deliver the
> view to the management. From a managmeent standpoint I say put a
> topic/queue register the listeners to that. Do you see that? forget
pagers,
> talk event and messages... wrong brain function...
Ok, I think I'd agree with you here, filtering rules for support events can
be done in the email system. I would like to point out that the Queue needs
to
be quite low-tech, these events would could be sent in the event that
serious
things have gone wrong, so JMS would be out, simple email seems to be fine.
> What do you use for that? the JDMK classes from SUN? we can't really
> distribute them so either we rewrite some of these or we really on bare
JMX.
No, already re-wrote them, they really sucked. I can't believe that code
from Sun
could be so anti-OO. I couldn't find a single sweet spot to plug in any
code, totally
non-extensible.
I'm still digesting what you said about installation, we would need to
consider versioning
later. Normal MLet behaviour is to drag classes out of jars and instantiate
them, wouldn't
it be better to download the jars, store them with the 'SPINE' then proceed
to boot as normal.
Decoupled. Time stamps and version checks could give an auto-update
facility. But perhaps MLet loading
over the net can be cached via some other means. MLet MBean extended
URLClassLoader, so does XMLet MBean,
could just go further and chuck in cacheing.
I've looked at some cluster JVM projects (cJVM, Microsofts Borg) and have
been amazed. To implement a SSI at the JMX level
requires much less work than those gurus did but still a lot. Once an MBean
is located in an Agent moving it to another would require significant work
so it's probably better to keep MBeans at a fixed location. Ok, so since the
MBeans are fixed it becomes a question of how to distribute the MBeans... Ah
got it, Each time the server runs it could gather information on resource
use and mbean-mbean interaction, the next time the server runs it would
optimise locality of interactions and balance resources. I think that
generic code to migrate them could be written and it's bloody interesting
just not necessary now, when the generic code is done the balancing and
optimisation of the JBoss Farm would be near-realtime.
I don't know much about .jcml. I think it's an interesting point that you
mention that it's not the final state of JBoss that gets persisted to .jcml
but the transitions from the initial state to the final state. This of
course could not be applied to all transitions, this is like recording and
playing back, so a simple GUI would be video controls. When the Agent is in
Record mode it stores the transitions, when in normal mode it only stores
the end state. The system would actually always record the transitions, it
would need to do this to be able to show an admin the changes, but it would
only use the transitions when the 'record' button was selected.
I'm starting to think that this stuff is too complex to do in java mbean
code, better to use JSP + RMI queries to Agents. Currently I'm having to
juggle several Agents each with it's own HTTPConnectors, not nice; putting
all this into a single view via JSP would be better.
I'll get back to you when I've had a look at JSR77.
Stacy.
P.S. Thanks for trusting me, I went quiet for a while because the focus of
my work isn't centered around JMX so much at the moment, I'm going to have
to continue a lot of development from home, this of course slows things down
a bit.
-----Original Message-----
From: marc fleury [mailto:[EMAIL PROTECTED]]
Sent: 06 April 2001 15:18
To: Stacy Curl; [EMAIL PROTECTED]; 'Andreas
Schaefer'
Subject: RE: [JBoss-dev] JMX in jBoss, Clustering, Monitoring, Failover,
better bootstrapp ing via pluggable XMLet
|Hello, I'd like to tell you about the code that I've developed and
|contributed to JBoss.
Thank you stacy, when we first talked over the phone 3 month ago, I thought
you were not for real, you have proven me wrong and I am glad I trusted you
with a RW passwd without prior work.
Comments below,
|I've developed some code to augment the basic capabilities that JMX gives
|you. I've written an MBean that will 'mirror' MBeans, that is it
|will take a
|subset of MBeans on one JMX agent and create proxies for them on
|another JMX
|agent (the idea of proxying is wide place but the inspiration for this
|instance was taken from the CascadingAgent from JDMK).
What is the mechanism for proxying? RMI? What do you use to connect to the
distant JMX instances?
|Secondly I've created a 'Watchdog' MBean that monitors the health of
|'Watchable' (*) MBeans and calls restart on the watchable MBeans when they
|have failed. I've also wrapped the JMX agent in an RMI Activatable
|object so
|that the whole JMX agent itself can be restarted (also by the Watchdog) in
|the event that JVM dies.
|(*) Actually they are called 'startable' and should be renamed to
|'watchable'.
|I've also rewritten the MLet loading class as a pluggable XMLet loading
|class, using this method I've been able to specify in the XMLet resource
|file which MBeans are critical (failure to start means the JMX agent is
|useless) to optional (failure to start leaves the JMX agent still
|functional).
Interesting
|The stuff I wrote can be found at
|http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/jbossmx/?cvsroot=jboss
Oh yeah I need to make sure that there is no confusion on the website about
JBossMX and JBossMail. JBossMX are our JMX related efforts.
|code in JBoss. This also means that I'll be adding more features
|to JBoss in
|the future and getting paid for it :). Of course this point is
|moot if those
|features are not interesting to the JBoss group.
This is good news and the feature is of course very interesting. (don't be
silly of course we want that feature)
|Since JBoss is based on JMX and will in the near future have a
|number of JMX
|Agents to enable clustering of the EJB Containers then monitoring of these
|Agents and containers seem crucial, there is much work to be done but I
|think that my code should serve as a reasonable base.
|
|Ideas for the future:
|- Have multiple JMX Agent (got), but specify a single XMLet resource file,
|let the MBeans be distributed evenly across the Agents, this is called a
|Single System Image (SSI JMX)
Yes this is very important, I believe we need to be able to launch n VM/JMX
instance and automatically distribute the XMLet resource to them. In fact I
would want to see the following done soon:
put a HTTP repository somewhere with the JBoss codebase,
Use the SSI infrastructure/activation to start all the JMX instances in a
farm of machines.
Call the XMLet loaders on each passing the XMLet file that contains
URL="http://jbosswebserver(even www.jboss.org if you wanted)/latest
codebase/jboss.jar" and starts loading all the machines with the latest
software image FROM THE WEBSERVER.
You are very close to home with the SSI facility, I have been thinking about
it for awhile in the context of JSR77 (management). More coming but
basically you manage a single instance and REMOTELY INSTALL JBOSS through
HTTP and the MLET mechanism... the management and distribution is missing.
In short use http to load 10 machines with jboss, all that resides on the
target machines is the SPINE
SPINE = base JMX + watchdog wrappers. IT NEEDS TO FIT ON A DISKETTE.
SO that an administrator can go out and install the SPINE on an empty
machine, bring it on and see it load its software from the network like an
applet and teh JavaOS (remember?) did. THAT was really cool and I believe it
will really burn in the server side of things. Version management with
local copies would EVEN BE COOLER (like a flashRAM/marimba delta kind of
management)
|- Offer different styles of monitoring, currently it's poll based,
|could use
|a 'dead-man's handle', or pure event based monitoring, combinations also
|possible.
|- Are there parallels between JMX and RMI, there is an RMIRegistry, does it
|make sense to have a JMXRegistry and access the MBeans through URL's ?
|jmx://machineName:agentName/domainName/objectName or
|jmx://agentName/domainName/objectName, or in conjunction with the SSI JMX:
|jmx://domainName/objectName
Yes, ok you are looking into the research field I am playing with in JSR77.
The short answer is there are many parallels between *what you want to do
with JMX* (NOT JMX!) and the naming service of RMI. Mad andy, that you
rightfully put in copy did develop an RMI connector that fulfill some of the
naming exposure. If you ask me this needs to rest in a naming repository a
la jndi, ie. standardized. What you implement behind the proxy is the
second million dollar question.
Right now, this is what I am thinking about these days (contrary to some
WebLogic folks) that we should have higher functions in the bare brain. In
short that when an instance comes up it bring a "admin EJB container". That
would be the simplest way to offer some of that management online. One
thing is trivial to see if you think about it, a lot of the functionality
you want to offer is really remoteness and naming consistency so that you
offer navigation on your domain (SSI is the management image, you need node
navigation within it). It is VERY close to what I am developing for the
JSR77 specification. There are still some areas in my mind that are not
fully researched, mainly revolving around the idea of "critical-ness" of the
service in the SPINE. If you ask me, the "brain" needs a minimum
infrastructure to manage the SPINE (human parallel intended) that is a
function of the brain that comes on. In other words, we might need a simple
container to offer the programmatic view and I am not sure that "bare" RMI
is the simplest way to manage the cluster. Don't know. It could be a mix
of JMS and EJB... still thinking.
I will tell you what is driving my thinking. From a usability standpoint
the registry is the key to unified management. If you think about it,
windows PC' ease of use come from teh central registration of module that
one can find by hierarchical name and PROGRAMMATIC approach adn semantics.
uber powerful because at the nexus between programs and admin. With 3 month
of research with the 77ers I am now convinced that this registry is the key,
but probably in JNDI as opposed to bare RMI... or JMX for that matter. JMX
is not the way ... it is the infrastructure, the agent, not the management
view (WHAT THE FUCK was Simon viennot thinking when he developed that ugly
semantic for the invocation... man ok you detype your back but type your
front at LEAST with XML semantics... JUST LIKE WE REDID IT IN JBOSS for a
french men I am not proud of him)
|- Current system offers emails when corrective action is taken on a failed
|MBean, future system could monitor performance and email on crossing
|critical boundaries (90% maximum I/O, memory, etc).
So you ship the MAIL module as a critical module of each agent?
|- Can only send emails to support staff at present, other possibilities may
|be useful such as sending pagers.
I would publish on a Queue frankly. It is not your domain to deliver the
view to the management. From a managmeent standpoint I say put a
topic/queue register the listeners to that. Do you see that? forget pagers,
talk event and messages... wrong brain function...
Decouple, think local, mark happner showed the local way... JMX is not
federated for that very reason (imho)
|- Can only see (in the HTML view) the current state of the system, it would
|be useful to be able to run through a movie of previous states and also
|highlight differences.
Ok that is a VERY interesting point. The fact is that jboss-auto.jcml sucks
donkey dicks (and does a poor job at it). I believe that the management
operations need to be persisted in a transactional way. Here is where a
bare ejb container would be interesting. I would then code teh persistence
engine for that bare bean to be file based with XML parser... all of the
sudden you are way more heavy but it might still be worth it. Either that
or we serialize the different version of the bean (still withthe custom
persistence engine) and we generate the tool/homme views when we have very
high brain functions, such as an XML parser (big puppy)). Do you see that?
that is absolutely RED HOT. IT is super cool, the persistent scenario and
recovery will make us the darling of every serious admin shop. This is the
modern GUI or AUI (Admin UI)
The versioned serialization won't work with JAWS (and I don't want a
database in the basic brain) and the bare serialization is the one that is
RAZOR sharp in terms of the space and speed. That would really kick ass.
Auditing can also be done with 3 party tools.
The final interest here, that I believe is missing COMPLETELY even from
JSR77 is the transactional nature of some management operations. Imagine
that our "persistence engine" is really a JMX invoker that says, "you want
persistence? SURE it is the STATE of the MBean that is your persistence, the
serialized or jcml file being mere static mirrors of it" THEN you are
building an infrastructure which operations are 2PC and can synchronize
through transactions. Advanced brain function still deep management, you
configure the body from the spine up with transactional units... all or
nothing, makes me weepy it is so pretty.
|- The HTML view is quite minimal, I've created a pluggable version which
|enables MBeans to specify somewhat their display, this can be improved
|significantly which will aide support staff.
What do you use for that? the JDMK classes from SUN? we can't really
distribute them so either we rewrite some of these or we really on bare JMX.
good work,
I am still fuzzy on how does the infrastructure relate to the application
side and I now believe they are a bit orthogonal.
marc
_______________________________________
"In myself, I will dance, I will laugh,
I will communicate, I will cooperate,
I will participate, I will motivate,
In myself... "
-- The pledge, DJ icey --
_______________________________________
|--- Brain Dump ENDS ------
|
|Regards,
|Stacy.
|
|_______________________________________________
|Jboss-development mailing list
|[EMAIL PROTECTED]
|http://lists.sourceforge.net/lists/listinfo/jboss-development
_______________________________________________
Jboss-development mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/jboss-development
