On Thursday 01 November 2001 16:05, Chuck Esterbrook wrote:
> At 03:54 PM 11/1/2001 -0800, Tavis Rudd wrote:
> > > - a Webware component is simply a Python package with some
> > > additional conventions
> >
> >sure, but we shouldn't stipulate that the docs must be written in
> > the same form as Webware's (HTML). I'm thinking from the
> > perspective of Cheetah's docs here.
>
> I would like to stipulate that after a standard installation, a
> Webware component will have a Docs/index.html in the current style,
> which is actually generated from the Properties.py of the
> component.
Thanks sounds sensible, but I still think the component should be
required to provide the docs rather requiring Webware to know where
to find the components docs.
My argument isn't to do with what's provided, but rather who is
responsible for providing it. It would make more sense to me for the
component to install some docs to a central Webware location when its
setup.py is run. That way no assumptions are needed about the
structure of the component. This is essentially what goes on with
the docs in Python standard library. When a module or package is
added to the Python library all relevant documentation is added to
the Python-x.x/Doc directory. The onus is on the component
to play nicely with Webware, rather than on Webware to play nicely
with the component.
> Whether the manual(s) are generated to or written in HTML, I don't
> really care. What I care is that we can bind all these docs
> together so they are exceptionally easy to surf.
A valid goal!
> > > - a WebKit plug-in is simply a Webware component loaded at
> > > launch time - if a WebKit plug-in needs to so some special set
> > > up, it does so via __init__.py like any good Python package
> > > does
> >
> >But the loading logic should be handled by the launcher script
> > rather than the AppServer itself. This is what I was getting at.
>
> I didn't know that was your view. You said something about how
> plug-ins needed to be classes (they are currently packages).
Oh, I was definitely thinking of each component being structured as a
Python package. The rest was just thinking aloud:
* make AppServer and Application a little more generic by stripping
all the 'launch' behaviour out of them and putting it in the
launcher script --- exactly what I've done in Launcher.py in the
redesign code. AppServer and Application should only be responsible
for handling requests, etc. the loading of config settings, plugins,
etc. should all be handled by the launcher.
* the launcher should control where components are loaded from. To
facilitate this I'd make a Plugin class that components would
subclass. The constructor would accept the AppServer instance as an
arg and do whatever it wants with it. The component's plugin
subclass would be loaded by the launcher. The launcher would expect
the components to be in a particular place and have particular
structure, BUT the AppServer wouldn't. This makes it easy to
customize the launch procedure.
At the moment AppServer is creating an instance of PlugIn from the
path of the component. This is backwards! The plugin should be in
control, not the AppServer. Speaking in terms of a sentence, the
AppServer is the object, not the subject.
Launcher -->> Plugin -->> AppServer
* the launcher script should also handle the monitoring behaviour and
launching a non-persistent version of the AppServer. See the
attached module.
> The Launcher.py script is a lightweight bootstrapper that solves
> some of the subtle issues of Python imports, paths, etc. I'd
> prefer to keep it lightweight (just 60 lines) and continue to focus
> on AppServer and Application as we always have in the past.
This seems to be at odds with your philosophy of not making WebKit
monolithic, with which I agree. As Terrel has written before,
AppServer and Application are already too large for their own good.
Some refactory feels necessary here. We've adopted the unit testing
and wiki aspects of ExtremeProgramming so why not the 'aggresive
refactoring'? ;-)
At the moment AppServer and Application are doing things that they
shouldn't be responsible for. Here's how I see WebKit working:
Launcher -- is responsible for all startup stuff. It reads the
config files then loads the various pieces of the
system with the settings from those files and puts
them together. pieces = Dispatcher, AppServer,
Application(s), Plugins, etc.
---------------------------------------------------------------------------------------------------
Dispatcher -- listens to a set of ports and dispatches incoming
connections to the appropriate services, such as
the AppServer (aka MultiPortServer)
AppServer -- turns the incoming stream into a rawRequest and
dispatches the request to the appropriate
Application
Application(s) (Transaction, Request, Response,
Session, SessionStore, Servlet,
Cookie, etc.) -- handles the request
I'll post this on the Wiki as well for more comments.
> See more below.
>
> > > - It's "from SomeKit import SomeModule" and will not be
> > > "from Webware.SomeKit import SomModule"
> >
> >What do other people think about this? My dislike of it
> > primarilly stems from 'MiscUtils' and 'WebUtils'. If the are
> > going to be top-level packages they should have names that
> > clearly identify them as part of Webware.
>
> What clearly identifies MiddleKit and UserKit as being part of
> Webware any more than *Utils?
Nothing really, but they're unique names not likely to be used
anywhere else.
> > > I think that:
> > >
> > > - Webware programs like Generate.py and Launch.py need to use
> > > their own Webware components (I know this from experience). The
> > > Webware/ umbrella directory facilitates this.
> >
> >I can see your argument there, but how are externally developed
> >packages such as Cheetah to fit in with this model?
>
> I would think Webware Deluxe would put Webware components under
> Webware/ and put non-components like MySQLdb in their typical
> place.
More thought is needed on this as the setup.py of some components,
such as Cheetah, must be called. Just plopping them into a Webware/
dir won't cut it.
> > > - more discussion as to the virtues of TaskKit
> > > ^ I suggest these become discussion threads in their own right.
> >
> >If people in Geoff's company are using it maybe the discussion
> > should be about whether it should be used for SessionStore. My
> > argument isn't that it's not usefull, but that it's not
> > appropriate for SessionStore.
>
> So now for the next step:
> Why is it not useful and not appropriate?
I'm not making any statement about whether it's useful, because I
haven't used it. It's not a appropriate for SessionStore because it
makes that piece of logic more complex rather than simplifying it,
and it's slower than just using a session manager thread that is
owned by the SessionStore itself.
> > > You already have:
> > >
> > > - the ability to dictate what plug-ins are loaded or not loaded
> > > via WebKit/Configs/AppServer.config
> >
> >But the loading procedure is handled by the AppServer and assumes
> > a particular directory structure. My argument is that the
> > launcher script is a more natural place to load the plugins from.
>
> The directory locations for plug-ins are NOT assumed. They are read
> from AppServer.config. It's true that the out-of-the-box config
> looks in Webware/, but then what did you expect?
>
> If you want the plug-ins from somewhere else, tweak
> AppServer.config and you're done. I can't make it any easier than
> that.
But, don't they all have to come from single dir? What happens when
you have plugins coming from several physical locations?
Tavis
#!/usr/bin/env python
# $Id: Launcher.py,v 1.6 2001/10/26 19:10:31 tavis Exp $
"""Provides a Launcher class for starting Webware.
"""
__author__ = 'The Webware Development Team'
__version__ = "$Revision: 1.6 $"[11:-2]
##################################################
## DEPENDENCIES ##
import sys
import signal
import getopt
import os.path
import time
from types import DictType
# intra-package imports ...
import Webware # for path calculations when loading 'compatibilityMode'
from Version import version
from Utilities import mergeNestedDictionaries
from SettingsManager import SettingsManager
from MultiPortServer import MultiPortServer
from AppServer import AppServer
from AdminServer import AdminServer
from Monitor import Monitor
from Application import Application
from HTTPServer import HTTPServer
##################################################
## GLOBALS & CONSTANTS ##
True = (1==1)
False = (1==0)
##################################################
## CLASSES ##
class Error(Exception):
pass
class Launcher(SettingsManager):
_workingDir = False
"""A class for launching (i.e. running) an installation of Webware.
This class provides its own command line interface via the
runFromCommandLine() method, but it can also be used manually by other
interfaces such as the webware_win32_service.
The general startup sequence is as follows:
1) get the 'path' of the config file, or set it to None if you want to search for one
in the default locations
2) self.loadSettings(path)
3) self.processSettings()
4) self.loadServer()
5) self.startServer()
"""
def __init__(self):
"""Prepare to accept requests from the ServiceController."""
SettingsManager.__init__(self)
self._initializeSettings()
def _initializeSettings(self):
self._settings = {
'ApplicationClass': Application,
'MultiPortServerClass': MultiPortServer,
'AppServerClass':AppServer,
'MonitorClass':Monitor,
'HTTPServerClass':HTTPServer,
'monitorPIDFile':'monitor.pid',
'appServerServiceName':'Webware App Server',
}
def loadSettings(self, path=None):
"""Start the server using 'path' as the path of the config file. If path
is None, it will search for a config file in the default locations."""
if not path:
if os.path.exists('.webware_config.py'):
path = '.webware_config.py'
elif os.path.exists('.webware_config'):
path = '.webware_config'
elif os.path.exists(
os.path.join(os.path.expanduser('~'), '.webware_config.py')):
path = os.path.join(os.path.expanduser('~'), '.webware_config.py')
elif os.path.exists(os.path.join(os.path.expanduser('~'), '.webware_config')):
path = os.path.join(os.path.expanduser('~'), '.webware_config')
else:
self.usage()
raise Error("No config file was specified " +
"and none could be found in the default locations")
self._theSettings = self._settingsFromConfigFile(path)
return self._theSettings
def _settingsFromConfigFile(self, fileName):
"""Collect settings from the specified file. If the fileName extension
is .py the file will be parsed for settings in Python syntax, otherwise
it will be parsed for ConfigParser/ini syntax."""
fileName = self.normalizePath(fileName)
if fileName.endswith('py'):
theSettings = self.readFromPySrcFile(fileName)
else:
fp = open(fileName)
theSettings = self.readFromConfigFileObj(fp)
fp.close()
return theSettings
def processSettings(self, settings=None):
"""Extract the various settings dictionaries out of the config file:
self._MultiPortServerSettings
self._AdminServerSettings
self._MonitorServiceSettings
self._ErrorHandling
self._AppDefaults
self._AppServerSettings
self._AppSettings
"""
if not settings:
settings = self._theSettings
multiPortServerSettings = {}
adminServerSettings = {}
monitorServiceSettings = {}
appServerSettings = {}
errorHandling = {}
appDefaults = {}
appSettings = {}
HTTPServerSettings = {}
for key, val in settings.items():
if type(val) == DictType:
if key == 'MultiPortServer':
multiPortServerSettings = val
elif key == 'AdminServer':
adminServerSettings = val
elif key == 'MonitorService':
monitorServiceSettings = val
elif key == 'ErrorHandling':
errorHandling = val
elif key == 'AppServer':
appServerSettings = val
elif key == 'HTTPServer':
HTTPServerSettings = val
elif key == 'ApplicationDefaults':
appDefaults = val
else:
appSettings[key] = val
appDefaults = mergeNestedDictionaries(errorHandling, appDefaults)
for key, val in appSettings.items():
appSettings[key] = mergeNestedDictionaries(appDefaults, val)
self._AdminServerSettings = adminServerSettings
self._MonitorServiceSettings = monitorServiceSettings
self._AppSettings = appSettings
self._ErrorHandling = errorHandling
self._AppDefaults = appDefaults
self._HTTPServerSettings = HTTPServerSettings
self._AppServerSettings = mergeNestedDictionaries(errorHandling, appServerSettings)
if multiPortServerSettings:
self._MultiPortServerSettings = multiPortServerSettings
else:
self._MultiPortServerSettings = appServerSettings
if self._MonitorServiceSettings.get('on',False):
self.MONITOR = True
def loadServer(self):
"""Create the Application objects, the AppServer object and the
MultiPortServer object. Register the Applications with the AppServer,
and the AppServer with the MultiPortServer. Store the MultiPortServer
in self.multiPortServer and return a ref to it.
This method can only be called after self.processSettings."""
appServerSettings = self._AppServerSettings
## create the multiPortServer and the appServer
multiPortServer = self.setting('MultiPortServerClass')(
settings=self._MultiPortServerSettings)
appServer = self.setting('AppServerClass')(
settings=self._AppServerSettings)
## load the backwards compatibility package
if appServer.setting('compatibilityMode'):
compatibilityModePath = os.path.join(
os.path.dirname(Webware.__file__),
'compatibilityMode')
sys.path.insert(0,compatibilityModePath)
## register the appServer
appServerDispatcher =multiPortServer.bindService(
appServer,
serviceName=self.setting('appServerServiceName'),
address= (appServer.setting('hostName','localhost'),
appServer.setting('port',8086)),
settings=appServer.settings(),
)
## start and register the AdminServer if required
if self.MONITOR or \
self._AdminServerSettings.get('on', False):
adminSettings = self._AdminServerSettings
adminServer = AdminServer(multiPortServer, appServer,
appServerDispatcher,
settings=adminSettings)
multiPortServer.bindService(
adminServer,
serviceName="AdminServer",
address= (adminServer.setting('hostName'),
adminServer.setting('port')),
settings=adminServer.settings(),
)
## create the applications and register them with the appServer
for appID, appSettings in self._AppSettings.items():
appServer.registerApplication(
self.setting('ApplicationClass')(appID, settings=appSettings)
)
if self.HTTPSERVER:
if self.HTTPSERVER_port:
self._HTTPServerSettings['port'] = self.HTTPSERVER_port
HTTPServer = self.setting('HTTPServerClass')(appServer,
settings=self._HTTPServerSettings)
multiPortServer.bindService(
HTTPServer,
serviceName=HTTPServer.setting("SERVER_SOFTWARE"),
address= (HTTPServer.setting('hostName'),
HTTPServer.setting('port')),
settings=HTTPServer.settings(),
)
##
self.multiPortServer = multiPortServer
return multiPortServer # probably won't be used by anyone directly
def startServer(self):
"""Start self.multiPortServer. This can only be called after self.loadServer().
Once you call this method you won't have control of the process until
the server has been shutdown. You can add custom shutdown operations by
including them after this call in your use of Launcher."""
self.multiPortServer.start()
def runFromCommandLine(self):
"""Run the launcher from the command line and process the args."""
self.processCommandLineArgs()
self.loadSettings(self.CONFIGFILE)
self.processSettings()
self.executeCommand()
def processCommandLineArgs(self):
""" determine the COMMAND and get setup to process it"""
try:
self._opts, self._args = getopt.getopt(
sys.argv[1:], "hlc:w:Hmo",
["help","license",
"config=",
"workingDir=",
"HTTP",
"HTTP_port=",
"monitor",
"oneshot"
])
except getopt.GetoptError:
# print help information and exit:
self.usage()
sys.exit(2)
self.CONFIGFILE = None
self.MONITOR = False
self.ONESHOT = False
self.HTTPSERVER = False
self.HTTPSERVER_port = None
self.START_SERVER = True
self.DAEMON = False
self.STOP_SERVER = False
self.PAUSE_SERVER = False
self.RESUME_SERVER = False
if self._args:
self.COMMAND = self._args[0]
else:
self.COMMAND = None
for o, a in self._opts:
if o in ("-h", "--help"):
print self.usage()
sys.exit()
if o in ("-l", "--license"):
print self.license()
sys.exit()
if o in ("-c", "--config"):
self.CONFIGFILE = a
if o in ("-w", "--workingDir"):
self._workingDir = a
if o in ("-m", "--monitor"):
self.MONITOR = True
if o in ("-o", "--oneshot"):
self.ONESHOT = True
self.MONITOR = True
print "The 'oneshot' option is implemented, but very BUGGY at this stage."
if o in ("-H", "--HTTP"):
self.HTTPSERVER = True
if o in ("--HTTP_port",):
self.HTTPSERVER_port = int(a)
if not self.COMMAND:
pass # start as normal process
elif self.COMMAND.lower() == 'start':
self.DAEMON = True
elif self.COMMAND.lower() == 'restart':
self.DAEMON = True
self.STOP_SERVER = True
self.START_SERVER = True
elif self.COMMAND.lower() == 'stop':
self.STOP_SERVER = True
self.START_SERVER = False
elif self.COMMAND.lower() == 'pause':
self.PAUSE_SERVER = True
elif self.COMMAND.lower() == 'resume':
self.RESUME_SERVER = True
elif self.COMMAND.lower() == 'status':
print 'The STATUS command is not implemented yet.'
sys.exit()
else:
print >> sys.stderr, self.COMMAND, 'is not a valid command.'
print
print self.usage()
sys.exit()
def executeCommand(self):
"""Execute the command this process was given on the command line."""
self.changeToWorkingDir()
## check for a running process as recorded by the PIDFile
PIDFile = self._MultiPortServerSettings.get('PIDFile', 'webware.pid')
if os.path.exists(PIDFile):
oldPID = int(open(PIDFile).read())
else:
oldPID = None
if self.PAUSE_SERVER:
if oldPID:
os.kill(oldPID, signal.SIGUSR1)
time.sleep(2)
return
else:
print >> sys.stderr, "The pause command failed because no Webware process is running."
return
elif self.RESUME_SERVER:
if oldPID:
os.kill(oldPID, signal.SIGUSR2)
time.sleep(2)
return
else:
print >> sys.stderr, "The resume command failed because no Webware process is running."
return
elif self.STOP_SERVER:
if self.stopOldProcess(oldPID, PIDFile):
oldPID = None
# can be followed by a START_SERVER
if not self.START_SERVER:
sys.exit()
if oldPID:
err = sys.stderr
print >> err
print >> err, "A Webware process appears to already be running for the specified config-file."
print >> err, oldPID, "was recorded as the ID of that process."
print >> err, "If that process doesn't exist, remove the file", PIDFile, "and try again."
print >> err
sys.exit()
if self.DAEMON:
if os.name == "posix":
pid = os.fork()
elif os.name == "nt":
pid = None
print "Daemon mode is not available on your OS"
if pid:
# recording of the pid is done by the MultiPortServer
# wait for a few seconds to make avoid mucking up the startup notice
time.sleep(1)
sys.exit()
if self.MONITOR:
self.spawnMonitor()
if self.START_SERVER:
self.loadServer()
self.startServer()
def changeToWorkingDir(self):
"""Change dir to the 'workingDir' if specified in the config-file or
command-line options."""
appServerSettings = self._AppServerSettings
if not self._workingDir and appServerSettings.has_key('workingDir') and \
appServerSettings['workingDir']:
os.chdir(appServerSettings['workingDir'])
elif self._workingDir:
os.chdir(self._workingDir)
def stopOldProcess(self, oldPID, PIDFile):
"""Run the stop command."""
if os.path.exists(self.setting('monitorPIDFile')):
try:
monitorPID = int(open(self.setting('monitorPIDFile')).read())
if monitorPID:
os.kill(monitorPID, signal.SIGTERM)
except OSError:
print >> sys.stderr, "Could not shutdown Monitor process", oldPID
print >> sys.stderr, "It may not exist."
return False
os.remove(self.setting('monitorPIDFile'))
if not oldPID:
print "Could not find a PID file for a running webware process."
return False
else:
try:
os.kill(oldPID, signal.SIGTERM)
except OSError:
print >> sys.stderr, "Could not shutdown AppServer process", oldPID
print >> sys.stderr, "It may not exist. Removing the PIDFile."
os.remove(PIDFile)
return True
else:
# MultiPortServer will os.remove(PIDFile) at the end of the shutdown
startTime = time.time()
while 1:
if not os.path.exists(PIDFile):
return True
if time.time() > startTime + (15):
# test for only 15 seconds
print
print >> sys.stderr, "The PIDFile (" + PIDFile + ") wasn't removed sucessfully."
print >> sys.stderr, "Process", oldPID, "might have stalled. Kill it manually!"
print
return False
def spawnMonitor(self):
"""Spawn a monitor process to provide fault tolerance"""
mainPID = os.getpid()
if os.name == "posix":
pid = os.fork()
if pid:
fp = open(self.setting('monitorPIDFile'),'w')
fp.write(str(pid))
fp.close()
print 'Monitor service started with process ID:', pid
else:
self.START_SERVER = False
if self.ONESHOT:
self._MonitorServiceSettings['oneshot'] = True
monitor = self.setting('MonitorClass')(
self,
mainPID,
settings=self._MonitorServiceSettings)
monitor.start()
elif os.name == "nt":
print "The Monitor service is not available on your OS"
def usage(self):
return """Webware %(version)s Launcher Script
USAGE: webware [opts]
webware [opts] [command]
-h, --help print this usage information and exit
-l, --license print license and exit
-c, --config <filename> read settings from the <filename> config-file
instead of from one of the defaults. The defaults
are './.webware_config.py', './.webware_config',
'~/.webware_config.py', and '~/.webware_config'
-m, --monitor monitor the AppServer to provide fault-tolerance
-o, --oneshot restart the AppServer for every new connection
IMPLEMENTED, BUT VERY BUGGY. Can you fix it??
-H, --HTTP start the builtin HTTPServer
--HTTP_port [port] start the HTTP server on [port],
default port is 8087
-w, --workingDir <path> use <path> as current working dir.
overrides 'workingDir' in the config file
POSSIBLE COMMANDS:
<no command> start AppServer as normal process
start start AppServer as daemon on Posix systems,
or as normal process on Windows
stop stop AppServer when running as daemon
Only available on Posix systems
restart 'stop-then-start' if running;
'start' if not running.
Only available on Posix systems
pause temporarily stop listening for connections
resume resume listening for connections after a 'pause'
status print status info when AppServer running as daemon
Only available on Posix systems.
NOT IMPLEMENTED YET!
Unlike the old WebKit, this Launcher script is not bound to a particulary
directory structure and can be run from any location. If it is started with the
'--workingDir' command-line option or the 'workingDir' setting is set in the
config-file, the script will change to that directory before executing any
commands. Otherwise the directory it is run from is the 'workingDir'. All
paths in the config settings will be processed relative to 'workingDir'. This
includes the paths that define where temporary files belong, such as the PIDFile
and the Session files.
See the file '.webware_config_annotated' in the Webware source distribution for
a detailed explanation of all the configuration options.
""" % {'version':version}
def license(self):
return """This is an EXPERIMENTAL version of Webware for Python
Copyright 1999-2001 by The Webware Development Team. All Rights Reserved.
Permission to use, copy, modify, and distribute this software and its
documentation for any purpose and without fee is hereby granted, provided that
the above copyright notice appear in all copies and that both that copyright
notice and this permission notice appear in supporting documentation, and that
the names of the authors not be used in advertising or publicity pertaining to
distribution of the software without specific, written prior permission.
THE AUTHORS DISCLAIM ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHORS
BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION
WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
Please visit http://webware.sourceforge.net for more information.
"""
##################################################
## if run from the command line ##
if __name__ == '__main__':
Launcher().runFromCommandLine()
