I have recently been doing a re-write of eclean and one of the problems to address was the need to run "emaint fix packages" when doing any cleaning of the packages directory. Since eclean and emaint are coded in python, I looked into importing the emaint code and running it from within eclean. Currently emaint is an all inclusive script without any easily import-able modules. Asking Zac, he stated that it would be fine to change that to a cli script and library.
So, going through the code, I found it pretty much was already
modularized but just stuck together in the one file. In there was also
a TODO message:
# TODO: Create a system that allows external modules to be added without
# the need for hard coding.
modules = {
"world" : WorldHandler,
"binhost":BinhostHandler,
"moveinst":MoveInstalled,
"movebin":MoveBinary,
"cleanresume":CleanResume
}
So having a bit of experience with porthole's plug-in system, I took it
upon myself to create a small plug-in system. I also designed it to load
as little code as possible. With this plug-in system emaint actually
runs slightly faster than the original script. The only changes
being done were to use the plug-in system and independent module files
containing the original unmodified module class. I believe the
speed increase is due to python not having to load as much script, as
only the module's __init__.py for all modules is loaded initially and
then the requested operation module file is loaded.
I think you will find this plug-in system very straightforward, perhaps
a little overkill for emaint. But it does offer a number of advantages
including less code changes to add/remove modules, full module usability
by other apps, etc.. Fully re-useable plug-in system for other apps
(when complete).
I have a bzr branch of portage's svn trunk with the modified emaint. It
is fully functional for your testing and review of the code. I do have a
little more work to do on it. So far I have not modified the code to
make use of the 'functions' and 'func_desc' data provided by the modules
__init__.py module_spec dictionary. It is currently relying on the
original hard coded ['check', 'fix'] functions in the different module
classes. It has also been suggested to make the modules dictionary a
LazyLoad dictionary instead.
I have a bzr branch available:
bzr co http://dev.gentooexperimental.org/~dol-sen/bzr/portage-trunk/
If I did it correct, I also have migrated it to a git branch too:
git remote add emaint
http://dev.gentooexperimental.org/~dol-sen/git/portage.git/
git fetch emaint
git branch emaint
and test/review it.
I have attached 2 module __init__.py's and the module.py file for
your perusal without the need to checkout a copy of my repo.
For those of you getting a bzr checkout or git remote to test. Try
moving modules in and out of the modules dir and then run emaint -h.
Also the vdbkeys module is old and no longer usable, so will not be
included in the final version should it be approved and implemented. I
include it here so you can see how easily modules can be added/removed.
I have also coded up a dummy module demonstrating the ability to
automatically change the class returned for a module name according to
an environment variable setting. But there are a number of other ways
that it could use too.
--
Brian Dolbec <brian.dol...@gmail.com>
#!/usr/bin/python -O
# vim: noet :
#
# Copyright 2010 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2 or later
#
# $Header$
from __future__ import print_function
from portage import os
from portage.exception import PortageException
class InvalidModuleName(PortageException):
"""An invalid or unknown module name."""
class Module(object):
"""Class to define and hold our plug-in modules
@type name: string
@param name: the module name
@type path: the path to the new module
"""
def __init__(self, name, path):
"""Some variables initialization"""
self.name = name
self.path = path
self.kids_names = []
self.initialized = self._initialize()
def _initialize(self):
"""Initialize the plug-in module
@rtype: boolean
"""
try:
os.chdir(self.path)
mod_name = ".".join(["emaint", "modules", self.name])
self._module = __import__(mod_name, [],[], ["not empty"])
self.valid = True
except ImportError, e:
self.valid = False
return False
self.module_spec = self._module.module_spec
#self._module.get_class = self.get_class
self.kids = {}
for submodule in self.module_spec['provides']:
kid = self.module_spec['provides'][submodule]
kidname = kid['name']
kid['module_name'] = '.'.join([mod_name, self.name])
#kid['parent'] = self._module
kid['is_imported'] = False
self.kids[kidname] = kid
self.kids_names.append(kidname)
return True
def get_class(self, name):
if not name or name not in self.kids_names:
raise InvalidModuleName("Module name '%s' was invalid or not"
%modname + "part of the module '%s'" %self.name)
kid = self.kids[name]
if kid['is_imported']:
module = kid['instance']
else:
try:
module = __import__(kid['module_name'], [],[], ["not empty"])
kid['instance'] = module
kid['is_imported'] = True
except ImportError, e:
raise
mod_class = getattr(module, kid['class'])
return mod_class
class Modules(object):
"""Dynamic modules system for loading and retrieving any of the
installed emaint modules and/or provided class's
@param path: Optional path to the "modules" directory or
defaults to the directory of this file + '/modules'
"""
def __init__(self, path=None):
if path:
self.module_path = path
else:
self.module_path = os.path.join((
os.path.dirname(os.path.realpath(__file__))), "modules")
self.modules = self._get_all_modules()
self.module_names = list(self.modules)
self.module_names.sort()
def _get_all_modules(self):
"""scans the emaint modules dir for loadable modules
@rtype: dictionary of module_plugins
"""
module_dir = self.module_path
importables = []
names = os.listdir(module_dir)
for entry in names:
if os.path.isdir(os.path.join(module_dir, entry)):
try:
statinfo = os.stat(os.path.join(module_dir, entry, '__init__.py'))
if statinfo.st_nlink == 1:
importables.append(entry)
except EnvironmentError, er:
pass
os.chdir(module_dir)
kids = {}
for entry in importables:
new = Module(entry, module_dir) #, self)
for module_name in new.kids:
kid = new.kids[module_name]
kid['parent'] = new
kids[kid['name']] = kid
return kids
def get_module_names(self):
"""Convienence function to return the list of installed modules
available
@rtype: list
@return: the installed module names available
"""
return self.module_names
def get_class(self, modname):
"""Retrieves a module class desired
@type modname: string
@param modname: the module class name
"""
if modname and modname in self.module_names:
mod = self.modules[modname]['parent'].get_class(modname)
else:
raise InvalidModuleName("Module name '%s' was invalid or not"
%modname + "found")
return mod
def get_description(self, modname):
"""Retrieves the module class decription
@type modname: string
@param modname: the module class name
@type string
@return: the modules class decription
"""
if modname and modname in self.module_names:
mod = self.modules[modname]['description']
else:
raise InvalidModuleName("Module name '%s' was invalid or not"
%modname + "found")
return mod
def get_functions(self, modname):
"""Retrieves the module class exported functions
@type modname: string
@param modname: the module class name
@type list
@return: the modules class exported functions
"""
if modname and modname in self.module_names:
mod = self.modules[modname]['functions']
else:
raise InvalidModuleName("Module name '%s' was invalid or not"
%modname + "found")
return mod
def get_func_descriptions(self, modname):
"""Retrieves the module class exported functions descriptions
@type modname: string
@param modname: the module class name
@type list
@return: the modules class exported functions descriptions
"""
if modname and modname in self.module_names:
desc = self.modules[modname]['func_desc']
else:
raise InvalidModuleName("Module name '%s' was invalid or not"
%modname + "found")
return desc
#!/usr/bin/python -O
# vim: noet :
#
# Copyright 2010 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2 or later
#
# $Header$
"""'This emaint module provides checks and maintenance for:
1) "Performing package move updates for installed packages",
2)"Perform package move updates for binary packages"
"""
module_spec = {
'name': 'move',
'description': "Provides functions to check for and move packages " +\
"either installed or binary packages stored on this system",
'provides':{
'module1': {
'name': "moveinst",
'class': "MoveInstalled",
'description': "Perform package move updates for installed packages",
'options': ['check', 'fix']
},
'module2':{
'name': "movebin",
'class': "MoveBinary",
'description': "Perform package move updates for binary packages",
'functions': ['check', 'fix'],
'func_desc': {}
}
}
}
#!/usr/bin/python -O
# vim: noet :
#
# Copyright 2010 Gentoo Technologies, Inc.
# Distributed under the terms of the GNU General Public License v2 or later
#
# $Header$
"""'This emaint program module provides checks for a test of
environment set changes which change the class exported via the plug-in system.
"""
import os
DEFAULT_CLASS = "EnvOne"
AVAILABLE_CLASSES = [ "EnvOne", "EnvTwo", "EnvThree"]
options = {"1": "EnvOne", "2": "EnvTwo", "3": "EnvThree"}
config_class = DEFAULT_CLASS
try:
test_param = os.environ["TESTIT"]
if test_param in options:
config_class = options[test_param]
except KeyError:
pass
module_spec = {
'name': 'testdummy',
'description': "Provides functions to scan, check and " + \
"Generate an example auto config module",
'provides':{
'module1': {
'name': "envset",
'class': config_class,
'description': "Dummy module used to test environment settings " + \
"to change the class being exported by the plug-in system",
'functions': ['check', 'fix'],
'func_desc': {}
}
}
}
signature.asc
Description: This is a digitally signed message part
