Re: [courier-users] Loading / reloading Pythonfilter
On Wed 07/Jun/2017 21:39:34 +0200 Gordon Messmer wrote: > On 06/07/2017 04:32 AM, Alessandro Vesely wrote: >> Last time I issued courierfilter stop, and then start. >> However, that way I have to reload also clamav databases, which takes a long >> time. Is there any better way to upgrade the running Python process? > > That should only be the case if you were running pyclamav, which is no longer > available or supported. The supported means of using clamav is to run an > instance of clamd as the courier user, and use pyclamd to scan messages. In > that configuration, you should be able to "filterctl stop pythonfilter/start > pythonfilter" without significant delays. Hm... I wouldn't recommend that. During the 17 seconds it took to reload "C" filter databases, I only had one "432 Mail filters temporarily unavailable". Of course, stopping and restarting pythonfilter alone would have taken much less. Short as that time might have been, messages with bad attachments would have slipped through unfiltered in the meanwhile. Ale -- signature.asc Description: OpenPGP digital signature -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ courier-users mailing list courier-users@lists.sourceforge.net Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users
Re: [courier-users] Loading / reloading Pythonfilter
On 06/07/2017 04:32 AM, Alessandro Vesely wrote: Last time I issued courierfilter stop, and then start. However, that way I have to reload also clamav databases, which takes a long time. Is there any better way to upgrade the running Python process? That should only be the case if you were running pyclamav, which is no longer available or supported. The supported means of using clamav is to run an instance of clamd as the courier user, and use pyclamd to scan messages. In that configuration, you should be able to "filterctl stop pythonfilter/start pythonfilter" without significant delays. BTW, in order to tell distributed filter from home-brewed ones, I install sym links in the dist-package/pythonfilter directory, which point to the source file. I note that Python does not produce any .pyc in that case. That's true, but that only means that python will take slightly longer to parse those files when they're imported. Since that only happens on startup, it's hard to make the case that producing pyc files will significantly impact performance at all. Since the pyc is what should get loaded when loading a module, an idea could be to reload modules if they have been recompiled since the last time they were loaded. Let me attach the idea. Beware: I did'n run it, not even once, and didn't fully understand the caveats about using reload(), see e.g.: https://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module#answer-438845 pythonfilter uses threads in order to allow modules to keep statistics in memory. As far as I can tell, reloading isn't going to be safe. One more question, about virtualenv. Has anyone had experience loading Pythonfilter that way under Courier? Another possibility is to install Pythonfilter globally, but keep additional packages in a virtualenv, just to ease tracking requirements. I haven't, personally. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ courier-users mailing list courier-users@lists.sourceforge.net Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users
[courier-users] Loading / reloading Pythonfilter
I changed the code of a filter which is already configured and running. Now I
have to install it. Last time I issued courierfilter stop, and then start.
However, that way I have to reload also clamav databases, which takes a long
time. Is there any better way to upgrade the running Python process?
BTW, in order to tell distributed filter from home-brewed ones, I install sym
links in the dist-package/pythonfilter directory, which point to the source
file. I note that Python does not produce any .pyc in that case. However, the
following command compiles in the current directory irrespectively of whatever:
python -m compileall -l attachments.py
Since the pyc is what should get loaded when loading a module, an idea could be
to reload modules if they have been recompiled since the last time they were
loaded. Let me attach the idea. Beware: I did'n run it, not even once, and
didn't fully understand the caveats about using reload(), see e.g.:
https://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module#answer-438845
One more question, about virtualenv. Has anyone had experience loading
Pythonfilter that way under Courier? Another possibility is to install
Pythonfilter globally, but keep additional packages in a virtualenv, just to
ease tracking requirements.
TIA
Ale
---
../courier-pythonfilter/courier-pythonfilter-1.11/build/scripts-2.7/pythonfilter
2016-10-31 17:21:05.543243721 +0100
+++ ../pythonfilter-modified-but-not-tested.py 2017-06-07 13:09:59.818621251
+0200
@@ -111,6 +111,11 @@
(importError[0], importError[1]))
sys.stderr.write(''.join(traceback.format_tb(importError[2])))
sys.exit()
+try:
+moduleDate = os.stat(module.__file__).st_mtime
+except OSError:
+moduleDate = None
+
if hasattr(module, 'initFilter'):
try:
module.initFilter()
@@ -126,7 +131,7 @@
try:
# Store the name of the filter module and a reference to its
# dofilter function in the "filters" array.
-filters.append((moduleName, module.doFilter, bypass))
+filters.append((moduleName, module.doFilter, bypass, module,
moduleDate))
except AttributeError:
# Log bad modules
importError = sys.exc_info()
@@ -203,13 +208,23 @@
# Prepare a set of filters that will not be run if a module returns
# a 2XX code, and specifies a list of filters to bypass.
bypass = set()
-for i_filter in filters:
+for i_i, i_filter in enumerate(filters):
# name = i_filter[0]
# function = i_filter[1]
# bypass = i_filter[2]
if i_filter[0] in bypass:
continue
try:
+if i_filter[4] is not None:
+moduleDate = os.stat(i_filter[3].__file__).st_mtime
+if moduleDate > i_filter[4]:
+reload(i_filter[3])
+i_filter[4] = moduleDate
+filters[i_i] = (i_filter[0], i_filter[1], i_filter[2],
i_filter[3], i_filter[4])
+sys.stderr.write('Reloaded %s\n' % i_filter[0])
+except:
+sys.stderr.write('Not reloaded %s\n' % i_filter[0])
+try:
replyCode = i_filter[1](bodyFile, controlFileList)
except:
filterError = sys.exc_info()
signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
courier-users mailing list
courier-users@lists.sourceforge.net
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users
