Poll for output in Minions:
This implementation is a result of an idea that was post on ML a few weeks
ago.
The main idea was to be able to get some output during the minion side
execution.
The idea maybe useful for tasks which are kind of long and user wants to get
some
notification about what is going on on the other side. Because we are talking
about
long running operations that stuff is reasonable only for async stuff. Here is
some example
of tracking a long operation on overlord :
[r...@acerfedora func]# func "*" call --logpoll --async echo run_str_log
"eeeeee"
Logging data is initializing ...
------HOST : acerfedora -------
2009-07-28 20:06:55,013 - 1248800814.958679-minion - INFO - Starting counting
logger ...
------HOST : acerfedora -------
2009-07-28 20:08:00,034 - 1248800814.958679-minion - INFO - Calling method
with counter is 0
------HOST : acerfedora -------
2009-07-28 20:08:05,033 - 1248800814.958679-minion - INFO - Calling method
with counter is 1
------HOST : acerfedora -------
2009-07-28 20:08:10,034 - 1248800814.958679-minion - INFO - Calling method
with counter is 2
------HOST : acerfedora -------
2009-07-28 20:08:15,034 - 1248800814.958679-minion - INFO - Calling method
with counter is 3
That example is for lots of machines (overlord spec) but polling output seems
more reasonable for doing it for an one host :
[r...@acerfedora func]# func "*" call --nopoll --async echo run_str_log
"eeeeee"
JOB_ID: '*-echo-run_str_log-1248802501.952774'
[r...@acerfedora func]#
[r...@acerfedora func]# func "*" call --logone="acerfedora" "*-echo-
run_str_log-1248802501.952774"
2009-07-28 20:35:02,318 - 1248802502.256377-minion - INFO - Starting counting
logger ...
2009-07-28 20:36:07,318 - 1248802502.256377-minion - INFO - Calling method
with counter is 0
2009-07-28 20:36:12,317 - 1248802502.256377-minion - INFO - Calling method
with counter is 1
2009-07-28 20:36:17,317 - 1248802502.256377-minion - INFO - Calling method
with counter is 2
2009-07-28 20:36:22,317 - 1248802502.256377-minion - INFO - Calling method
with counter is 3
2009-07-28 20:36:27,318 - 1248802502.256377-minion - INFO - Calling method
with counter is 4
2009-07-28 20:36:32,318 - 1248802502.256377-minion - INFO - Calling method
with counter is 5
.
.
.
Making a method to have trackable output : It is pretty easy let see it on an
example :
a method in echo module :
def run_str_log(self,command):
import time
self.run_str_log.logger.info("Starting counting logger ...")
time.sleep(60)
for i in range(100):
time.sleep(5)
self.run_str_log.logger.info("Calling method with counter is
%d"%i)
return command
By default every method in minion modules is attached with a standart logger
instance.To reach the
log instance what you need is calling self.method.logger and you have all the
stuff in a standart logging.
Python API :
To poll logs on minion side with API :
In [1]: from func.overlord.client import Overlord
In [2]: fc = Overlord("*",async=True)
In [3]: j=fc.echo.run_str_log("blip")
In [4]: j
Out[4]: '*-echo-run_str_log-1248802917.3343029'
In [5]: fc.tail_log(j)
Out[5]:
({'acerfedora': ['2009-07-28 20:41:57,572 - 1248802917.536685-minion - INFO -
Starting counting logger ...\n',
'']},
False)
About how to make that code useful in your applications you should take a look
into overlord/cmd_modules/call.py
Also you can poll a single machine if you need , which maybe useful in webapps
or some other stuff.
In [6]: fc = Overlord("*",async=True)
In [7]: j=fc.echo.run_str_log("blip")
In [8]: fc.tail_log(j,host="acerfedora")
Out[8]:
({'acerfedora': ['2009-07-28 20:43:45,233 - 1248803025.1892841-minion - INFO -
Starting counting logger ...\n',
'']},
False)
In [9]:
Under the hood :
Because we use xml-rpc as transport protocol these kind of things are not very
efficient in func. But the best way
to do that was (if someone has better ideas please tell) to log all async
requests in different log files according
to their minion job ids. When a new async request comes a log object is
attached to called method and new log file
is created named with minion job id.Therefore user can use that log object to
log stuff and overlord stuff can track
that log file via its (overlord job_id)--(minion job_id) matching db.That
structure maybe very useful for methods that
take really long time to finish its stuff. The cons of the method is we have
lots of separate log files and we have kind
of high one to many transport communication,polling 100 minions for log
information will not be very effcient.However it
will work for single or a few machines very good !
Comments ,ideas ?
Branch name : logger
http://github.com/makkalot/func/tree/logger
_______________________________________________
Func-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/func-list
