2012/4/16 Patrick Ohly <[email protected]>:
> On Fri, 2012-04-13 at 15:28 +0200, Patrick Ohly wrote:
>> This is just a quick-and-dirty solution, not committed anywhere. Shall I
>> do the base work for a few tests and then let you continue with adding
>> the rest?
>
> Below are the commits, already merged into master after a quick test
> run:
> http://syncev.meego.com/2012-04-16-12-37_lucid-amd64_prebuilt-amd64_testing_dbus/
>
> I started with your initial commit and then included the changes that we
> were discussing, plus a fix for a failure still visible in that test
> run.
>
> The reason is that your scanFiles() implementation is slightly different
> from the C++ version. In contrast to the C++ version it includes lines
> like:
>
> sources/addressbook/config.ini:# ActiveSync Address Book = eas-contacts
> sources/addressbook/config.ini:# ActiveSync Events = eas-events
> sources/addressbook/config.ini:# ActiveSync Todos = eas-todos
> sources/addressbook/config.ini:# ActiveSync Memos = eas-memos
>
> These need to be excluded by the comparison because these comments can
> change, depending on how the binaries were compiled or run. Your
> reference text in the ScheduleWorld config includes one particular
> variation and therefore it worked for me when testing manually, but the
> nightly testing differs.
>
> A good test for the assertEqualDiff() :-) See:
>
> http://syncev.meego.com/2012-04-16-12-37_lucid-amd64_prebuilt-amd64_testing_dbus/testing-amd64/7-dbus/Cmdline_SetupScheduleWorld.log.html
>
> With everything in place, do you still want to convert some more tests?
>
> I think one commit per converted test would be good, with C++ removed
> and Python added in the same test. I expect that most tests can and
> should be moved. The only reason for keeping a test in the C++ code
> would be if it explicitly tests some aspect that cannot be tested when
> going through D-Bus - not sure whether there is any such case.
>
>
> commit 2925b97bac45ae4dc4dac92b04b91683cad29ddc
> Author: Krzesimir Nowak <[email protected]>
> Date: Thu Apr 12 15:45:02 2012 +0200
>
> D-Bus Testing: add Cmdline tests framework
>
> This is a translation of the corresponding C++ TestCmdline from
> Cmdline.cpp into Python.
>
> commit aa562e5ef97eba78b7c45ab095df5071134840dc
> Author: Patrick Ohly <[email protected]>
> Date: Mon Apr 16 09:56:50 2012 +0200
>
> D-Bus Testing: added assertEqualDiff()
>
> Replaces the diffStrings() from the previous commit. Renamed it
> because it really is specifying an assertion and moved it to
> DBusUtil because it may be useful outside of the command line
> testing.
>
> Also made sure that the diff is part of the assertion because then it
> will be repeated when listing the failure.
>
> commit 5389b8eeeddc328e29a86c91aec8606ac3fb9741
> Author: Patrick Ohly <[email protected]>
> Date: Mon Apr 16 10:54:52 2012 +0200
>
> D-Bus Testing: ensure that we have assertRegexpMatches()
>
> unittest.assertRegexpMatches() is very useful for command line
> output testing. But it is not yet available in Ubuntu Lucid,
> the current base line for SyncEvolution releases. Therefore
> provide a fallback implementation.
>
> commit 0e63bc8e59b53a8821beff9f29d061cae48a66fc
> Author: Patrick Ohly <[email protected]>
> Date: Mon Apr 16 17:26:12 2012 +0200
>
> D-Bus Testing: fixed scanFiles()
>
> The function must return all property assignments, including those
> which are commented out. The Python version accidentally included some
> comment lines which looked like assignments, partly because it ignored
> leading spaces, partly because it looked anywhere in the line for an
> equal sign.
>
> Fixed by using a regex for the matching.
>
> commit 67ce74b57fcb51f57b8cf29e353e13da16d60b8b
> Author: Patrick Ohly <[email protected]>
> Date: Mon Apr 16 12:13:14 2012 +0200
>
> D-Bus Testing: eliminate TestCmdline.testdir
>
> All D-Bus tests already have their private xdg_root = 'temp-test-dbus'
> which is cleared before running each test. Use it also for TestCmdline
> instead of creating our own temporary directory, for the sake of
> consistency.
>
> Another advantage is that the syncevo-dbus-server is running with that
> same XDG root and reading/writing files in it automatically. The
> command line client currently *does* send its XDG env variables and
> the server uses them, but perhaps that feature will have to be
> removed, because it has a design mistake (*), so don't depend on it.
>
> (*) If different clients and the auto-syncing code in the server are
> allowed to use different config roots, the 'config name' alone no
> longer is a unique identifier for configs.
>
> commit 7c2de4ed5199f9c971d1474595f5a04988c1eac3
> Author: Patrick Ohly <[email protected]>
> Date: Mon Apr 16 12:58:52 2012 +0200
>
> D-Bus Testing: added running of command line tool
>
> The 'syncevolution' command line tool is started inside the same D-Bus
> session as the 'syncevo-dbus-server' that is started before running
> each test. They use the same XDG env variables.
>
> The test-dbus.py script is able to watch and report all D-Bus traffic
> between tool and server.
>
> Because tests are meant to be realistic, SYNCEVOLUTION_DEBUG has to be
> optional and must be turned off for all tests which test for real
> output as seen by users.
>
> Every single invocation of runCmdline() should check stdout and
> stderr, to detect unexpected output. The return code is already tested
> by runCmdline() itself, unless explicitly turned off (may be useful for
> tests which expect certain kind of failures).
>
> Ideally the entire string gets tested, either via an exact string
> comparison or a full regex match. Note that assertRegexpMatches() only
> does a search, so use ^ and $ to ensure full match.
>
> If (and only if) the command is expected to produce output on both
> stdout and stderr where the order matters (for example, syncing), then
> stdout and stderr should be combined into one stream. Otherwise the
> order cannot be tested.
>
> commit 470f0e7d7a56b96adff21e5e918c58ee0a9e3420
> Author: Patrick Ohly <[email protected]>
> Date: Mon Apr 16 13:02:23 2012 +0200
>
> D-Bus Testing: added testSetupScheduleWorld
>
> The test is a straight copy from Cmdline.cpp. Because the hard-coded
> default SSL certificate path is not known in test-dbus.py, it has to
> extract that string by printing a template once with the command line
> tool (only the command line exposes the comments, normal D-Bus API
> doesn't) and finding it in the output.
>
> This allows copying the C++ tests without making them less strict by
> filtering out the SSL certificate path before the result comparisons.
>
> commit a27f2af580a00d81107f69bc5802c079cef23766
> Author: Patrick Ohly <[email protected]>
> Date: Mon Apr 16 14:38:34 2012 +0200
>
> D-Bus Testing: work around Python 2.6 subprocess bug
>
> Explicitly pass an environment when running the command line
> tool. Otherwise subprocess.Popen() from Python 2.6 uses not
> os.environ (which would be okay) but rather the environment passed to
> a previous call to subprocess.Popen() (which will fail if the previous
> test ran with an environment which had SYNCEVOLUTION_DEBUG set).
>
Some nitpicks here:
+ # Explicitly pass an environment. Otherwise subprocess.Popen()
+ # from Python 2.6 uses not os.environ (which would be okay)
+ # but rather the environment passed to a previous call to
+ # subprocess.Popen() (which will fail if the previous test ran
+ # with an environment which had SYNCEVOLUTION_DEBUG set).
+ if env == None:
+ env=os.environ
1. @property("debug", False) is not honored in runCmdline. If
os.environ has SYNCEVOLUTION_DEBUG key then most of tests are going to
fail, because stderr is not empty - it contains debugging information.
2. Same with "ENV" property - it is not honored. There is a test name
testMatchTemplate which needs some different template directory
(actually the one in test/testcases/templates). So for now the only
way to do it is to deepcopy os.environ and set different
SYNCEVOLUTION_TEMPLATE_DIR.
You wrote at some point in this thread:
"Regarding the environment, better refactor the code that is used for
starting syncevo-dbus-server and use the same environment for both
processes."
Well, this clearly does not happen here. Would not it be better if
environment used for running syncevo-dbus-server is stored in
self.storedenv and that would be used for running syncevolution?
--- a/test/test-dbus.py
+++ b/test/test-dbus.py
@@ -371,6 +371,7 @@ class DBusUtil(Timeout):
quit_events = []
reply = None
pserver = None
+ storedenv = {}
def getTestProperty(self, key, default):
"""retrieve values set with @property()"""
@@ -422,6 +423,16 @@ class DBusUtil(Timeout):
# and increase log level
if self.getTestProperty("debug", True):
env["SYNCEVOLUTION_DEBUG"] = "1"
+ else:
+ # we explicitly specified @property("debug", False")
+ # thus we do not want to have any debugging output.
+ try:
+ del env["SYNCEVOLUTION_DEBUG"]
+ except KeyError:
+ # there was no such key anyway
+ pass
+
+ self.storedenv = env
# can be set by a test to run additional tests on the content
# of the D-Bus log
@@ -3551,8 +3562,10 @@ class TestCmdline(unittest.TestCase, DBusUtil):
# but rather the environment passed to a previous call to
# subprocess.Popen() (which will fail if the previous test ran
# with an environment which had SYNCEVOLUTION_DEBUG set).
+ cmdline_env = copy.deepcopy(self.storedenv)
if env == None:
- env=os.environ
+ env={}
+ cmdline_env.update(env)
if preserveOutputOrder:
s = subprocess.Popen(a, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
env=env)
> --
> Best Regards, Patrick Ohly
>
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
>
>
_______________________________________________
SyncEvolution mailing list
[email protected]
http://lists.syncevolution.org/listinfo/syncevolution
