2012/4/17 Krzesimir Nowak <[email protected]>:
> 2012/4/16 Patrick Ohly <[email protected]>:
>> On Fri, 2012-04-13 at 15:28 +0200, Patrick Ohly wrote:
>>> This is just a quick-and-dirty solution, not committed anywhere. Shall I
>>> do the base work for a few tests and then let you continue with adding
>>> the rest?
>>
>> Below are the commits, already merged into master after a quick test
>> run:
>> http://syncev.meego.com/2012-04-16-12-37_lucid-amd64_prebuilt-amd64_testing_dbus/
>>
>> I started with your initial commit and then included the changes that we
>> were discussing, plus a fix for a failure still visible in that test
>> run.
>>
>> The reason is that your scanFiles() implementation is slightly different
>> from the C++ version. In contrast to the C++ version it includes lines
>> like:
>>
>> sources/addressbook/config.ini:# ActiveSync Address Book = eas-contacts
>> sources/addressbook/config.ini:# ActiveSync Events = eas-events
>> sources/addressbook/config.ini:# ActiveSync Todos = eas-todos
>> sources/addressbook/config.ini:# ActiveSync Memos = eas-memos
>>
>> These need to be excluded by the comparison because these comments can
>> change, depending on how the binaries were compiled or run. Your
>> reference text in the ScheduleWorld config includes one particular
>> variation and therefore it worked for me when testing manually, but the
>> nightly testing differs.
>>
>> A good test for the assertEqualDiff() :-) See:
>>
>> http://syncev.meego.com/2012-04-16-12-37_lucid-amd64_prebuilt-amd64_testing_dbus/testing-amd64/7-dbus/Cmdline_SetupScheduleWorld.log.html
>>
>> With everything in place, do you still want to convert some more tests?
>>
>> I think one commit per converted test would be good, with C++ removed
>> and Python added in the same test. I expect that most tests can and
>> should be moved. The only reason for keeping a test in the C++ code
>> would be if it explicitly tests some aspect that cannot be tested when
>> going through D-Bus - not sure whether there is any such case.
>>
>>
>> commit 2925b97bac45ae4dc4dac92b04b91683cad29ddc
>> Author: Krzesimir Nowak <[email protected]>
>> Date: Thu Apr 12 15:45:02 2012 +0200
>>
>> D-Bus Testing: add Cmdline tests framework
>>
>> This is a translation of the corresponding C++ TestCmdline from
>> Cmdline.cpp into Python.
>>
>> commit aa562e5ef97eba78b7c45ab095df5071134840dc
>> Author: Patrick Ohly <[email protected]>
>> Date: Mon Apr 16 09:56:50 2012 +0200
>>
>> D-Bus Testing: added assertEqualDiff()
>>
>> Replaces the diffStrings() from the previous commit. Renamed it
>> because it really is specifying an assertion and moved it to
>> DBusUtil because it may be useful outside of the command line
>> testing.
>>
>> Also made sure that the diff is part of the assertion because then it
>> will be repeated when listing the failure.
>>
>> commit 5389b8eeeddc328e29a86c91aec8606ac3fb9741
>> Author: Patrick Ohly <[email protected]>
>> Date: Mon Apr 16 10:54:52 2012 +0200
>>
>> D-Bus Testing: ensure that we have assertRegexpMatches()
>>
>> unittest.assertRegexpMatches() is very useful for command line
>> output testing. But it is not yet available in Ubuntu Lucid,
>> the current base line for SyncEvolution releases. Therefore
>> provide a fallback implementation.
>>
>> commit 0e63bc8e59b53a8821beff9f29d061cae48a66fc
>> Author: Patrick Ohly <[email protected]>
>> Date: Mon Apr 16 17:26:12 2012 +0200
>>
>> D-Bus Testing: fixed scanFiles()
>>
>> The function must return all property assignments, including those
>> which are commented out. The Python version accidentally included some
>> comment lines which looked like assignments, partly because it ignored
>> leading spaces, partly because it looked anywhere in the line for an
>> equal sign.
>>
>> Fixed by using a regex for the matching.
>>
>> commit 67ce74b57fcb51f57b8cf29e353e13da16d60b8b
>> Author: Patrick Ohly <[email protected]>
>> Date: Mon Apr 16 12:13:14 2012 +0200
>>
>> D-Bus Testing: eliminate TestCmdline.testdir
>>
>> All D-Bus tests already have their private xdg_root = 'temp-test-dbus'
>> which is cleared before running each test. Use it also for TestCmdline
>> instead of creating our own temporary directory, for the sake of
>> consistency.
>>
>> Another advantage is that the syncevo-dbus-server is running with that
>> same XDG root and reading/writing files in it automatically. The
>> command line client currently *does* send its XDG env variables and
>> the server uses them, but perhaps that feature will have to be
>> removed, because it has a design mistake (*), so don't depend on it.
>>
>> (*) If different clients and the auto-syncing code in the server are
>> allowed to use different config roots, the 'config name' alone no
>> longer is a unique identifier for configs.
>>
>> commit 7c2de4ed5199f9c971d1474595f5a04988c1eac3
>> Author: Patrick Ohly <[email protected]>
>> Date: Mon Apr 16 12:58:52 2012 +0200
>>
>> D-Bus Testing: added running of command line tool
>>
>> The 'syncevolution' command line tool is started inside the same D-Bus
>> session as the 'syncevo-dbus-server' that is started before running
>> each test. They use the same XDG env variables.
>>
>> The test-dbus.py script is able to watch and report all D-Bus traffic
>> between tool and server.
>>
>> Because tests are meant to be realistic, SYNCEVOLUTION_DEBUG has to be
>> optional and must be turned off for all tests which test for real
>> output as seen by users.
>>
>> Every single invocation of runCmdline() should check stdout and
>> stderr, to detect unexpected output. The return code is already tested
>> by runCmdline() itself, unless explicitly turned off (may be useful for
>> tests which expect certain kind of failures).
>>
>> Ideally the entire string gets tested, either via an exact string
>> comparison or a full regex match. Note that assertRegexpMatches() only
>> does a search, so use ^ and $ to ensure full match.
>>
>> If (and only if) the command is expected to produce output on both
>> stdout and stderr where the order matters (for example, syncing), then
>> stdout and stderr should be combined into one stream. Otherwise the
>> order cannot be tested.
>>
>> commit 470f0e7d7a56b96adff21e5e918c58ee0a9e3420
>> Author: Patrick Ohly <[email protected]>
>> Date: Mon Apr 16 13:02:23 2012 +0200
>>
>> D-Bus Testing: added testSetupScheduleWorld
>>
>> The test is a straight copy from Cmdline.cpp. Because the hard-coded
>> default SSL certificate path is not known in test-dbus.py, it has to
>> extract that string by printing a template once with the command line
>> tool (only the command line exposes the comments, normal D-Bus API
>> doesn't) and finding it in the output.
>>
>> This allows copying the C++ tests without making them less strict by
>> filtering out the SSL certificate path before the result comparisons.
>>
>> commit a27f2af580a00d81107f69bc5802c079cef23766
>> Author: Patrick Ohly <[email protected]>
>> Date: Mon Apr 16 14:38:34 2012 +0200
>>
>> D-Bus Testing: work around Python 2.6 subprocess bug
>>
>> Explicitly pass an environment when running the command line
>> tool. Otherwise subprocess.Popen() from Python 2.6 uses not
>> os.environ (which would be okay) but rather the environment passed to
>> a previous call to subprocess.Popen() (which will fail if the previous
>> test ran with an environment which had SYNCEVOLUTION_DEBUG set).
>>
>
> Some nitpicks here:
>
> + # Explicitly pass an environment. Otherwise subprocess.Popen()
> + # from Python 2.6 uses not os.environ (which would be okay)
> + # but rather the environment passed to a previous call to
> + # subprocess.Popen() (which will fail if the previous test ran
> + # with an environment which had SYNCEVOLUTION_DEBUG set).
> + if env == None:
> + env=os.environ
>
> 1. @property("debug", False) is not honored in runCmdline. If
> os.environ has SYNCEVOLUTION_DEBUG key then most of tests are going to
> fail, because stderr is not empty - it contains debugging information.
>
> 2. Same with "ENV" property - it is not honored. There is a test name
> testMatchTemplate which needs some different template directory
> (actually the one in test/testcases/templates). So for now the only
> way to do it is to deepcopy os.environ and set different
> SYNCEVOLUTION_TEMPLATE_DIR.
>
> You wrote at some point in this thread:
> "Regarding the environment, better refactor the code that is used for
> starting syncevo-dbus-server and use the same environment for both
> processes."
>
> Well, this clearly does not happen here. Would not it be better if
> environment used for running syncevo-dbus-server is stored in
> self.storedenv and that would be used for running syncevolution?
>
> --- a/test/test-dbus.py
> +++ b/test/test-dbus.py
> @@ -371,6 +371,7 @@ class DBusUtil(Timeout):
> quit_events = []
> reply = None
> pserver = None
> + storedenv = {}
>
> def getTestProperty(self, key, default):
> """retrieve values set with @property()"""
> @@ -422,6 +423,16 @@ class DBusUtil(Timeout):
> # and increase log level
> if self.getTestProperty("debug", True):
> env["SYNCEVOLUTION_DEBUG"] = "1"
> + else:
> + # we explicitly specified @property("debug", False")
> + # thus we do not want to have any debugging output.
> + try:
> + del env["SYNCEVOLUTION_DEBUG"]
> + except KeyError:
> + # there was no such key anyway
> + pass
> +
> + self.storedenv = env
>
> # can be set by a test to run additional tests on the content
> # of the D-Bus log
> @@ -3551,8 +3562,10 @@ class TestCmdline(unittest.TestCase, DBusUtil):
> # but rather the environment passed to a previous call to
> # subprocess.Popen() (which will fail if the previous test ran
> # with an environment which had SYNCEVOLUTION_DEBUG set).
> + cmdline_env = copy.deepcopy(self.storedenv)
> if env == None:
> - env=os.environ
> + env={}
> + cmdline_env.update(env)
> if preserveOutputOrder:
> s = subprocess.Popen(a, stdout=subprocess.PIPE,
> stderr=subprocess.STDOUT,
> env=env)
>
Eh, should be:
--- a/test/test-dbus.py
+++ b/test/test-dbus.py
@@ -371,6 +371,7 @@ class DBusUtil(Timeout):
quit_events = []
reply = None
pserver = None
+ storedenv = {}
def getTestProperty(self, key, default):
"""retrieve values set with @property()"""
@@ -422,6 +423,16 @@ class DBusUtil(Timeout):
# and increase log level
if self.getTestProperty("debug", True):
env["SYNCEVOLUTION_DEBUG"] = "1"
+ else:
+ # we explicitly specified @property("debug", False")
+ # thus we do not want to have any debugging output.
+ try:
+ del env["SYNCEVOLUTION_DEBUG"]
+ except KeyError:
+ # there was no such key anyway
+ pass
+
+ self.storedenv = env
# can be set by a test to run additional tests on the content
# of the D-Bus log
@@ -3551,14 +3562,16 @@ class TestCmdline(unittest.TestCase, DBusUtil):
# but rather the environment passed to a previous call to
# subprocess.Popen() (which will fail if the previous test ran
# with an environment which had SYNCEVOLUTION_DEBUG set).
+ cmdline_env = copy.deepcopy(self.storedenv)
if env == None:
- env=os.environ
+ env = {}
+ cmdline_env.update(env)
if preserveOutputOrder:
s = subprocess.Popen(a, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
- env=env)
+ env=cmdline_env)
else:
s = subprocess.Popen(a, stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
- env=env)
+ env=cmdline_env)
out, err = s.communicate()
if expectSuccess and s.returncode != 0:
result = 'syncevolution command failed.\nOutput:\n%s' % out
>
>> --
>> Best Regards, Patrick Ohly
>>
>> The content of this message is my personal opinion only and although
>> I am an employee of Intel, the statements I make here in no way
>> represent Intel's position on the issue, nor am I authorized to speak
>> on behalf of Intel on this matter.
>>
>>
_______________________________________________
SyncEvolution mailing list
[email protected]
http://lists.syncevolution.org/listinfo/syncevolution
