Hi Dan,
See comments inline.
Dan Steinicke wrote:
Hi Brian,
I personally went through the commit logs for the changes to the tests
in the old framework and updated the
new tests with these same changes. Your i18n changes were the bulk of
this (including adding the uw() method)
but there were also a number of changes from other devs. The amount
of effort that must now be expended to maintain
two sets of tests is the main reason I am now pushing for moving
quickly forward to using the new tests. Once
the new tests are running on tinderbox without problems the plan is to
stop maintaining the old tests.
+1 makes sense to migrate as soon as possible.
The new framework is mainly about how the tests are run and logged,
with little change to the function of the tests themselves.
I don't believe we have made any special efforts to adapt the CATS-0.2
framework for i18n testing specifically, other than the
new framework being more flexible and easier to develop tests for. My
first thought on this is that we should
continue to push ahead to get the new framework working on tinderbox
and once that is accomplished look work on the
modifications you mention to test for i18n issues.
Sounds good!
I would welcome your input on the new framework and its suitability to
testing i18n issues and expect to work closely with you to develop
this functionality in the future. (links to docs in original msg)
Yes, we will need to work together to create a good i18n testing framework.
If you wish to continue this discussion, can we start a new thread?
This an important issue but one that is pretty far afield from
my original request seeking comment on the new logger output.
Yes lets start a new thread called: "Internationalization Requirements
for CATS testing Framework".
At this stage it makes sense to complete your CATS migration work. I
will review the new framework to make
sure that the i18n changes from CATS-0.1 all got ported to CATS-0.2.
At that point we can start talking about additional i18n requirements
for CATS-0.3.
Thanks,
Brian
Dan
Brian Kirsch wrote:
Hi Dan,
It is very important that CATS-0.2 be design with
internationalization support in mind.
The most common issues / bugs that arise related to i18n are subtle
and a good testing
framework is essential.
Could you detail what work / thought has gone in to CATS-0.2
regarding i18n and what is left to be done.
Specifically in CATS-0.1 I went through and wrapped all displayable
strings in uw() methods from the
i18n.tests package. The uw method inserted a semi-random unicode
character at the beginning of the
displayable string and a unicode character at the end of the
displayable string.
Adding this uw() wrap was key to finding many subtle bugs including
incorrectly logging
unicode characters and not encoding unicode to bytes when using the
Python file system API's.
It even uncovered issues with the CATS-0.1 framework itself including
a few places where items were
converted to strings i.e. str(item) which is a no no!
Using unicode strings is only a small aspect of an i18n testing
framework.
In CATS I would like the framework to provide support for running in
different locales and of course timezones.
For each locale the expected output for many operations will change.
For example a date / time string will
be different, the first day or the week in a calendar can be
different, The UI menu labels and auto generated
collection names will be different.
I am happy to sync up with you guys but I do want to make sure at
this stage we are writing frameworks that
meet the needs of Chandler.
Almost all of the Internationalization work will be completed in the
Alpha 4 time frame.
The testing framework is the key to preventing regression bugs in
i18n as we drive towards Chandler 1.0.
Thanks,
Brian
Dan Steinicke wrote:
We are getting close to being able to switch over to the new test
framework (CATS-0.2). As it is currently implemented the output
from the tests will be quite different from the current test output.
Please take a moment to look over the test output samples below and
let us know what you think so we can address peoples concerns before
the new framework goes live, rather than after.
Two sample outputs are shown below, one where a test fails without a
traceback, another of a failure with a traceback.
For more info on the new framework see :
general docs
http://wiki.osafoundation.org/bin/view/Projects/ChandlerAutomatedTestSystemZeroPointTwo
writing new tests
http://wiki.osafoundation.org/bin/view/Projects/WritingChandlerAutomatedTestsWithCATSZeroPointTwo
Thanks
Dan
############# Here is sample output from do_tests where one tests
fails without a traceback:
Test Report;
*Suite ""ChandlerTestSuite"" Failed :: Total Time ""0:04:55.434000""
:: Comment ""None""
**Test ""TestSwitchTimezone"" Failed :: Total Time
""0:00:00.931000"" :: Comment ""None
None""
***Action ""CheckBlockVisibility"" Failed :: Total Time ""0:00:00""
:: Comment ""(On EditTimeZone Visibility) || detail view = Fa
lse ; expected value = True""
****Report ""(On EditTimeZone Visibility) || detail view = False ;
expected value = True"" Failed :: Comment ""None""
***Action ""CheckEditableBlock"" Failed :: Total Time ""0:00:00"" ::
Comment ""(On EditTimeZone Checking) || detail view value =
Floating ; expected value = US/Pacific""
****Report ""(On EditTimeZone Checking) || detail view value =
Floating ; expected value = US/Pacific"" Failed :: Comment ""None"
"
***Action ""CheckBlockVisibility"" Failed :: Total Time ""0:00:00""
:: Comment ""(On EditTimeZone Visibility) || detail view = Fa
lse ; expected value = True""
****Report ""(On EditTimeZone Visibility) || detail view = False ;
expected value = True"" Failed :: Comment ""None""
***Action ""CheckEditableBlock"" Failed :: Total Time ""0:00:00"" ::
Comment ""(On EditTimeZone Checking) || detail view value =
Floating ; expected value = US/Pacific""
****Report ""(On EditTimeZone Checking) || detail view value =
Floating ; expected value = US/Pacific"" Failed :: Comment ""None"
"
$Suites run=1, pass=0, fail=1 :: Tests run=26, pass=25, fail=1 ::
Actions run=367, pass=363, fail=4 :: Reports run=556, pass=552,
fail=4
#TINDERBOX# Testname = ChandlerTestSuite
#TINDERBOX# Time elapsed = 0:04:55.434000 (seconds)
#TINDERBOX# Status = FAILED
- + - + - + - + - + - + - + - + - + - + - + - + - + - + - +
The following tests failed
(debug)C:\cygwin\home\Dan\chandler\tools\cats\Functional\FunctionalTestSuite.py
(release)C:\cygwin\home\Dan\chandler\tools\cats\Functional\FunctionalTestSuite.py
########## here is some sample output of a test failing with a
traceback
Test Report;
*Suite ""ChandlerTestSuite"" Failed :: Total Time ""0:00:06.139000""
:: Comment ""None""
**Test ""TestCauseTrace"" Failed :: Total Time ""0:00:00.040000"" ::
Comment ""None
Test Failure due to traceback
Traceback (most recent call last):
File "C:\cygwin\home\Dan\chandler\tools\cats\framework\runTests.py",
line 51, in run_tests
test.runTest()
File
"C:\cygwin\home\Dan\chandler\tools\cats\framework\ChandlerTestCase.py",
line 68, in runTest
self.startTest()
File
"C:\cygwin\home\Dan\chandler\tools\cats\Functional\TestCauseTrace.py",
line 24, in startTest
1/0
ZeroDivisionError: integer division or modulo by zero
""
***Action ""Divide by zero "" Failed :: Total Time
""0:00:00.040000"" :: Comment ""None""
****Report ""Action Failure due to traceback"" Failed :: Comment
""None""
$Suites run=1, pass=0, fail=1 :: Tests run=3, pass=2, fail=1 ::
Actions run=5, pass=4, fail=1 :: Reports run=28, pass=27, fail=1
#TINDERBOX# Testname = ChandlerTestSuite
#TINDERBOX# Time elapsed = 0:00:06.139000 (seconds)
#TINDERBOX# Status = FAILED
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
--
Brian Kirsch
Internationalization Architect/ Mail Service Engineer
Open Source Applications Foundation
543 Howard Street 5th Floor
San Francisco, CA 94105
http://www.osafoundation.org
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
