Re: [Trac] New plugins: a true Test Case Manager
Hi Olemis, Cool ! It's very nice to realize that you're really enthusiastic about developing the plugin . Hopefully I will be able to adapt it to my own needs sooner or later (I already bought the hammer ;o) XD However I'd really like to have an RPC interface to perform these actions (because in my deployment environment there will be external tools e.g. Hudson, ... and a BPMS that will be in «touch» with the test case manager). I can dedicate some time to get this done, and maintain the RPC handlers. I'd really like to join | contribute to the project . The major difficulty to do it (from my perspective ;o) is that the VCS is SVN (and I cannot interact with SVN repos since long time ago :-( ... ) . If the switch to a DVCS like bzr, hg, git is not an option , as a workaround , I'll try to create (somehow) an HgSVN repos and publish it as well as a patch queue @ Bitbucket . I'll develop patches and send them back to you . Is it ok ? I chose SVN just because it is the one used by both sourceforge and track-hacks... but I too can't access it from the job, have to do it once back home. If you manage to do the same, that may be an option. Otherwise, just send me the patches and I'll do the merge. If you give me your account, I may allow you to checkin code into sourcefprge. PS: Could you please register the plugin in PyPI ? It's already there from the start :-) Search for TestManager The supported statuses are currently: - TO_BE_TESTED - SUCCESSFUL - FAILED Support for custom test results may be an important addition for future versions (there are lots of threads about this at TiP ML ;o) . An approach to get this done would be to consider test statuses as dumb plain text identifiers, and give them some meaning inside Trac using workflow actions (in a similar way to tickets actions). Yes, I agree with this. It may be the next stuff into the pipeline :-) - There are traceability relationships between TCs and tickets (more technically speaking, requirements tracked by tickets ;o) so that it's possible to let's say know what TCs should be run on changing a ticket state manually | by workflow or whatever . The traceability is currently only from a Ticket top the corresponding Test Case, but not the other way around. Ok, but ideally it would be nice to have some kind of traceability matrices (probably implemented as sparse matrices ;o) and custom traceability relations. The traceability model in use (in this concrete scenario) is based on Requisite Pro traceability matrices, but it would be nice to have something like that inside Trac ;o). Of course , this could be handled at a higher level of abstraction ;o) You can open a Ticket and have a traceback to the (e.g. failed) Test Case as follows: Open a Ticket on a Test Case: Whether you deploy TracTicketTemplatePlugin or not, you can get the following URL, where testCaseNumber is the Test Case complete path, planid is the Test Plan ID and planName is its name [...] Do you use custom ticket fields ? I have an Open a ticket for this test case button on the test case (or test case in plan). Clickin on it, I open a new ticket for editing with a summary recalling the test case title and a wiki reference link in the ticket description, allowing the user to navigate back to the corresponding test case. See below how to allow for programmatic reference. - Tickets , test cases , build jobs and everything participate in a workflow , even if there are multiple tools involved. I get the idea, but maybe orchestrating different tools in a workflow should be the task of a different tool (a process server?). Well as I mentioned before in this particular scenario there are two concurrent workflows installed . Trac built-in workflow orchestrates *EVERYTHING* related to software dev process and part of QA. Another BPEL-WS enabled tool considers Trac (and it's workflow ;o) as a block of a wider enterprise process. It encloses other areas (e.g. CRM, HRM, ...) and interacts with Trac via RPC . Cool. - How extensible are TC management components ? I haven't currently defined any extension points... do you guys have any idea what could be useful here (besides custom test case execution verdicts)? I'll take a look and suggest concrete extension points. I just need to prepare the DVCS stuff ;o) I already have received some other ideas for extensions, like custom properties for test plans (platform, ...). - Wouldn't it be nice to have workflow actions related to TC management ? Being test cases (and catalogs) actually Wiki pages gives us for free the versioning and history capabilities. I have also added with this release custom security permissions for test case and test plan management. What else may be in a workflow then, approval processes? Notifications? E.g. consider the case when you
Re: [Trac] New plugins: a true Test Case Manager
Hi, the TestCaseManager seems to really beat any other solutions - it's great! But I'm looking just one feature - which maybe already exists inside the plugin: Each test case gets a status (works great) and a comment (didn't found an input field - do I have to open a ticket for comments?) from the tester. The goal for a software test is to get a report, where all tests are listed with their result and the comments of the tester. Based on this the developer / reviewer creates tickets for failed tests, if needed. Further, a test plan / report must contain informations about the tested subject and environment (app version, build date, test date, name of tester, used operation system...) - this is very important to compare test reports by app version and to trac changes to an app (e.g. to find regressions). If my request is not too far away from the plugin's concept and goal, I would post this as a feature request. Thanks for this great plugin! I'm looking forward to use it for my daily work. Currently I try to port the plugin to Python 2.4 (our production server is very old - hope to get a newer one within this year). ciao, Chris -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
Re: [Trac] New plugins: a true Test Case Manager
Hi Christian, thanks for the kudos :-) Mmmh, currently there's no way to add comments when changing the status of a test case, other than opening a ticket for it... but this is undesirable in case of success, of course. This is something that may come into the base code, I think. About your second point, instead, adding sevral environment information to a test plan, the first thing that comes to my mind is to provide extension points to the plugin, instead of including all sorts of information with the base version. This way, anyone could add the information more pertinent to his/her working environment and needs. I'll think about which extension points may be added, and I'm open to suggestions of course. Probably, supporting a list of custom properties associated to each record in the testplans table may be one way... Also, please, feel free to open enhancements requests, and to provide patches if you wish to :-) Thanks, ciao, Roberto 2010/8/25 Christian Dähn da...@asinteg.de Hi, the TestCaseManager seems to really beat any other solutions - it's great! But I'm looking just one feature - which maybe already exists inside the plugin: Each test case gets a status (works great) and a comment (didn't found an input field - do I have to open a ticket for comments?) from the tester. The goal for a software test is to get a report, where all tests are listed with their result and the comments of the tester. Based on this the developer / reviewer creates tickets for failed tests, if needed. Further, a test plan / report must contain informations about the tested subject and environment (app version, build date, test date, name of tester, used operation system...) - this is very important to compare test reports by app version and to trac changes to an app (e.g. to find regressions). If my request is not too far away from the plugin's concept and goal, I would post this as a feature request. Thanks for this great plugin! I'm looking forward to use it for my daily work. Currently I try to port the plugin to Python 2.4 (our production server is very old - hope to get a newer one within this year). ciao, Chris -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.comtrac-users%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/trac-users?hl=en. -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
Re: [Trac] New plugins: a true Test Case Manager
On Wed, Aug 18, 2010 at 10:05 AM, Roberto Longobardi secc...@gmail.com wrote: Hi Olemis, :o) I have introduced the Test Plans feature in the plugin. Cool ! It's very nice to realize that you're really enthusiastic about developing the plugin . Hopefully I will be able to adapt it to my own needs sooner or later (I already bought the hammer ;o) The scenario you describe below would now look like as follows. - TCs are *discovered* (generated ?) from source code (e.g. ) committed to a repos and they are using static code analysis, abstract models , XML specs ... - Test execution , etc , is performed by an external CI tool (currently Hudson + Bitten ...) In this scenario , I'd like to be able to - Retrieve test results from CI tool using an API - Trigger build jobs on changing TC states or when aforementioned test campaigns (which will be mapped to in Hudson builds) are started , explicitly executed . - ... well some other , but have no time to say it all You can create test catalogs and cases programmatically by means of the following http requests: [...] Create a root test catalog: [...] Create a sub-catalog: [...] Create a Test Case: [...] Create a Test Plan from a specific catalog: [...] Set a Test Case execution verdict, in the context of a Test Plan: [...] This is really good news !!! However I'd really like to have an RPC interface to perform these actions (because in my deployment environment there will be external tools e.g. Hudson, ... and a BPMS that will be in «touch» with the test case manager). I can dedicate some time to get this done, and maintain the RPC handlers. I'd really like to join | contribute to the project . The major difficulty to do it (from my perspective ;o) is that the VCS is SVN (and I cannot interact with SVN repos since long time ago :-( ... ) . If the switch to a DVCS like bzr, hg, git is not an option , as a workaround , I'll try to create (somehow) an HgSVN repos and publish it as well as a patch queue @ Bitbucket . I'll develop patches and send them back to you . Is it ok ? PS: Could you please register the plugin in PyPI ? The supported statuses are currently: - TO_BE_TESTED - SUCCESSFUL - FAILED Support for custom test results may be an important addition for future versions (there are lots of threads about this at TiP ML ;o) . An approach to get this done would be to consider test statuses as dumb plain text identifiers, and give them some meaning inside Trac using workflow actions (in a similar way to tickets actions). - There are traceability relationships between TCs and tickets (more technically speaking, requirements tracked by tickets ;o) so that it's possible to let's say know what TCs should be run on changing a ticket state manually | by workflow or whatever . The traceability is currently only from a Ticket top the corresponding Test Case, but not the other way around. Ok, but ideally it would be nice to have some kind of traceability matrices (probably implemented as sparse matrices ;o) and custom traceability relations. The traceability model in use (in this concrete scenario) is based on Requisite Pro traceability matrices, but it would be nice to have something like that inside Trac ;o). Of course , this could be handled at a higher level of abstraction ;o) You can open a Ticket and have a traceback to the (e.g. failed) Test Case as follows: Open a Ticket on a Test Case: Whether you deploy TracTicketTemplatePlugin or not, you can get the following URL, where testCaseNumber is the Test Case complete path, planid is the Test Plan ID and planName is its name [...] Do you use custom ticket fields ? - Tickets , test cases , build jobs and everything participate in a workflow , even if there are multiple tools involved. I get the idea, but maybe orchestrating different tools in a workflow should be the task of a different tool (a process server?). Well as I mentioned before in this particular scenario there are two concurrent workflows installed . Trac built-in workflow orchestrates *EVERYTHING* related to software dev process and part of QA. Another BPEL-WS enabled tool considers Trac (and it's workflow ;o) as a block of a wider enterprise process. It encloses other areas (e.g. CRM, HRM, ...) and interacts with Trac via RPC . - How extensible are TC management components ? I haven't currently defined any extension points... do you guys have any idea what could be useful here (besides custom test case execution verdicts)? I'll take a look and suggest concrete extension points. I just need to prepare the DVCS stuff ;o) - Wouldn't it be nice to have workflow actions related to TC management ? Being test cases (and catalogs) actually Wiki pages gives us for free the versioning and history capabilities. I have also added with this release custom security permissions for test case and test plan management. What else may
Re: [Trac] New plugins: a true Test Case Manager
Hi Stephen, the problem you're seeing is caused by the upgrade db code trying to create the wiki page /TC programmatically, by inserting directly into the wiki table. I now changed this code to use the Wiki class instead of writing into the table, but haven't released this code yet. BTW, this wiki page is used as the root of all test catalogs. The error is complaining about you having already a wiki page with that name... this is surprising, unless you had already installed the plugin (maybe a lower version) some time before. Uninstalling the plugin does not remove this page, so upon uninstall you should delete the TC wiki page manually, if you plan to reinstall the plugin eventually. In a fresh environment this should never happen, so you can try and deploy the plugin into a new environment if you can. Please, let me know if this does not fix the problem and I'll try to provide you with a script that drops the new tables so to be able to rerun the update db once again. Ciao, Roberto 2010/8/19 Stephen Bash b...@genarts.com Roberto- Everything compiled fine. We ran into an issue during the database upgrade. I've included the trac log output below. Now trac-admin reports the database is up to date, so I can't reproduce the issue anymore. Thanks, Stephen Trac DEBUG log: 2010-08-18 15:29:58,752 Trac[env] INFO: testmanager.model.TestManagerModelProvider upgrading... 2010-08-18 15:29:58,752 Trac[model] DEBUG: CREATE TABLE testconfig ( propname text PRIMARY KEY, value text ); 2010-08-18 15:29:59,476 Trac[model] DEBUG: CREATE INDEX testconfig_propname_idx ON testconfig (propname); 2010-08-18 15:29:59,484 Trac[model] DEBUG: CREATE TABLE testcases ( id text, planid text, status text, UNIQUE (id,planid) ); 2010-08-18 15:29:59,496 Trac[model] DEBUG: CREATE INDEX testcases_id_idx ON testcases (id); 2010-08-18 15:29:59,506 Trac[model] DEBUG: CREATE INDEX testcases_planid_idx ON testcases (planid); 2010-08-18 15:29:59,519 Trac[model] DEBUG: CREATE TABLE testcasehistory ( id text, planid text, time integer, author text, status text, UNIQUE (id,planid,time) ); 2010-08-18 15:29:59,530 Trac[model] DEBUG: CREATE INDEX testcasehistory_id_planid_time_idx ON testcasehistory (id,planid,time); 2010-08-18 15:29:59,543 Trac[model] DEBUG: CREATE TABLE testplans ( planid text PRIMARY KEY, catid text, catpath text, name text, author text, time integer ); 2010-08-18 15:29:59,556 Trac[model] DEBUG: CREATE INDEX testplans_planid_idx ON testplans (planid); 2010-08-18 15:29:59,574 Trac[model] DEBUG: CREATE INDEX testplans_catid_idx ON testplans (catid); 2010-08-18 15:29:59,610 Trac[console] ERROR: Exception in trac-admin command: Traceback (most recent call last): File build/bdist.linux-i686/egg/trac/admin/console.py, line 107, in onecmd rv = cmd.Cmd.onecmd(self, line) or 0 File /usr/lib/python2.5/cmd.py, line 218, in onecmd return self.default(line) File build/bdist.linux-i686/egg/trac/admin/console.py, line 257, in default return cmd_mgr.execute_command(*args) File build/bdist.linux-i686/egg/trac/admin/api.py, line 123, in execute_command return f(*fargs) File build/bdist.linux-i686/egg/trac/env.py, line 780, in _do_upgrade self.env.upgrade(backup=no_backup is None) File build/bdist.linux-i686/egg/trac/env.py, line 523, in upgrade with_transaction(self)(participant.upgrade_environment) File build/bdist.linux-i686/egg/trac/db/api.py, line 77, in transaction_wrapper fn(ldb) File build/bdist.linux-i686/egg/testmanager/model.py, line 60, in upgrade_environment self._upgrade_db(db) File build/bdist.linux-i686/egg/testmanager/model.py, line 110, in _upgrade_db ' ', '', 0)) File build/bdist.linux-i686/egg/trac/db/util.py, line 65, in execute return self.cursor.execute(sql_escape_percent(sql), args) File build/bdist.linux-i686/egg/trac/db/sqlite_backend.py, line 78, in execute result = PyFormatCursor.execute(self, *args) File build/bdist.linux-i686/egg/trac/db/sqlite_backend.py, line 56, in execute args or []) File build/bdist.linux-i686/egg/trac/db/sqlite_backend.py, line 48, in _rollback_on_error return function(self, *args, **kwargs) IntegrityError: columns name, version are not unique - Original Message - From: Roberto Longobardi secc...@gmail.com To: Gary Oberbrunner ga...@genarts.com Cc: trac-users@googlegroups.com, Stephen Bash b...@genarts.com Sent: Thursday, August 19, 2010 12:32:48 AM Subject: Re: [Trac] New plugins: a true Test Case Manager Hi Gary, I would like to make it compatible with 0.12, but didn't have a chance to work on it yet. I'll take a look today at what's wrong and see if it's something quick. Ciao, Roberto 2010/8/18 Gary Oberbrunner ga...@genarts.com On 8/18/2010 10:05 AM, Roberto Longobardi wrote: Hi Olemis, I have introduced the Test Plans
Re: [Trac] New plugins: a true Test Case Manager
...@gmail.com To: Gary Oberbrunner ga...@genarts.com Cc: trac-users@googlegroups.com, Stephen Bash b...@genarts.com Sent: Thursday, August 19, 2010 12:32:48 AM Subject: Re: [Trac] New plugins: a true Test Case Manager Hi Gary, I would like to make it compatible with 0.12, but didn't have a chance to work on it yet. I'll take a look today at what's wrong and see if it's something quick. Ciao, Roberto 2010/8/18 Gary Oberbrunner ga...@genarts.com On 8/18/2010 10:05 AM, Roberto Longobardi wrote: Hi Olemis, I have introduced the Test Plans feature in the plugin http://trac-hacks.org/wiki/TestManagerForTracPlugin . We tried to install this here but it appears to be incompatible with Trac 0.12. Are you planning to update it to 0.12? -- Gary -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
Re: [Trac] New plugins: a true Test Case Manager
Hi Olemis, I have introduced the Test Plans feature in the pluginhttp://trac-hacks.org/wiki/TestManagerForTracPlugin . The scenario you describe below would now look like as follows. - TCs are *discovered* (generated ?) from source code (e.g. ) committed to a repos and they are using static code analysis, abstract models , XML specs ... - Test execution , etc , is performed by an external CI tool (currently Hudson + Bitten ...) In this scenario , I'd like to be able to - Retrieve test results from CI tool using an API - Trigger build jobs on changing TC states or when aforementioned test campaigns (which will be mapped to in Hudson builds) are started , explicitly executed . - ... well some other , but have no time to say it all Q: - Considering scenarios like that one , is it possible to use interfaces so as , let's say , retrieve, synch TC history from other sources e.g. Hudson build history You can create test catalogs and cases programmatically by means of the following http requests: *Create a root test catalog:* Get the following URL: http://yourserver/yourproject/testcreate?type=catalogpath=TCtitle=My%20new%20catalog this will assign a unique ID (the number 0 in the URL below) to the new catalog and redirect to a Wiki edit page: http://yourserver/yourproject/wiki/TC_TT0?action=edittext=%3D%3D+My+new+catalog+%3D%3D that you can simply post the form to submit the page and thus create the catalog. *Create a sub-catalog:* Get the wollowing URL, where path is the name of the parent catalog. http://yourserver/yourproject/testcreate?type=catalogpath=TC_TT0title=My%20sub%20catalog this will assign a unique ID (the number 1 in the URL below) to the new catalog and redirect to a Wiki edit page: http://yourserver/yourproject/wiki/TC_TT0_TT1?action=edittext=%3D%3D+My+sub+catalog+%3D%3D that you can simply post the form to submit the page and thus create the sub-catalog. *Create a Test Case:* Get the wollowing URL, where path is the name of the parent (sub-)catalog. http://yourserver/yourproject/testcreate?type=testcasepath=TC_TT0_TT1title=My%20new%20Test%20Case this will assign a unique ID (the number 0 in the URL below) to the new test case and redirect to a Wiki edit page: http://yourserver/yourproject/wiki/TC_TT0_TT1_TC0?action=edittext=%3D%3D+My+new+Test+Case+%3D%3D that you can simply post the form to submit the page and thus create the test case. You can create a new Test Plan (e.g. for each nightly build) programmatically as follows. *Create a Test Plan from a specific catalog:* Get the wollowing URL, where path is the name of the (sub-)catalog to create the test plan against. http://yourserver/yourproject/testcreate?type=testplanpath=TC_TT0title=Test%20Plan%20for%2020100818 this will assign a unique ID (the number 1 in the URL below) to the new test plan and redirect to displaying the Test Plan: http://yourserver/yourproject/wiki/TC_TT0?planid=1 The Test Plan will contain all of the test cases in the specified catalog, with a verdict of Untested. *Set a Test Case execution verdict, in the context of a Test Plan:* Then, you can set the verdict for any test case in the plan, by means of the following. Get the following URL, where id is the Test Case ID and planid is the Test Plan ID: http://yourserver/yourproject/teststatusupdate?id=5planid=1status=SUCCESSFUL The supported statuses are currently: - TO_BE_TESTED - SUCCESSFUL - FAILED - There are traceability relationships between TCs and tickets (more technically speaking, requirements tracked by tickets ;o) so that it's possible to let's say know what TCs should be run on changing a ticket state manually | by workflow or whatever . The traceability is currently only from a Ticket top the corresponding Test Case, but not the other way around. You can open a Ticket and have a traceback to the (e.g. failed) Test Case as follows: *Open a Ticket on a Test Case:* Whether you deploy TracTicketTemplatePlugin or not, you can get the following URL, where testCaseNumber is the Test Case complete path, planid is the Test Plan ID and planName is its name: http://yourserver/yourproject/newticket?testCaseNumber=TC_TT0_TC0planId=1planName=Test%20Plan%20for%2020100818description=Test%20Case:%20[wiki:TC_TT0_TC0],%20Test%20Plan:%20Test%20Plan%20for%2020100818%20(1) this will open a Ticket edit page, with the Test Case-in-Test Plan hyperlink already added to the ticket description (as Wiki page reference). You can simply post the form to create the Ticket. Probably you can directly POST a pre-built form to do everything with a single call, but I didn't test this... - Tickets , test cases , build jobs and everything participate in a workflow , even if there are multiple tools involved. I get the idea, but maybe orchestrating different tools in a workflow should be the task of a different tool (a process server?). - How extensible are TC management
Re: [Trac] New plugins: a true Test Case Manager
On 8/18/2010 10:05 AM, Roberto Longobardi wrote: Hi Olemis, I have introduced the Test Plans feature in the plugin http://trac-hacks.org/wiki/TestManagerForTracPlugin. We tried to install this here but it appears to be incompatible with Trac 0.12. Are you planning to update it to 0.12? -- Gary -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
Re: [Trac] New plugins: a true Test Case Manager
Hi Gary, I would like to make it compatible with 0.12, but didn't have a chance to work on it yet. I'll take a look today at what's wrong and see if it's something quick. Ciao, Roberto 2010/8/18 Gary Oberbrunner ga...@genarts.com On 8/18/2010 10:05 AM, Roberto Longobardi wrote: Hi Olemis, I have introduced the Test Plans feature in the plugin http://trac-hacks.org/wiki/TestManagerForTracPlugin. We tried to install this here but it appears to be incompatible with Trac 0.12. Are you planning to update it to 0.12? -- Gary -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
Re: [Trac] New plugins: a true Test Case Manager
Quoting Roberto Longobardi secc...@gmail.com: The first is a true test case manager plugin, since I couldn't find anything suitable before, and my users wanted to have test case management and bug tracking integrated in the same place: http://trac-hacks.org/wiki/TestManagerForTracPlugin OK, here my first observations after a short test: - This testing framework is much more to my personal needs than other, similar plugins I tried in the past. Good! - It would be nice, if a test case result is not immediately recorded on mouse click, but if there were a save result button. It is too easy to click accidently on the coloured bullet. - Maybe the Status change history should be reversed, showing the latest (= most important) result first. - This is only a matter of taste, but I'm used to a different terminology: A test catalog for one project would be a test suite (TS), a sub catalog (or sub sub catalog) would be a test group (TG), and a single test would be a test case (TC). - When one creates a ticket from a (failed) test, it would be nice, if the new ticket already defaults to a useful subject (Failed test: Basic Sleep - Sleeping Monster). This can be easily achieved by using the link: .../newticket?summary=Failed%20test:%20Basic%20Sleep... - It would be cool, if one could define a set of possible verdicts. You have Successful, Untested, and Failed, which is OK for many purposes. But if you do tests according to one of the many standards, you might have different needs. E.g. ISO-9646 has five verdicts: None = no result yet, untested Pass = good, successful Inconc = inconclusive, unclear Fail = bad, failed Error = there was an error in performing the test If you test according to POSIX 1003.3 you have the verdicts: PASS = good, successful FAIL = bad, failed UNRESOLVED = inconclusive, unclear UNTESTED = no result yet, untested Some testing frameworks add XFAIL (= expected fail), UPASS (= unexpected pass) and UNSUPPORTED (= the implementation under test doesn't support a feature) to the POSIX verdicts. - In the long run, it would be useful (but not trivial to implement) to differentiate between the definition of the tests and running a test campaign. An example: I have a software product, a plugin for Trac. Before doing a release 1.0 I have to perform some interactive tests. Therefore I start a new test campaign with svn revision 1234, where all test verdicts Untested. Let's say, one or two tests failed. I conclude the test campaign unsuccessful. Now I have to fix the bugs and have to start a new test campaign, again with all test verdicts set to Untested. Keep on the good work, thanks! -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
Re: [Trac] New plugins: a true Test Case Manager
Hi Martin, thanks for your review and for the kudos. I have opened some enhancement requests on trac-hacks to keep track of your comments. Just one thing about openening a ticket on the current test case. I had tried your way in the first place, but since we also deploy the TracTicketTemplatePluginhttp://trac-hacks.org/wiki/TracTicketTemplatePlugin, it didn't work (this plugin replaces the whole ticket body with the template upon page load). So what I did is to patch the abov plugin to accept parameters on the URL and substitute them in the template. You may have noticed that a couple of such are passed to the newticket page. This way I actually end up with a ticket that has a hyperlink (wiki) reference to the test case. I'll try to make this patch go into the plugin main stream, and try to support the environment where such plugin isn't found. About your last point, sure it is something I was thinking about, but as you say not easy to implement... Anyway, since the data model for the test results is completely separated from the one for the test cases themselves (i.e. the wiki table), it may not be SO hard to do. Help in this direction is welcome of course :-) Thanks, ciao Roberto 2010/8/12 W. Martin Borgert deba...@debian.org Quoting Roberto Longobardi secc...@gmail.com: The first is a true test case manager plugin, since I couldn't find anything suitable before, and my users wanted to have test case management and bug tracking integrated in the same place: http://trac-hacks.org/wiki/TestManagerForTracPlugin OK, here my first observations after a short test: - This testing framework is much more to my personal needs than other, similar plugins I tried in the past. Good! - It would be nice, if a test case result is not immediately recorded on mouse click, but if there were a save result button. It is too easy to click accidently on the coloured bullet. - Maybe the Status change history should be reversed, showing the latest (= most important) result first. - This is only a matter of taste, but I'm used to a different terminology: A test catalog for one project would be a test suite (TS), a sub catalog (or sub sub catalog) would be a test group (TG), and a single test would be a test case (TC). - When one creates a ticket from a (failed) test, it would be nice, if the new ticket already defaults to a useful subject (Failed test: Basic Sleep - Sleeping Monster). This can be easily achieved by using the link: .../newticket?summary=Failed%20test:%20Basic%20Sleep... - It would be cool, if one could define a set of possible verdicts. You have Successful, Untested, and Failed, which is OK for many purposes. But if you do tests according to one of the many standards, you might have different needs. E.g. ISO-9646 has five verdicts: None = no result yet, untested Pass = good, successful Inconc = inconclusive, unclear Fail = bad, failed Error = there was an error in performing the test If you test according to POSIX 1003.3 you have the verdicts: PASS = good, successful FAIL = bad, failed UNRESOLVED = inconclusive, unclear UNTESTED = no result yet, untested Some testing frameworks add XFAIL (= expected fail), UPASS (= unexpected pass) and UNSUPPORTED (= the implementation under test doesn't support a feature) to the POSIX verdicts. - In the long run, it would be useful (but not trivial to implement) to differentiate between the definition of the tests and running a test campaign. An example: I have a software product, a plugin for Trac. Before doing a release 1.0 I have to perform some interactive tests. Therefore I start a new test campaign with svn revision 1234, where all test verdicts Untested. Let's say, one or two tests failed. I conclude the test campaign unsuccessful. Now I have to fix the bugs and have to start a new test campaign, again with all test verdicts set to Untested. Keep on the good work, thanks! -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
Re: [Trac] New plugins: a true Test Case Manager
On Thu, Aug 12, 2010 at 9:05 AM, Roberto Longobardi secc...@gmail.com wrote: 2010/8/12 W. Martin Borgert deba...@debian.org Quoting Roberto Longobardi secc...@gmail.com: [...] BTW, what's wrong with the TC - ticket mapping ? I've seen similar mappings in other products ... - In the long run, it would be useful (but not trivial to implement) to differentiate between the definition of the tests and running a test campaign. An example: I have a software product, a plugin for Trac. Before doing a release 1.0 I have to perform some interactive tests. Therefore I start a new test campaign with svn revision 1234, where all test verdicts Untested. Let's say, one or two tests failed. I conclude the test campaign unsuccessful. Now I have to fix the bugs and have to start a new test campaign, again with all test verdicts set to Untested. [...] About your last point, sure it is something I was thinking about, but as you say not easy to implement... Anyway, since the data model for the test results is completely separated from the one for the test cases themselves (i.e. the wiki table), it may not be SO hard to do. Help in this direction is welcome of course :-) Here I go . I'm looking for some kind of tool like this one but allowing to integrate TC management with CI tools . My environment will look like this (i.e. part of this is done but the whole will work in a near future, this is kind of what I want to get ;o). - TCs are *discovered* (generated ?) from source code (e.g. ) committed to a repos and they are using static code analysis, abstract models , XML specs ... - Test execution , etc , is performed by an external CI tool (currently Hudson + Bitten ...) - There are traceability relationships between TCs and tickets (more technically speaking, requirements tracked by tickets ;o) so that it's possible to let's say know what TCs should be run on changing a ticket state manually | by workflow or whatever . - Tickets , test cases , build jobs and everything participate in a workflow , even if there are multiple tools involved. In this scenario , I'd like to be able to - Retrieve test results from CI tool using an API - Trigger build jobs on changing TC states or when aforementioned test campaigns (which will be mapped to in Hudson builds) are started , explicitly executed . - ... well some other , but have no time to say it all Q: - Considering scenarios like that one , is it possible to use interfaces so as , let's say , retrieve, synch TC history from other sources e.g. Hudson build history - How extensible are TC management components ? - Wouldn't it be nice to have workflow actions related to TC management ? - Wouldn't it be nice to provide RPC handlers to perform remote actions on TC specs (e.g. from inside VCS hooks ;o) Keep on the good work, thanks! Did I already say this ? ;o) -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
[Trac] New plugins: a true Test Case Manager and a Resource planning and reservation
Hi, just to let you know I have uploaded a couple of new plugins on the trac-hacks site that I developed for my trac users. The first is a true test case manager plugin, since I couldn't find anything suitable before, and my users wanted to have test case management and bug tracking integrated in the same place: http://trac-hacks.org/wiki/TestManagerForTracPlugin The second one is a simple plugin to allow for planning and reserving the future use of resources on a time schedule. We are using it to reserve consumable test data (employees in our test database that once tested cannot be used anymore since the test changes their data), but it could be used as well to plan and coordinate the use of test machines, for example. Here is the page on trac-hacks: http://trac-hacks.org/wiki/ResourceReservationPlugin Both are currently only tested on Trac 0.11 and python 2.5. Feedback and bugs/requests for features are both really appreciated, thanks! Roberto -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
Re: [Trac] New plugins: a true Test Case Manager
Hi Roberto, Quoting Roberto Longobardi secc...@gmail.com: The first is a true test case manager plugin, since I couldn't find anything suitable before, and my users wanted to have test case management and bug tracking integrated in the same place: http://trac-hacks.org/wiki/TestManagerForTracPlugin The screenshots look very promising. I found the existing plugins for test management not really up to my needs, so your approach is very welcome. Just two hints for the page http://trac-hacks.org/wiki/TestManagerForTracPlugin: 1. For some reason I cannot download the .ppt attachments. I get an error message: The requested URL /attachment/wiki/TestManagerForTracPlugin/Test Manager plugin for Trac - User Guide part 1.ppt was not found on this server. I will download the files from sourceforge for now. But wouldn't it be better to have the complete description directly in the wiki page? This way people could improve it and you would have version control. 2. To directly display the screenshots, you may use a macro: [[Image(screen2.JPG)]] Cheers -- You received this message because you are subscribed to the Google Groups Trac Users group. To post to this group, send email to trac-us...@googlegroups.com. To unsubscribe from this group, send email to trac-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.
