Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support
Hi Nate, That's awesome; thanks a million! I just took a look at the auto generated emails on galaxy-commits; nicely done. Even if it wasn't much work, I think the general benefit to the quality of Galaxy as a platform and to the developer community is palpable. And definitely thanks for expressly letting me know; that is much appreciated. Happy hacking, Eric From: Nate Coraor [n...@bx.psu.edu] Sent: Tuesday, December 06, 2011 1:04 PM To: Greg Von Kuster Cc: Paniagua, Eric; galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support On Nov 30, 2011, at 6:35 PM, Greg Von Kuster wrote: Eric, We always welcome help from the Galaxy community, so if you are interested in enhancing the Galaxy code, the best way to go about it is to create your own fork off the Galaxy central repo in bitbucket, and when you have something to contribute initiate a pull request for us to review and merge into the central repo. This way you don't have to deal with ongoing merges. A bunch of people have asked recently and it wasn't much work, so I just added a new config param 'cleanup_job' that allows control over when job-related files are cleaned up (always, never, or only on job success). --nate Thanks! On Nov 30, 2011, at 2:24 PM, Paniagua, Eric wrote: Thanks for the quick replies again! Yeah, from a technical standpoint such support is certainly doable. My employer strongly discourages modifying the Galaxy code base too invasively (if at all), which is pretty fair since I'm not in a position to take responsibility for performing future Galaxy upgrades which may have messy merges as a consequence of my tinkering downstream of you. That's primarily the reason I was curious about whether such features were in the works or at least on the horizon at the Galaxy Development Team proper. Anyway, thanks for communicating. Have a great day :) Best, Eric From: Greg Von Kuster [g...@bx.psu.edu] Sent: Wednesday, November 30, 2011 1:22 PM To: Paniagua, Eric Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support On Nov 30, 2011, at 11:11 AM, Paniagua, Eric wrote: Hi Greg, I appreciate your response! Thanks for clarifying the capabilities and limitations of the current functional testing framework. (BTW, did you mean current as in current on galaxy-dist or also current on galaxy-central ?) If I can find the time, these are areas I would be interested in helping enhance. Current on Galaxy central, very likely the same as Galaxy dist. Sorry my diction was unclear; by output files I meant the transient files that a running job is free to create and manipulate in its job_working_directory. (I'm writing transient rather than temporary to avoid confusion with, e.g. files created with tempfile.mkstemp - although conceivably one might want to be able to inspect their contents on failure as well, actually saving them might be a very different problem.) Normally, upon successful or failed job completion the job working directory and anything in it are erased. For certain tools, it could be helpful in debugging if the developer (or Galaxy system administrator) were able to inspect their contents to see, for instance, if they are properly formed. I can disable removal of job working directories globally, but then of course the job working directories also persist for successful jobs, which can add up to a lot of unnecessary storage (the reason they're deleted by default in the first place). I'm not sure how work is divided on your team, but can you tell me (a) if the preceding paragraphs actually clarify anything for you, Yes, the functional test framework does not currently deal with anything in job_working_directory as far as I know. I am not a primary tool developer on the development team, however, so there may be some peripheral test components working in this realm of which I am not aware. My understanding of the use of the job_working_directory is that some files are moved out of it into permanent locations, while others are deleted upon job completion. You should be able to enhance the job code to enable you to inspect certain elements of this directory during job processing, but I'm not sure how difficult this may be. and (b) whether that issue is on the radar of your team and specifically on the radar of the primary developer(s) / maintainer(s) of the testing framework? This issue is not currently anywhere on the development team's radar. Sorry if this is an inconvenience. Thanks again for responding to my email. Best, Eric From: Greg Von Kuster [g...@bx.psu.edu] Sent: Wednesday
Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support
Hello Eric, Submitted by epani...@cshl.edu Hi all, I've read the Wiki page on Writing Functional Tests (http://wiki.g2.bx.psu.edu/Admin/Tools/Writing%20Tests) and I've been looking through test/base and test/functional and I am left with two questions: * Is it possible to write a test to validate metadata directly on an (optionally composite) output dataset? I'm sure this is possible, but it would require enhancements to the current functional test framework. Everything described on the above page is file oriented. I see that there is TwillTestCase.check_metadata_for_string, but as far as I can tell this is a bit nonspecific since it appears to just do a text search on the Edit page. This is correct. I don't yet fully understand the context in which tests run, but is there some way to access a live dataset's metadata directly, either as a dictionary or just as attributes? Or even to get the actual dataset object? Not with the current functional test framework. Doing this would require enhancements to the framework. * Does the test harness support retaining output files only for failed tests? Ideally with a cap on how much output data to save. If not, would this be difficult to configure? I'm not sure what you mean by output files in your question. If you mean output datasets that result from running a functional test for a tool, then I believe there is no difference if the test passed or failed. Thanks, Eric Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support
Hi Greg, I appreciate your response! Thanks for clarifying the capabilities and limitations of the current functional testing framework. (BTW, did you mean current as in current on galaxy-dist or also current on galaxy-central ?) If I can find the time, these are areas I would be interested in helping enhance. Sorry my diction was unclear; by output files I meant the transient files that a running job is free to create and manipulate in its job_working_directory. (I'm writing transient rather than temporary to avoid confusion with, e.g. files created with tempfile.mkstemp - although conceivably one might want to be able to inspect their contents on failure as well, actually saving them might be a very different problem.) Normally, upon successful or failed job completion the job working directory and anything in it are erased. For certain tools, it could be helpful in debugging if the developer (or Galaxy system administrator) were able to inspect their contents to see, for instance, if they are properly formed. I can disable removal of job working directories globally, but then of course the job working directories also persist for successful jobs, which can add up to a lot of unnecessary storage (the reason they're deleted by default in the first place). I'm not sure how work is divided on your team, but can you tell me (a) if the preceding paragraphs actually clarify anything for you, and (b) whether that issue is on the radar of your team and specifically on the radar of the primary developer(s) / maintainer(s) of the testing framework? Thanks again for responding to my email. Best, Eric From: Greg Von Kuster [g...@bx.psu.edu] Sent: Wednesday, November 30, 2011 9:56 AM To: Paniagua, Eric Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [Internal - Galaxy-dev #2159] (New) [galaxy-dev] 2 questions about Galaxy's functional testing support Hello Eric, Submitted by epani...@cshl.edumailto:epani...@cshl.edu Hi all, I've read the Wiki page on Writing Functional Tests (http://wiki.g2.bx.psu.edu/Admin/Tools/Writing%20Tests) and I've been looking through test/base and test/functional and I am left with two questions: * Is it possible to write a test to validate metadata directly on an (optionally composite) output dataset? I'm sure this is possible, but it would require enhancements to the current functional test framework. Everything described on the above page is file oriented. I see that there is TwillTestCase.check_metadata_for_string, but as far as I can tell this is a bit nonspecific since it appears to just do a text search on the Edit page. This is correct. I don't yet fully understand the context in which tests run, but is there some way to access a live dataset's metadata directly, either as a dictionary or just as attributes? Or even to get the actual dataset object? Not with the current functional test framework. Doing this would require enhancements to the framework. * Does the test harness support retaining output files only for failed tests? Ideally with a cap on how much output data to save. If not, would this be difficult to configure? I'm not sure what you mean by output files in your question. If you mean output datasets that result from running a functional test for a tool, then I believe there is no difference if the test passed or failed. Thanks, Eric Greg Von Kuster Galaxy Development Team g...@bx.psu.edumailto:g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support
On Nov 30, 2011, at 11:11 AM, Paniagua, Eric wrote: Hi Greg, I appreciate your response! Thanks for clarifying the capabilities and limitations of the current functional testing framework. (BTW, did you mean current as in current on galaxy-dist or also current on galaxy-central ?) If I can find the time, these are areas I would be interested in helping enhance. Current on Galaxy central, very likely the same as Galaxy dist. Sorry my diction was unclear; by output files I meant the transient files that a running job is free to create and manipulate in its job_working_directory. (I'm writing transient rather than temporary to avoid confusion with, e.g. files created with tempfile.mkstemp - although conceivably one might want to be able to inspect their contents on failure as well, actually saving them might be a very different problem.) Normally, upon successful or failed job completion the job working directory and anything in it are erased. For certain tools, it could be helpful in debugging if the developer (or Galaxy system administrator) were able to inspect their contents to see, for instance, if they are properly formed. I can disable removal of job working directories globally, but then of course the job working directories also persist for successful jobs, which can add up to a lot of unnecessary storage (the reason they're deleted by default in the first place). I'm not sure how work is divided on your team, but can you tell me (a) if the preceding paragraphs actually clarify anything for you, Yes, the functional test framework does not currently deal with anything in job_working_directory as far as I know. I am not a primary tool developer on the development team, however, so there may be some peripheral test components working in this realm of which I am not aware. My understanding of the use of the job_working_directory is that some files are moved out of it into permanent locations, while others are deleted upon job completion. You should be able to enhance the job code to enable you to inspect certain elements of this directory during job processing, but I'm not sure how difficult this may be. and (b) whether that issue is on the radar of your team and specifically on the radar of the primary developer(s) / maintainer(s) of the testing framework? This issue is not currently anywhere on the development team's radar. Sorry if this is an inconvenience. Thanks again for responding to my email. Best, Eric From: Greg Von Kuster [g...@bx.psu.edu] Sent: Wednesday, November 30, 2011 9:56 AM To: Paniagua, Eric Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [Internal - Galaxy-dev #2159] (New) [galaxy-dev] 2 questions about Galaxy's functional testing support Hello Eric, Submitted by epani...@cshl.edumailto:epani...@cshl.edu Hi all, I've read the Wiki page on Writing Functional Tests (http://wiki.g2.bx.psu.edu/Admin/Tools/Writing%20Tests) and I've been looking through test/base and test/functional and I am left with two questions: * Is it possible to write a test to validate metadata directly on an (optionally composite) output dataset? I'm sure this is possible, but it would require enhancements to the current functional test framework. Everything described on the above page is file oriented. I see that there is TwillTestCase.check_metadata_for_string, but as far as I can tell this is a bit nonspecific since it appears to just do a text search on the Edit page. This is correct. I don't yet fully understand the context in which tests run, but is there some way to access a live dataset's metadata directly, either as a dictionary or just as attributes? Or even to get the actual dataset object? Not with the current functional test framework. Doing this would require enhancements to the framework. * Does the test harness support retaining output files only for failed tests? Ideally with a cap on how much output data to save. If not, would this be difficult to configure? I'm not sure what you mean by output files in your question. If you mean output datasets that result from running a functional test for a tool, then I believe there is no difference if the test passed or failed. Thanks, Eric Greg Von Kuster Galaxy Development Team g...@bx.psu.edumailto:g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ Greg Von Kuster Galaxy Development Team g...@bx.psu.edu ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy
Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support
Thanks for the quick replies again! Yeah, from a technical standpoint such support is certainly doable. My employer strongly discourages modifying the Galaxy code base too invasively (if at all), which is pretty fair since I'm not in a position to take responsibility for performing future Galaxy upgrades which may have messy merges as a consequence of my tinkering downstream of you. That's primarily the reason I was curious about whether such features were in the works or at least on the horizon at the Galaxy Development Team proper. Anyway, thanks for communicating. Have a great day :) Best, Eric From: Greg Von Kuster [g...@bx.psu.edu] Sent: Wednesday, November 30, 2011 1:22 PM To: Paniagua, Eric Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support On Nov 30, 2011, at 11:11 AM, Paniagua, Eric wrote: Hi Greg, I appreciate your response! Thanks for clarifying the capabilities and limitations of the current functional testing framework. (BTW, did you mean current as in current on galaxy-dist or also current on galaxy-central ?) If I can find the time, these are areas I would be interested in helping enhance. Current on Galaxy central, very likely the same as Galaxy dist. Sorry my diction was unclear; by output files I meant the transient files that a running job is free to create and manipulate in its job_working_directory. (I'm writing transient rather than temporary to avoid confusion with, e.g. files created with tempfile.mkstemp - although conceivably one might want to be able to inspect their contents on failure as well, actually saving them might be a very different problem.) Normally, upon successful or failed job completion the job working directory and anything in it are erased. For certain tools, it could be helpful in debugging if the developer (or Galaxy system administrator) were able to inspect their contents to see, for instance, if they are properly formed. I can disable removal of job working directories globally, but then of course the job working directories also persist for successful jobs, which can add up to a lot of unnecessary storage (the reason they're deleted by default in the first place). I'm not sure how work is divided on your team, but can you tell me (a) if the preceding paragraphs actually clarify anything for you, Yes, the functional test framework does not currently deal with anything in job_working_directory as far as I know. I am not a primary tool developer on the development team, however, so there may be some peripheral test components working in this realm of which I am not aware. My understanding of the use of the job_working_directory is that some files are moved out of it into permanent locations, while others are deleted upon job completion. You should be able to enhance the job code to enable you to inspect certain elements of this directory during job processing, but I'm not sure how difficult this may be. and (b) whether that issue is on the radar of your team and specifically on the radar of the primary developer(s) / maintainer(s) of the testing framework? This issue is not currently anywhere on the development team's radar. Sorry if this is an inconvenience. Thanks again for responding to my email. Best, Eric From: Greg Von Kuster [g...@bx.psu.edu] Sent: Wednesday, November 30, 2011 9:56 AM To: Paniagua, Eric Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [Internal - Galaxy-dev #2159] (New) [galaxy-dev] 2 questions about Galaxy's functional testing support Hello Eric, Submitted by epani...@cshl.edumailto:epani...@cshl.edu Hi all, I've read the Wiki page on Writing Functional Tests (http://wiki.g2.bx.psu.edu/Admin/Tools/Writing%20Tests) and I've been looking through test/base and test/functional and I am left with two questions: * Is it possible to write a test to validate metadata directly on an (optionally composite) output dataset? I'm sure this is possible, but it would require enhancements to the current functional test framework. Everything described on the above page is file oriented. I see that there is TwillTestCase.check_metadata_for_string, but as far as I can tell this is a bit nonspecific since it appears to just do a text search on the Edit page. This is correct. I don't yet fully understand the context in which tests run, but is there some way to access a live dataset's metadata directly, either as a dictionary or just as attributes? Or even to get the actual dataset object? Not with the current functional test framework. Doing this would require enhancements to the framework. * Does the test harness support retaining output files only for failed tests? Ideally with a cap on how much output data to save
Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support
Eric, We always welcome help from the Galaxy community, so if you are interested in enhancing the Galaxy code, the best way to go about it is to create your own fork off the Galaxy central repo in bitbucket, and when you have something to contribute initiate a pull request for us to review and merge into the central repo. This way you don't have to deal with ongoing merges. Thanks! On Nov 30, 2011, at 2:24 PM, Paniagua, Eric wrote: Thanks for the quick replies again! Yeah, from a technical standpoint such support is certainly doable. My employer strongly discourages modifying the Galaxy code base too invasively (if at all), which is pretty fair since I'm not in a position to take responsibility for performing future Galaxy upgrades which may have messy merges as a consequence of my tinkering downstream of you. That's primarily the reason I was curious about whether such features were in the works or at least on the horizon at the Galaxy Development Team proper. Anyway, thanks for communicating. Have a great day :) Best, Eric From: Greg Von Kuster [g...@bx.psu.edu] Sent: Wednesday, November 30, 2011 1:22 PM To: Paniagua, Eric Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [galaxy-dev] [Internal - Galaxy-dev #2159] (New) 2 questions about Galaxy's functional testing support On Nov 30, 2011, at 11:11 AM, Paniagua, Eric wrote: Hi Greg, I appreciate your response! Thanks for clarifying the capabilities and limitations of the current functional testing framework. (BTW, did you mean current as in current on galaxy-dist or also current on galaxy-central ?) If I can find the time, these are areas I would be interested in helping enhance. Current on Galaxy central, very likely the same as Galaxy dist. Sorry my diction was unclear; by output files I meant the transient files that a running job is free to create and manipulate in its job_working_directory. (I'm writing transient rather than temporary to avoid confusion with, e.g. files created with tempfile.mkstemp - although conceivably one might want to be able to inspect their contents on failure as well, actually saving them might be a very different problem.) Normally, upon successful or failed job completion the job working directory and anything in it are erased. For certain tools, it could be helpful in debugging if the developer (or Galaxy system administrator) were able to inspect their contents to see, for instance, if they are properly formed. I can disable removal of job working directories globally, but then of course the job working directories also persist for successful jobs, which can add up to a lot of unnecessary storage (the reason they're deleted by default in the first place). I'm not sure how work is divided on your team, but can you tell me (a) if the preceding paragraphs actually clarify anything for you, Yes, the functional test framework does not currently deal with anything in job_working_directory as far as I know. I am not a primary tool developer on the development team, however, so there may be some peripheral test components working in this realm of which I am not aware. My understanding of the use of the job_working_directory is that some files are moved out of it into permanent locations, while others are deleted upon job completion. You should be able to enhance the job code to enable you to inspect certain elements of this directory during job processing, but I'm not sure how difficult this may be. and (b) whether that issue is on the radar of your team and specifically on the radar of the primary developer(s) / maintainer(s) of the testing framework? This issue is not currently anywhere on the development team's radar. Sorry if this is an inconvenience. Thanks again for responding to my email. Best, Eric From: Greg Von Kuster [g...@bx.psu.edu] Sent: Wednesday, November 30, 2011 9:56 AM To: Paniagua, Eric Cc: galaxy-dev@lists.bx.psu.edu Dev Subject: Re: [Internal - Galaxy-dev #2159] (New) [galaxy-dev] 2 questions about Galaxy's functional testing support Hello Eric, Submitted by epani...@cshl.edumailto:epani...@cshl.edu Hi all, I've read the Wiki page on Writing Functional Tests (http://wiki.g2.bx.psu.edu/Admin/Tools/Writing%20Tests) and I've been looking through test/base and test/functional and I am left with two questions: * Is it possible to write a test to validate metadata directly on an (optionally composite) output dataset? I'm sure this is possible, but it would require enhancements to the current functional test framework. Everything described on the above page is file oriented. I see that there is TwillTestCase.check_metadata_for_string, but as far as I can tell this is a bit nonspecific since it appears to just do a text search on the Edit
