Re: [galaxy-dev] Twill comparisons ignore GALAXY_TEST_NO_CLEANUP

2014-02-11 Thread Dave Bouvier

Peter,

Thanks for pointing this out, I've applied your changes in 
12478:15fc8675064e on the default branch.


   --Dave B.

On 02/11/2014 09:13 AM, Peter Cock wrote:

Hi all,

Something very handy for tool developers running functional tests
is if the environment variable GALAXY_TEST_NO_CLEANUP
is set, run_functional_tests.sh doesn't remove the test output.
This is great when debugging why a test is failing.

However, I was having trouble with failures like this:

base.twilltestcase: INFO: ## files diff on
/mnt/galaxy/galaxy-central/test-data/ecoli.mira4_de_novo.fasta and
/tmp/tmpTvyobN/tmpYsEezt/new_files_path_pk7YKe/tmpXr9NYeecoli.mira4_de_novo.fasta
lines_diff=0, found diff = 3010

Sadly the temp file here was being deleted - the patch at the
end of the email fixes the diff code to respect the configuration
setting.

Having done that and looked at the preserved file, somehow
the wrong output file was being compared to the FASTA
output file (it was looking at my log file instead).

Because the diff was really between two different files,
showing the first forty lines wasn't informative. Instead
I've made it show the first 25 and last 25 lines of the diff.
I find this a great deal more informative :)

Meanwhile this points at a possible bug in the test
framework, if anyone wants to attempt to reproduce
it I used this version of my MIRA4 wrapper:
https://github.com/peterjc/pico_galaxy/commit/3f095eac3f5609b2890440b851029a983fe14a20

$ export GALAXY_TEST_NO_CLEANUP=on
$ ./run_functional_tests.sh -id mira_4_0_de_novo
...

$ hg summary
parent: 13513:f3dc213a5773 tip
  Backout of 28d43f4.
branch: default
commit: 7 modified, 229 unknown
update: (current)


Regards,

Peter

--

$ hg diff test/base/twilltestcase.py
diff -r f3dc213a5773 test/base/twilltestcase.py
--- a/test/base/twilltestcase.pyMon Feb 10 22:13:35 2014 -0600
+++ b/test/base/twilltestcase.pyTue Feb 11 14:09:30 2014 +
@@ -95,7 +95,10 @@
  diff = list( difflib.unified_diff( local_file,
history_data, "local_file", "history_data" ) )
  diff_lines = get_lines_diff( diff )
  if diff_lines > allowed_diff_count:
-diff_slice = diff[0:40]
+if len(diff) < 60:
+diff_slice = diff[0:40]
+else:
+diff_slice = diff[:25] + ["\n", "*
SNIP *\n", "\n"] + diff[-25:]
  #FIXME: This pdf stuff is rather special cased
and has not been updated to consider lines_diff
  #due to unknown desired behavior when used in
conjunction with a non-zero lines_diff
  #PDF forgiveness can probably be handled better
by not special casing by __extension__ here
@@ -897,7 +900,8 @@
  errmsg += str( err )
  raise AssertionError( errmsg )
  finally:
-os.remove( temp_name )
+if 'GALAXY_TEST_NO_CLEANUP' not in os.environ:
+os.remove( temp_name )

  def __default_dataset_fetcher( self ):
  def fetcher( hda_id, filename=None ):
@@ -971,7 +975,8 @@
  errmsg += str( err )
  raise AssertionError( errmsg )
  finally:
-os.remove( temp_name )
+if 'GALAXY_TEST_NO_CLEANUP' not in os.environ:
+os.remove( temp_name )

  def is_zipped( self, filename ):
  if not zipfile.is_zipfile( filename ):
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


[galaxy-dev] Twill comparisons ignore GALAXY_TEST_NO_CLEANUP

2014-02-11 Thread Peter Cock
Hi all,

Something very handy for tool developers running functional tests
is if the environment variable GALAXY_TEST_NO_CLEANUP
is set, run_functional_tests.sh doesn't remove the test output.
This is great when debugging why a test is failing.

However, I was having trouble with failures like this:

base.twilltestcase: INFO: ## files diff on
/mnt/galaxy/galaxy-central/test-data/ecoli.mira4_de_novo.fasta and
/tmp/tmpTvyobN/tmpYsEezt/new_files_path_pk7YKe/tmpXr9NYeecoli.mira4_de_novo.fasta
lines_diff=0, found diff = 3010

Sadly the temp file here was being deleted - the patch at the
end of the email fixes the diff code to respect the configuration
setting.

Having done that and looked at the preserved file, somehow
the wrong output file was being compared to the FASTA
output file (it was looking at my log file instead).

Because the diff was really between two different files,
showing the first forty lines wasn't informative. Instead
I've made it show the first 25 and last 25 lines of the diff.
I find this a great deal more informative :)

Meanwhile this points at a possible bug in the test
framework, if anyone wants to attempt to reproduce
it I used this version of my MIRA4 wrapper:
https://github.com/peterjc/pico_galaxy/commit/3f095eac3f5609b2890440b851029a983fe14a20

$ export GALAXY_TEST_NO_CLEANUP=on
$ ./run_functional_tests.sh -id mira_4_0_de_novo
...

$ hg summary
parent: 13513:f3dc213a5773 tip
 Backout of 28d43f4.
branch: default
commit: 7 modified, 229 unknown
update: (current)


Regards,

Peter

--

$ hg diff test/base/twilltestcase.py
diff -r f3dc213a5773 test/base/twilltestcase.py
--- a/test/base/twilltestcase.pyMon Feb 10 22:13:35 2014 -0600
+++ b/test/base/twilltestcase.pyTue Feb 11 14:09:30 2014 +
@@ -95,7 +95,10 @@
 diff = list( difflib.unified_diff( local_file,
history_data, "local_file", "history_data" ) )
 diff_lines = get_lines_diff( diff )
 if diff_lines > allowed_diff_count:
-diff_slice = diff[0:40]
+if len(diff) < 60:
+diff_slice = diff[0:40]
+else:
+diff_slice = diff[:25] + ["\n", "*
SNIP *\n", "\n"] + diff[-25:]
 #FIXME: This pdf stuff is rather special cased
and has not been updated to consider lines_diff
 #due to unknown desired behavior when used in
conjunction with a non-zero lines_diff
 #PDF forgiveness can probably be handled better
by not special casing by __extension__ here
@@ -897,7 +900,8 @@
 errmsg += str( err )
 raise AssertionError( errmsg )
 finally:
-os.remove( temp_name )
+if 'GALAXY_TEST_NO_CLEANUP' not in os.environ:
+os.remove( temp_name )

 def __default_dataset_fetcher( self ):
 def fetcher( hda_id, filename=None ):
@@ -971,7 +975,8 @@
 errmsg += str( err )
 raise AssertionError( errmsg )
 finally:
-os.remove( temp_name )
+if 'GALAXY_TEST_NO_CLEANUP' not in os.environ:
+os.remove( temp_name )

 def is_zipped( self, filename ):
 if not zipfile.is_zipfile( filename ):
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/