Re: [pymvpa] PyMVPA for Python 3 -- second report
oh boy... ;-) was it painful? Thanks! On Sun, 22 Apr 2012, Tiziano Zito wrote: Also since github PR accepted as merges, it would be best first to just cherry pick mine on top of yours to avoid merges and then do rebase -i done :) -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic ___ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
Re: [pymvpa] PyMVPA for Python 3 -- second report
was it painful? not really, just a bit of vi gymnastics ;) ciao, tiziano ___ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
Re: [pymvpa] PyMVPA for Python 3 -- second report
may be some skype/shared screen sprintee next week over remaining issues etc? yes, if michael is around we could meet in berlin or magdeburg and have get you on skype. There is a bulk of changes in some guys feature branches and if we do not merge now, later it might not be as easy ;) I guess I would also need to check how changes affect minimal compatible version in 2.x series (if affect at all). FWIC (for what i care) 2.6 should be good enough for me ;) I have tried to stay compatible with 2.5 so that you can backport to squeeze, but I did not test if this is really the case. but at times going through changes I got a bit scared ;-) e.g. that is the most controversial change, I think: 1. -self._unique_values = np.unique(self.value) +# get a 1-D array +arrvalue = np.asarray(self.value).ravel() +# check if we have None somewhere +nones = np.equal(arrvalue, None) +if nones.any(): +uniques = np.unique(arrvalue[nones == False]) +# put back one None +self._unique_values = np.insert(uniques, 0, None) +else: +self._unique_values = np.unique(arrvalue) does numpy handles its own arrays differently among pythons or what is the reason? the reason is that sometimes self.values contains None's. in python 3 arithmetic comparisons like or are not allowed among heterogeneous types, so you can not sort a list or an array that looks for example like this: [ 0, 1, None, -2]. and np.unique uses sort internally. the dance with the None's is not needed in py2, so we could special case with a sys.version = '3', but you have to decide if it is worth it. the new masked array type in newcoming numpy could probably used instead... it is not an inferior behaviour when compared to py2: actually I think I spotted at least one place (the tests is still failing) where PyMVPA was comparing non-comparable types and getting a basically random result: == ERROR: test_basic (mvpa2.tests.test_datameasure.SensitivityAnalysersTests) -- Traceback (most recent call last): File /home/tiziano/git/PyMVPA/build/py3k/mvpa2/testing/sweepargs.py, line 67 , in do_sweep method(*args_, **kwargs_) File /home/tiziano/git/PyMVPA/build/py3k/mvpa2/tests/test_datameasure.py, li ne 79, in test_basic self.assertTrue(f[0] 0.1) # some reasonably large value TypeError: unorderable types: Dataset() float() 2. -sample += random.sample((targets == r).nonzero()[0], npertarget[i] ) +sample += random.sample(list((targets == r).nonzero()[0]), npertarget[i] ) why does it need explicit conversion to list, are numpy arrays not iterables any longer, or random.sample now can't tollerate them... the reason is: Python 3.2.3rc2 (default, Mar 21 2012, 05:47:04) [GCC 4.6.3] on linux2 Type help, copyright, credits or license for more information. import numpy x = numpy.zeros((10,)) import random random.sample(x, 1) Traceback (most recent call last): File stdin, line 1, in module File /usr/lib/python3.2/random.py, line 298, in sample raise TypeError(Population must be a sequence or set. For dicts, use list(d).) TypeError: Population must be a sequence or set. For dicts, use list(d). I do not know why numpy arrays are not sequences in py3, but this is it. +Convert *py files with lib2to3. +Taken from MDP and numpy. so -- copyright of numpy and MDP developers? years? (BSD-3 I guess ;) ) will do :) ciao, tiziano ___ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
Re: [pymvpa] PyMVPA for Python 3 -- second report
cool -- you can consider me now a contributor to this little project ;-) I have sent a pull request against your branch with few changes, please review. I merged your pull request, thank you :) Also it might be nice to markup those commits which deal with python3 compatibility as PY3 (e.g. instead of BF), not sure if it is worth rewriting the history for this cause though :-) but might be good to start at least now that's a good idea. I fear my git foo stops at git rebase -i, to do what you want we would need a git filter-branch, right? I would prefer to rewrite history, if you give me a couple of hints on how to do it properly. ciao, tiziano ___ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
Re: [pymvpa] PyMVPA for Python 3 -- second report
On Sat, 21 Apr 2012, Tiziano Zito wrote: I merged your pull request, thank you :) glad to help ;) that's a good idea. I fear my git foo stops at git rebase -i, to do what you want we would need a git filter-branch, right? I would prefer to rewrite history, if you give me a couple of hints on how to do it properly. well -- git rebase -i is probably the way to go here anyways... it is just a bit cumbersome since commit msgs IIRC are not tuned 'inplace' and you first need to mark them as needed 'rewording' and only then it would bring editor for each commit msg. Also since github PR accepted as merges, it would be best first to just cherry pick mine on top of yours to avoid merges and then do rebase -i that is why I wasn't sure if it is worth an effort (although with GIT it is so easy to give a shout while being sure that nothing would be screwed/lost and git reset --hard is there to get things back ;-) ) -- =--= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic ___ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
Re: [pymvpa] PyMVPA for Python 3 -- second report
thank you Tiziano! I have started to glance over the changes -- lots of nice ones -- thanks again, and we should finalize/merge asap - may be some skype/shared screen sprintee next week over remaining issues etc? There is a bulk of changes in some guys feature branches and if we do not merge now, later it might not be as easy ;) I guess I would also need to check how changes affect minimal compatible version in 2.x series (if affect at all). FWIC (for what i care) 2.6 should be good enough for me ;) but at times going through changes I got a bit scared ;-) e.g. 1. -self._unique_values = np.unique(self.value) +# get a 1-D array +arrvalue = np.asarray(self.value).ravel() +# check if we have None somewhere +nones = np.equal(arrvalue, None) +if nones.any(): +uniques = np.unique(arrvalue[nones == False]) +# put back one None +self._unique_values = np.insert(uniques, 0, None) +else: +self._unique_values = np.unique(arrvalue) does numpy handles its own arrays differently among pythons or what is the reason? 2. -sample += random.sample((targets == r).nonzero()[0], npertarget[i] ) +sample += random.sample(list((targets == r).nonzero()[0]), npertarget[i] ) why does it need explicit conversion to list, are numpy arrays not iterables any longer, or random.sample now can't tollerate them... in either case sounds inferior to what python 2.x provided 3. +Convert *py files with lib2to3. +Taken from MDP and numpy. so -- copyright of numpy and MDP developers? years? (BSD-3 I guess ;) ) On Fri, 20 Apr 2012, Tiziano Zito wrote: hi all, with my today's efforts I got down to 13 errors and 15 failures out of 284 tests, so it's getting better :) on the same branch the same set of unittests passes without errors or failures with python 2.7, so I assume I did not break anything important. I attach here the error logs in case someone else feels like having a look. the remaining errors and failures are probably out of my reach, as fixing them requires a much deeper understanding of the very rich data types structure of PyMVPA ;) I am available for a short sprint in case someone is orbiting around northern germany... there are some commits that I think would require a deeper review by some core developer, as I am not 100% sure those changes are really as neutral as the lack of failures in the unittests seems to suggest. I had to disable a test that was hanging in an infinite loop: test_nfold_random_counted_selection_partitioner no idea what the problem there is, really. one point where I can directly ask for a solution is the usage of the variable nproc in BaseSearchlight (mvpa2/measures/searchlight.py): the doc-string seems to imply that setting it to None would use all available cores, but if the pprocess module is not installed this is not true, and an error is spit only in the case nproc 1 (which is false if nproc is None). so maybe the doc-string could be updated to say something like: nproc : None or int How many processes to use for computation. Requires `pprocess` external module. If None -- all available cores will be used if `pprocess` is available, a single core will be used otherwise. ciao, tiziano python3 setup.py config --noisy Converting to Python3 via 2to3... running config python3 setup.py build_ext --inplace Converting to Python3 via 2to3... running build_ext running build_src build_src building extension mvpa2.clfs.libsmlrc.smlrc sources building extension mvpa2.clfs.libsvmc._svmc sources building data_files sources build_src: building npy-pkg config files customize UnixCCompiler customize UnixCCompiler using build_ext customize UnixCCompiler customize UnixCCompiler using build_ext running scons touch build3-stamp I: Running py3-compatible unittests. None of them should ever fail. T: MVPA_SEED=1251509818 T: Testing for availability of external software packages. WARNING: libsvm verbosity control is not available. * Please note: warnings are printed only once, but underlying problem might occur many times * test_externals (mvpa2.tests.test_externals.TestExternals) ... ok test_externals_correct2nd_invocation (mvpa2.tests.test_externals.TestExternals) ... ok test_externals_no_double_invocation (mvpa2.tests.test_externals.TestExternals) ... ok test_basic (mvpa2.tests.test_dochelpers.DochelpersTests) ... ok test_borrow_doc (mvpa2.tests.test_dochelpers.DochelpersTests) ... ok test_borrow_kwargs (mvpa2.tests.test_dochelpers.DochelpersTests) ... ok test_searchlight_doc (mvpa2.tests.test_dochelpers.DochelpersTests) ... ok test_simple_som (mvpa2.tests.test_som.SOMMapperTests) ... ERROR test_blank_state (mvpa2.tests.test_state.StateTests) ... ok test_deep_copying_state_variable (mvpa2.tests.test_state.StateTests) ... ok Check if we can
