On Tue, 08 May 2012, Tiziano Zito wrote:
> > > is there a point where generic numpy.random gets explicitly seeded
> > > upon sklearn import?
> > No, and I don't think that this is desirable: it would be a weird side
> > effect of importing the scikit. It might be interesting to seed the
> > global RNG in the tests, but I have found such an approach inefficient,
> > as the order in which the tests can be executed is not reproducible.
> I concur with Gaël here. I think one good approach is to seed the
> RNG for every single unittest. This gets rid of the ordering
> problem, and it makes it possible to re-run just the failing test
> with a fixed seed. With MDP we use the same seed picked at random at
> the beginning of the testing session for every single test.
in PyMVPA we also do have @seed decorator which we use for some tests.
That @seed "uses the same seed pickled at random at the beginning
of the testing session". And no -- I wasn't not suggesting to seed RNGs
upon a normal import of sklearn.
in PyMVPA we are probably less prone to the random order of tests run
since we pretty much specify the order, that is why knowing global
seeding happening before running tests helps A LOT to reproduce failing
tests ... but @seed'ing every test is somewhat a burden (probably a
wise solution would be to come with with a nose plugin or smth to seed
RNG before running every test)
but what are we arguing now between is: unknown global seeding (current
situation) which might result in difficult to reproduce failure e.g. if seeding
was not hardcoded to use some fixed seed . Removing this unknown would be only
of help imho.
knowing global seeding might have been of help if order of tests would
happen to be the same and would of less help but still might be helpful if
execution order vaires. Typical usecase could be -- knowing that some test
fails, and looping through e.g. 1000 runs of that sweeping the seed and then
sharing that seed with a developer who might have better clue, e.g. atm I am
doing
(set -e; for s in {501..1500}; do echo $s; SKLEARN_SEED=$s PYTHONPATH=$PWD
nosetests -s -v sklearn/svm/tests/test_sparse.py; done; )
so if it fails for some specific seed, I could check if it gets
replicated by running the same test again with the same seed.
if it doesn't -- I know **for sure** that it is not related to having
random data but smth more fun, worth valgrinding for decisions based on
uninitialized memory etc. e.g. now it halted (burns cpu, doesn't return) with
seed 1072 (before actually it just crashed in the same
sklearn.svm.tests.test_sparse.test_sparse_svc_clone_with_callable_kernel and
didn't reproduce). So I could valgrind it to see e.g. plenty of
==12606== Conditional jump or move depends on uninitialised value(s)
==12606== at 0x153E5989: svm_csr_predict_probability (svm.cpp:2002)
==12606== by 0x153C9F69: csr_copy_predict_proba (libsvm_sparse_helper.c:280)
==12606== by 0x153D1155:
__pyx_pf_7sklearn_3svm_13libsvm_sparse_2libsvm_sparse_predict_proba
(libsvm_sparse.c:3588)
...
==12606== Conditional jump or move depends on uninitialised value(s)
==12606== at 0x58971E3: exp (w_exp.c:45)
==12606== by 0x153E5999: svm_csr_predict_probability (svm.cpp:2005)
==12606== by 0x153C9F69: csr_copy_predict_proba (libsvm_sparse_helper.c:280)
==12606== by 0x153D1155:
__pyx_pf_7sklearn_3svm_13libsvm_sparse_2libsvm_sparse_predict_proba
(libsvm_sparse.c:3588)
and
sklearn.svm.tests.test_sparse.test_sparse_svc_clone_with_callable_kernel ...
==12606== Invalid read of size 8
==12606== at 0x153E8DE1: svm_csr::Kernel::kernel_precomputed(int, int) const
(svm.cpp:345)
==12606== by 0x153E45C1: svm_csr::svm_train_one(svm_csr_problem const*,
svm_parameter const*, double, double) (svm.cpp:1405)
==12606== by 0x153E6B16: svm_csr_train (svm.cpp:2472)
==12606== by 0x153E75F5: svm_csr_train (svm.cpp:2148)
==12606== by 0x153CB761:
__pyx_pf_7sklearn_3svm_13libsvm_sparse_libsvm_sparse_train
(libsvm_sparse.c:1635)
altogether -- even if no direct help to reproduce, controlling seeding would be
beneficial. And for me in particular while building across a dozen of
debian/ubuntu releases and later on trying to figure out what was the cause --
random data or smth more fun.
but after all it is all up to you guys either to accept the PR at the cost of a
somewhat junky setup_module in sklearn namespace (which is probably noone
really explores much besides completing into a submodule ;) )
--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Scikit-learn-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
