That’s interesting. I believe n_jobs just wraps multiprocessing through joblib. So, setting it to -1 can essentially spawn as many processes as you have processors available, which may come with an undesired overhead in your environment? I am not sure if this is (still) and issue, but maybe have a look at this one here: https://twiki.cern.ch/twiki/bin/view/Main/PythonLoggingThreadingMultiprocessingIntermixedStudy
> Mixing Python modules multiprocessing and threading along with logging > error/debug messages with module logging is a very bad idea which leads to > unexpected process stalling. Indeed, having concurrent entities based on both > multiprocessing.Process and threading.Thread within a single application > causes right the multiprocessing.Process to stall and never the > threading.Thread, as the attached examples demonstrate. The problematic > behaviour demonstrates: Best, Sebastian > On Jun 17, 2016, at 11:01 AM, Philip Tully <tu...@csc.kth.se> wrote: > > Hi all, > > I notice when I train a model and expose the predict function through a web > API, predict takes longer to run in a multi-threaded environment than a > single-threaded one. I'm guessing the root cause has something to do with > thread collisions but I must be doing something incorrectly within the code > (I set n_jobs=-1 for both FeatureUnion and estimators/gridsearchers there) > > has someone else ran into a similar issue? I can provide more details if this > Q is rather opaque still > > best, > Philip > _______________________________________________ > scikit-learn mailing list > scikit-learn@python.org > https://mail.python.org/mailman/listinfo/scikit-learn _______________________________________________ scikit-learn mailing list scikit-learn@python.org https://mail.python.org/mailman/listinfo/scikit-learn
