2012/7/10 Lars Buitinck <[email protected]>: > 2012/7/10 Olivier Grisel <[email protected]>: >> When doing single node multi cpu parallel machine learning (e.g grid >> search, one vs all SGD, random forests), it would be great to avoid >> duplicating memory, especially for the input dataset that is used as a >> readonly resource in most of our common usecases. > > I may be entirely mistaken here, but I always assumed copy-on-write > semantics would apply in joblib, so getting data from the master to > the workers would be free as long as the workers don't change the > data. Is that not the case?
AFAIK the mem saving benefits of copy-on-write hardly applies since ref counting triggers writes. I did some simple experiments with joblib a while ago and found that memory consumption in the workers is usually twice the footprint of the master. I've played with Array once but it would require some effort when working with sparse matrices... (basically wrapping data, indptr, indices I guess) > > -- > Lars Buitinck > Scientific programmer, ILPS > University of Amsterdam > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Scikit-learn-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general -- Peter Prettenhofer ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
