If you're using something like n_jobs=-1, that will explode memory usage in proportion to the number of cores, and particularly so if you're passing the data as a list rather than array and hence can't take advantage of memmapped data parallelism.
On 16 March 2017 at 15:46, Carlton Banks <[email protected]> wrote: > The ndarray (6,3,3) => (row, col,color channels) > > I tried fixing it converting the list of numpy.ndarray to > numpy.asarray(list) > > but this causes a different problem: > > is grid use a lot a memory.. I am running on a super computer, and seem to > have problems with memory.. already used 62 gb ram.. > > > Den 16. mar. 2017 kl. 05.30 skrev Sebastian Raschka < > [email protected]>: > > > > Sklearn estimators typically assume 2d inputs (as numpy arrays) with > shape=[n_samples, n_features]. > > > >> list of Np.ndarrays of shape (6,3,3) > > > > I assume you mean a 3D tensor (3D numpy array) with shape=[n_samples, > n_pixels, n_pixels]? What you could do is to reshape it before you put it > in, i.e., > > > > data_ary = your_ary.reshape(n_samples, -1).shape > > > > then, you need to add a line at the beginning your CNN class that does > the reverse, i.e., data_ary.reshape(6, n_pixels, n_pixels).shape. Numpy’s > reshape typically returns view objects, so that these additional steps > shouldn’t be “too” expensive. > > > > Best, > > Sebastian > > > > > > > >> On Mar 16, 2017, at 12:00 AM, Carlton Banks <[email protected]> wrote: > >> > >> Hi… > >> > >> I currently trying to optimize my CNN model using gridsearchCV, but > seem to have some problems feading my input data.. > >> > >> My training data is stored as a list of Np.ndarrays of shape (6,3,3) > and my output is stored as a list of np.array with one entry. > >> > >> Why am I having problems parsing my data to it? > >> > >> best regards > >> Carl B. > >> _______________________________________________ > >> scikit-learn mailing list > >> [email protected] > >> https://mail.python.org/mailman/listinfo/scikit-learn > > > > _______________________________________________ > > scikit-learn mailing list > > [email protected] > > https://mail.python.org/mailman/listinfo/scikit-learn > > _______________________________________________ > scikit-learn mailing list > [email protected] > https://mail.python.org/mailman/listinfo/scikit-learn >
_______________________________________________ scikit-learn mailing list [email protected] https://mail.python.org/mailman/listinfo/scikit-learn
