Michał Dereziński <michal.derezin...@gmail.com> writes: >> Wiadomość napisana przez Jed Brown <j...@jedbrown.org> w dniu 24.05.2017, o >> godz. 12:06: >> >> Okay, do you have more parameters than observations? > > No (not necessarily). The biggest matrix is 50M observations and 12M > parameters. > >> And each segment >> of the matrix will be fully distributed? > > Yes. > >> Do you have a parallel file >> system? > > Yes. > >> Is your matrix sparse or dense? > > Yes.
By that you mean sparse? You'll need some sort of segmented storage (could be separate files or a file format that allows seeking). (If the matrix is generated by some other process, you'd benefit from skipping the file system entirely, but I understand that may not be possible.) I would use MatNest, creating a new one after each segment is loaded. There isn't currently a MatLoadBegin/End interface, but that could be created if it would be useful.
signature.asc
Description: PGP signature
