[Pytables-users] Speed of CArray writing sparse matrices

2013-06-22 Thread Giovanni Luca Ciampaglia
Hi all, I have a sparse 3.4M x 3.4M adjacency matrix with nnz = 23M and wanted to see if CArray was an appropriate solution for storing it. Right now I am using the NumPy binary format for storing the data in coordinate format and loading the matrix with Scipy's sparse coo_matrix class. As far

Re: [Pytables-users] Speed of CArray writing sparse matrices

2013-06-22 Thread Anthony Scopatz
Hi Giovanni! I think that you may have some misunderstanding about how chucking works, which is leading you to get terrible performance. In fact what you describe is a great strategy (right all and zip) for using normal Arrays. However, chunking and CArrays don't work like this. If a chunk cont