Matthias Boehm created SYSTEMML-2172:

             Summary: Repartitioning before caching ulta-sparse matrices
                 Key: SYSTEMML-2172
             Project: SystemML
          Issue Type: Bug
            Reporter: Matthias Boehm

Ultra-sparse matrices have dedicated serialized block representation which 
means that their in-memory storage in CSR can be much larger than on disk which 
leads to a blow-up of 128MB partitions to >1GB partitions. Accordingly, we 
should repartition the data before the initial caching in order to remove 
memory pressure and exploit the full parallelism.

This message was sent by Atlassian JIRA

Reply via email to