Fei Hu created SYSTEMML-1831:

             Summary: Improve the efficiency of matrix subsetting
                 Key: SYSTEMML-1831
                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1831
             Project: SystemML
          Issue Type: Improvement
            Reporter: Fei Hu

For the {{rangeReIndex}} operation, it needs to read the whole input matrix 
into memory first and do the subsetting in the memory. It is not efficient 
because it needs to read a lot of unnecessary data, and even out of memory if 
the size of input matrix exceeds the available memory. 

The plan here is to read the keys in the Hadoop sequence file, and identify the 
keys overlapped with the input range index. Then the values specified by the 
identified keys will be read.

This message was sent by Atlassian JIRA

Reply via email to