Baunsgaard commented on PR #1697: URL: https://github.com/apache/systemds/pull/1697#issuecomment-1250334572
I was thinking of the index-block design used in binary, but since the compression framework compress an entire matrix in CP i would have to decompose the compression into multiple blocks if we want this and, in CP, reading would have to combine them again. I think this overcomplicate things unless we somehow make the compression able to combine different blocks with the same compression plan. Furthermore if we write a compressed distributed block-indexed matrix to disk we get multiple blocks with different formats that would not be able to combine nicely in CP anyway. Enforcing that such a read should lead to SP instructions. In the end the problems make reading and writing the same way as binary blocks a bit challenging especially if you want same behavior. But i can suggest we always treat the compressed format as an index-block based file with a block size >= nCols && nRows ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
