Baunsgaard opened a new pull request #1148:
URL: https://github.com/apache/systemds/pull/1148
This commit contains various changes (some massive ones).
The biggest change is the ordering of compression steps, of which
now we classify first on a sample of the data. Since this was experimented
to be 10-30% faster. Furthermore this allows us to try compression at a
lower cost if the compression is not valid to perform.
Overall Compression time for covtype went from
- ~1.0 to 0.36 sec (including read from disk) 0.11 sec compression
Furthermore now unlike before the transpose is heuristically chosen, Since
it is more efficient not to transpose the matrix for compression in some
cases.
- Compressed Sparse matrix multiplication fix
- modified matrix multiplication to push down information of
transposing to the ba+* op. to allow not decompressing the matrix.
- Configuration option of enabling and disabling overlapping compression.
- decompress row section direct access to the matrix block not using
quick set/get.
- adding safe boolean to decompress to specify if management of
nnz should be done. This allows the decompression of intermediates at
near half the computation cost.
- Add configuration for sampling ratio default 0.01 but with a minimum
sample size of 2000 elements.
- DML Config settings for Cocode-Compression method default to COST
- add support for right sparse matrix multiplication with overlapping
output. Further improvements are on the way.
- Compression statistics are added when statistics and compression is
enabled
- Readers for extracting bitmaps are optimized for either transposed or
untransposed matrices giving 5-15% improved performance.
- Hashmaps are modified to improve insertion time since previously they
would hash values twice 10% improved performance. furthermore the
default sizes are modified to start smaller.
- Additional tests for multipication to cover different edge cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
