Matthias Boehm created SYSTEMML-945:

             Summary: Improve inter-procedure analysis to avoid more checkpoints
                 Key: SYSTEMML-945
             Project: SystemML
          Issue Type: Task
            Reporter: Matthias Boehm

Unnecessary caching can lead to substantial performance issues if not all 
cached intermediates fit into memory. Our inter-procedural analysis already 
removes unnecessary checkpoints (e.g., checkpoint - update - checkpoint, where 
we remove the first checkpoint for simple chains of operations, potentially 
across multiple statement blocks). However, we should extend this by the 
treatment of other common cases. For example, recent GLM experiments on a small 
cluster of 4 nodes showed the following:
GLM_poisson_log train ict=0 on mbperftest/binomial/X10M_1k_dense: 127
GLM_poisson_log train ict=1 on mbperftest/binomial/X10M_1k_dense: 2996
GLM_poisson_log train ict=2 on mbperftest/binomial/X10M_1k_dense: 1971

The reason was the additional append for intercept 1 and 2 which triggered a 
second checkpoint (before the outer loop after the append, in addition to the 
checkpoint after initial read). Note that the existing IPA removal of 
unnecessary checkpoints did not trigger because there was an additional 
sum(X^2) consumer breaking the simple update chain requirement.

Preliminary experiments with an improved handling of checkpoints showed 
reasonable performance of ~130s for both intercept 1 and 2.

This message was sent by Atlassian JIRA

Reply via email to