phaniarnab commented on PR #2050:
URL: https://github.com/apache/systemds/pull/2050#issuecomment-2241079330

   > > @WDRshadow, thanks for putting the numbers here. Did you take an average 
of 3 runs to capture the execution time? If not, please do that to avoid the 
JIT compilation and GC overheads. And I assume the numbers reported in this 
table only measure the total inference time and not the training time.
   > > The speedup from 2 GPUs is way less than I expected. Can you explain, 
why the speedup is not consistently 2x? If you are scoring n images, then each 
GPU gets n/2 images, which should lead to 2x speedup. I do not anticipate any 
additional overhead for two GPUs for this use case.
   > 
   > Thanks. Your assumptions are inaccurate. This time is the total execution 
time, which includes a exactly the same training process before the execution 
of the `parfor`loop. This is one reason. I am not familiar with `.dml` files 
and have no time to learn it, so I don't know how to store and read a trained 
model.
   
   Okay. In that case, try one of the two options: (1) write the model to disk, 
create separate dml scripts for inference where you read the model and 
immediately start the parfor loop. You can find plenty of read, write examples 
in the test scripts and the reproducibility scripts I shared with you. (2) use 
time() method before and after the parfor and report only the inference time. 
You can find an example of using time() here: 
https://github.com/damslab/reproducibility/blob/master/vldb2022-UPLIFT-p2528/FTBench/systemds/T1.dml
   
   For either option, make sure the intermediates are already materialized 
before the loop starts. SystemDS compiler sometime delays operations till used. 
You can print the sum of a matrix to force materialization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@systemds.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to