hops/rewrite/RewriteInjectSparkPReadCheckpointing class is responsible for
placing the checkpoint instruction after a persistent read, and
hops/ipa/IPAPassRemoveUnnecessaryCheckpoints removes unnecessary chkpoints.

Regards,
Arnab..


On Wed, Mar 15, 2023 at 6:05 PM <s...@bgaard.dk> wrote:

> If i execute a simple script of
>
> ```
> X = read($1)
> print(sum(X))
> ```
>
> on a 80GB file.
>
> Then we inject a spark chkpoint, when there are no other spark
> instructions.
> Ideally this checkpoint should not happen, but where do i go to fix this?
>
> explain:
>
> # EXPLAIN (RUNTIME):
> # Memory Budget local/remote = 143360MB/?MB/?MB/?MB
> # Degree of Parallelism (vcores) local/remote = 48/?
> PROGRAM ( size CP/SP = 6/1 )
> --MAIN PROGRAM
> ----GENERIC (lines 1-2) [recompile=false]
> ------CP createvar pREADX testFile.dat false MATRIX binary 1000000 10000
> 1000 10000000000 copy
> ------CP createvar _mVar0 scratch_space//_p70322_192.168.0.11//_t0/temp0
> true MATRIX binary 1000000 10000 1000 10000000000 copy
> ------SPARK chkpoint pREADX.MATRIX.FP64 _mVar0.MATRIX.FP64 MEMORY_AND_DISK
> ------CP uak+ _mVar0.MATRIX.FP64 _Var1.SCALAR.STRING 48
> ------CP rmvar _mVar0
> ------CP print _Var1.SCALAR.STRING.false _Var2.SCALAR.STRING 48
> ------CP rmvar _Var1 _Var2
>
> 4.966327461583818E9
> SystemDS Statistics:
> Total elapsed time: 46.230 sec.
> Total compilation time: 0.801 sec.
> Total execution time: 45.429 sec.
> Number of compiled Spark inst: 1.
> Number of executed Spark inst: 1.
> Cache hits (Mem/Li/WB/FS/HDFS): 0/0/0/0/1.
> Cache writes (Li/WB/FS/HDFS): 0/0/0/0.
> Cache times (ACQr/m, RLS, EXP): 24.751/0.000/0.000/0.000 sec.
> HOP DAGs recompiled (PRED, SB): 0/0.
> HOP DAGs recompile time: 0.000 sec.
> Spark ctx create time (lazy): 18.464 sec.
> Spark trans counts (par,bc,col):0/0/0.
> Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
> Spark async. count (pf,bc,op): 0/0/0.
> Total JIT compile time: 14.217 sec.
> Total JVM GC count: 4.
> Total JVM GC time: 0.1 sec.
> Heavy hitter instructions:
> #  Instruction  Time(s)  Count
> 1  uak+          26.248      1
> 2  sp_chkpoint   19.165      1
> 3  createvar      0.014      2
> 4  print          0.000      1
> 5  rmvar          0.000      2
>
>
>
>

Reply via email to