hops/rewrite/RewriteInjectSparkPReadCheckpointing class is responsible for placing the checkpoint instruction after a persistent read, and hops/ipa/IPAPassRemoveUnnecessaryCheckpoints removes unnecessary chkpoints.
Regards, Arnab.. On Wed, Mar 15, 2023 at 6:05 PM <s...@bgaard.dk> wrote: > If i execute a simple script of > > ``` > X = read($1) > print(sum(X)) > ``` > > on a 80GB file. > > Then we inject a spark chkpoint, when there are no other spark > instructions. > Ideally this checkpoint should not happen, but where do i go to fix this? > > explain: > > # EXPLAIN (RUNTIME): > # Memory Budget local/remote = 143360MB/?MB/?MB/?MB > # Degree of Parallelism (vcores) local/remote = 48/? > PROGRAM ( size CP/SP = 6/1 ) > --MAIN PROGRAM > ----GENERIC (lines 1-2) [recompile=false] > ------CP createvar pREADX testFile.dat false MATRIX binary 1000000 10000 > 1000 10000000000 copy > ------CP createvar _mVar0 scratch_space//_p70322_192.168.0.11//_t0/temp0 > true MATRIX binary 1000000 10000 1000 10000000000 copy > ------SPARK chkpoint pREADX.MATRIX.FP64 _mVar0.MATRIX.FP64 MEMORY_AND_DISK > ------CP uak+ _mVar0.MATRIX.FP64 _Var1.SCALAR.STRING 48 > ------CP rmvar _mVar0 > ------CP print _Var1.SCALAR.STRING.false _Var2.SCALAR.STRING 48 > ------CP rmvar _Var1 _Var2 > > 4.966327461583818E9 > SystemDS Statistics: > Total elapsed time: 46.230 sec. > Total compilation time: 0.801 sec. > Total execution time: 45.429 sec. > Number of compiled Spark inst: 1. > Number of executed Spark inst: 1. > Cache hits (Mem/Li/WB/FS/HDFS): 0/0/0/0/1. > Cache writes (Li/WB/FS/HDFS): 0/0/0/0. > Cache times (ACQr/m, RLS, EXP): 24.751/0.000/0.000/0.000 sec. > HOP DAGs recompiled (PRED, SB): 0/0. > HOP DAGs recompile time: 0.000 sec. > Spark ctx create time (lazy): 18.464 sec. > Spark trans counts (par,bc,col):0/0/0. > Spark trans times (par,bc,col): 0.000/0.000/0.000 secs. > Spark async. count (pf,bc,op): 0/0/0. > Total JIT compile time: 14.217 sec. > Total JVM GC count: 4. > Total JVM GC time: 0.1 sec. > Heavy hitter instructions: > # Instruction Time(s) Count > 1 uak+ 26.248 1 > 2 sp_chkpoint 19.165 1 > 3 createvar 0.014 2 > 4 print 0.000 1 > 5 rmvar 0.000 2 > > > >