Re: optimizer?
On 1/7/24 15:18, ToddAndMargo via perl6-users wrote:
sub DirectoryExists( Str $DirPath --> Bool ) { return
"$DirPath".IO.d.Bool; } # $Full DirPath format is `H:\MyDocsBackup`
Problem: sometimes DirectoryExists lies about
backup1 existing.
Here is what I am up against.
You will note the "junction" points. They should not be backed up.
But they are an being creasted as simple directories.
C:\Users\accounting\Documents>dir /a | grep -i junction
09/14/2020 07:39 AM My Music [C:\Users\accounting\Music]
09/14/2020 07:39 AM My Pictures
[C:\Users\accounting\Pictures]
09/14/2020 07:39 AM My Videos
[C:\Users\accounting\Videos]
This is the backup directory after a Windows Explorter erase.
Both DirectoryExists ($DirPath".IO.d.Bool" and Windows
Explorer with Hidden file turned on says it does not exist.
You will note that the command line says it does indeed exist.
And it does.
F:\MyDocsBackup\backup3\Documents 2023-08-15 16;28;47 (Full)>dir /a
Volume in drive F is BACKUP
Volume Serial Number is 0866-7D92
Directory of F:\MyDocsBackup\backup3\Documents 2023-08-15 16;28;47 (Full)
01/05/2024 07:49 PM .
01/05/2024 07:49 PM ..
08/15/2023 03:29 PM My Music
08/15/2023 03:29 PM My Pictures
08/15/2023 03:29 PM My Videos
0 File(s) 0 bytes
5 Dir(s) 480,294,719,488 bytes free
Windows is such a horrid kluge.
:-(
-T
Re: optimizer?
On 1/7/24 06:16, Parrot Raiser wrote:
What's the reason behind the request?
Hi Parrot,
The reason is to eliminate the Optimizer as a
source of weird problems.
I have written a wrapper for Cobian Backup/Reflector
that rotates backup sets. Cobain has is own database
to keep track of backup sets, but it falls apart when
you are rotating several removable drives as the target.
Removable drives in rotation defeats ransomware.
So, my program locates the drive letter the target
resides on based on drive label (it will do letter too),
then counts the backup sets. If these sets exceed
the limit of sets, I remove the oldest.
For example maximum backup sets is set to 4. Then backup4
gets erased, backup3 gets renamed to backup4, backup2
gets renamed to backup3, backup1 gets renamed to backup2.
At this point there should be no backup1. And I check for
it:
sub DirectoryExists( Str $DirPath --> Bool ) { return
"$DirPath".IO.d.Bool; } # $Full DirPath format is `H:\MyDocsBackup`
Problem: sometimes DirectoryExists lies about
backup1 existing. Failure to rotate has to do with
something is the backup set that won't move. These
are typically system files. When I find them, I enter
into Cobian's exclude mask:
desktop.ini
Thumbs.db
.dropbox.*
.dropbox.cache
sessions.jsoniz4
events
~*
~$
python.exe
python3.exe
I have traced this down to the "size" of backup1 too.
A directory in backup1 that refuses to move in
a large backups, will move in a simple test task.
Now to workaround the issue of DirectoryExists not
always returning the correct answer, I have written
my own code for it:
sub IsItReallyReallyGone( Str $Path --> Bool ) {
# checks to see if a file or directory is really gone and not stuck in cache
my Bool $Gone = False;
my Str $RtnStr = "";
# Note: "File Not Found" is sent to the STDERR
$RtnStr = RunCmd Q[dir ] ~ $Path ~ Q[ /A 2>&1];
if $RtnStr.lc.contains( "file not found" ) { $Gone = True ; }
# if %CommandLine { print " IsItReallyReallyGone:\n Gone
= <$Gone>\n Path = <$Path>\n RtnStr = <$RtnStr>\n"; }
if %CommandLine { print " IsItReallyReallyGone: $Path is
$Gone\n"; }
return $Gone;
}
And I flush a lot(same as Linux's "sync"):
sub FlushDriveCache( Str $Drive ) {
my Str $SingleLetter = $Drive.uc.substr( 0..0 );
my Str $RtnStr = "";
print " Flushing " ~ $SingleLetter ~ "'s drive cache\n";
$RtnStr = RunCmd "powershell.exe Write-VolumeCache " ~
$SingleLetter, True;
}
And IsItReallyReallyGone is dead nuts totally, repeatable accurate.
If backup1 is really not gone, a flush and a second delete usually
does the trick. (I can repeat this double erase with Windows
Explorer. First time through it coughs on certain files.
Second time through, it gets them all.)
sub FastRmdir( $DirPath ) {
RunCmd( "del /f/s/q " ~ $DirPath ~ " > nul" );
RunCmd( "rmdir " ~ $DirPath ~ " /S /Q" );
}
Now if backup1 still exists and is not rotated, Cobian keeps
writing to it and eventually the target drive fill up,
meaning no more new backups.
And that is a problem, especially with trying to get my
customer's to actually read their backup reports that
Cobian eMails them (and me, but I only check them
periodically and only as a courtesy).
So in summary, the issue is probably a cache issue with
Raku's IO commands not keeping up with what Windows
is doing. Disabling the Optimizer would remove that
issues as a problem. And Raku's IO commands needs
some work when dealing with large amounts of a data
in cache.
Sorry for being so long winded.
-T
My program's help file:
>raku CobianWrapper.pl6 --help
Welcome to CobianWrapper.pl6
CobianWrapper.pl6 usage:
CobianWrapper.pl6
--backup_path
Drive letter or partition label plus path to backup directory.
for a label, surround the label with square brackets [].
Note: the backup directory name must end in a 1
--rotates
--nokill
nokill will disable the killing of programs known to be
Shadow Copy (VSS) unfriendly
--debug
--help
Defaults:
--backup_path [BACKUP]\MyDocsBackup\backup1
--rotates 2
--nokill False
--debugFalse
--help False
For example
raku C:\NtUtil\CobianWrapper.pl6 --rotates 6 --backup_path
A:\MyDocsBackup\n\backup1
raku C:\NtUtil\CobianWrapper.pl6 --rotates 4 --backup_path
[BACKUP]\MyDocsBackup\backup1
more? (y/n) y
Sample Cobian Pre-backup Event call to CobianWrapper.pl6:
Note: Events, Command Line is currently not working Cobian Reflector as
of 1.0.0
Execute and wait: "C:\Program Files\Rakudo\bin\raku.exe"
"C:\NtUtil\CobianWrapper.pl6 --rotates 20 --backup_path
[BACKUP]\MyDocsBackup\backup1"
Sample Cobian File, Destination for use with CobianWrapper.pl6:
%DISKLABEL="BACKUP":\MyDocsBackup\backup1
Sample Cobian Post-backup Event call to CobianWrapper.pl6 (requires
USB_Disk_Eject-1.3.0.3.exe):
Execute
Re: optimizer?
On 7 Jan 2024, at 07:09, ToddAndMargo via perl6-users wrote: Hi All, Is there a switch on the command line to disable the code optimizer? Many thanks, -T On 1/7/24 04:24, Elizabeth Mattijsen wrote: > $ raku --help > > ... > --optimize=level use the given level of optimization (0..3) > ... > Thank you!
Re: optimizer?
What's the reason behind the request? On Sun, Jan 7, 2024 at 7:24 AM Elizabeth Mattijsen wrote: > $ raku --help > > ... > --optimize=level use the given level of optimization (0..3) > ... > > > On 7 Jan 2024, at 07:09, ToddAndMargo via perl6-users < > perl6-us...@perl.org> wrote: > > > > Hi All, > > > > Is there a switch on the command line to disable the code optimizer? > > > > Many thanks, > > -T > >
Re: optimizer?
$ raku --help ... --optimize=level use the given level of optimization (0..3) ... > On 7 Jan 2024, at 07:09, ToddAndMargo via perl6-users > wrote: > > Hi All, > > Is there a switch on the command line to disable the code optimizer? > > Many thanks, > -T
Re: Optimizer Documentation
Curtis Rawls wrote: As part of my Summer of Code project, I wanted to improve the documentation of the optimizer. Please svn add the .pod to docs/dev Thanks, leo
Re: Optimizer Documentation
Also, this is in plain text format. What is the standard way to
convert a document to POD format?
By rewriting it. Only took me a few minutes:
=head1 NANE
optimizer.pod - About the IMCC optimizer
=head1 ABSTRACT
This document describes how the IMCC optimizer works.
=head1 DESCRIPTION
The objective of the IMCC optimizer is to take a PASM function as
input and apply code-improving transformations to it to be more
efficient, i.e. improving execution time and reducing code size. It
must do this while preserving the same code behavior.
=head2 Data Structures
The optimizer uses a number of data structures to build a model of the
code to be optimized.
=over 4
=item IMC_Unit
The IMC_Unit structure contains all the information known about a
function.It is passed in to each optimizer method.
=item Instruction
Each instruction line has an Instruction structure. Pointers to the
first and last Instruction are stored in IMC_Unit. Instructions are
stored as a linked list. To iterate through all Instructions, use:
Instruction *ins;
for (ins = unit-instructions; ins; ins = ins-next) {
...
}
=item Basic_block
Basic blocks are the most important structure for optimization. A
basic block identifies a block of instructions that will all execute
in sequence without jumps into or out of that block. All labels will
appear at the beginning of a block, and all conditional or
unconditional jumps will appear at the end. Basic_block structures
are stored as an array of pointers, each with an index that denotes
their position in the array. Block 0 is implicitly the top block. To
iterate through all Basic_blocks, use:
int i;
for (i = 0; i unit-n_basic_blocks; i++) {
...
}
=item Edge
Edges denote the flow of control between Basic_blocks. Edges and
Basic_blocks together make up the basic CFG. Each Basic_block has
*pred_list and *succ_list pointer to the first predecessor edge and
successor edge, respectively. Each edge has a *to and *from pointer
to the Basic_blocks it joins. To iterate through all predecessor
Edges, use:
Edge *pred;
for (pred = to-pred_list; pred; pred=pred-pred_next) {
...
}
=item Loop_info
Loop_info structures denote the presence of loops in the instructions.
They are found by identifying backedges, where control passes from a
tail block to the head of the loop. Loop_info stores the header,
preheader, exit blocks, and depth of the loop.
=item Set
Set is a useful structure for defining sets of integers, which map to
indexes of structures. This is used most often to create sets of
Basic_blocks. Dominators, dominance frontiers, and loops use Set. A
Set must be a defined size, and cannot grow or shrink. Most standard
set operations are implemented: add, contains, copy, equal, union, and
intersection.
=back 4
=head2 Optimizations
Optimizations are organized into an optimization loop within
imc_reg_alloc() in reg_alloc.c. The ordering is based on the amount
of CFG information needed by each group of optimizations:
pre_optimize(), cfg_optimize(), and optimize(). Each optimization
function (group and individual) returns an int, with TRUE denoting
that an optimization has been performed and a change to the code has
been made. The power of the optimizer is that performing one
optimization may often allow another to be performed as well. Once
all optimizations have been run without changes, the optimizer is
finished.
The optimizer loop works as follows:
=over 4
=item 1. Run all pre_optimize() optimizations until none make a change.
=item 2. Build basic block info.
=item 3. Run all cfg_optimize() optimizations. If one makes a
change, go to step 1.
=item 4. Build all other CFG info (dominators, loops, life analysis).
=item 5. Run all optimize() optimizations. If one makes a change, go
to step 1.
=back 4
Two cfg_optimize() or optimize() optimizations cannot be run in a row.
This is because most of these make the CFG information invalid when a
change is performed, and the CFG must be rebuilt.
=head3 Pre optimizer
Optimizations using only Instruction info, no CFG constructed.
=over 4
=item strength_reduce()
converts an expensive instruction to a simpler one
=item if_branch()
Convert if/branch/label constructs to a simpler form
=back 4
=head3 CFG optimizer
Optimizations using Basic_block info. These functions invalidate the
CFG when a change is made.
=over 4
=item branch_branch()
Replaces a branch directly to another branch with a
single branch to the end of the chain
=item unused_label()
Removes unused labels
=item dead_code_remove()
Removes unreachable code
=back 4
=head3 Optimizer
=over 4
=item constant_propagation()
conservative constant propagation, i.e.
replaces 1 + 2 with 3
=item clone_remove()
remove a clone, TURNED OFF
=item used_once()
removes an instruction when the register written is only
used once (only appears in that instruction)
=item loop_optimization()
