Re: optimizer?

2024-01-07 Thread ToddAndMargo via perl6-users

On 1/7/24 15:18, ToddAndMargo via perl6-users wrote:
sub DirectoryExists( Str $DirPath --> Bool ) { return 
"$DirPath".IO.d.Bool; }  # $Full DirPath format is `H:\MyDocsBackup`

Problem:  sometimes DirectoryExists lies about
backup1 existing.

Here is what I am up against.

You will note the "junction" points.  They should not be backed up.
But they are an being creasted as simple directories.

C:\Users\accounting\Documents>dir /a | grep -i junction
09/14/2020  07:39 AM My Music [C:\Users\accounting\Music]
09/14/2020  07:39 AM My Pictures 
09/14/2020  07:39 AM My Videos 

This is the backup directory after a Windows Explorter erase.
Both DirectoryExists ($DirPath".IO.d.Bool" and Windows
Explorer with Hidden file turned on says it does not exist.

You will note that the command line says it does indeed exist.
And it does.

F:\MyDocsBackup\backup3\Documents 2023-08-15 16;28;47 (Full)>dir /a
 Volume in drive F is BACKUP
 Volume Serial Number is 0866-7D92

 Directory of F:\MyDocsBackup\backup3\Documents 2023-08-15 16;28;47 (Full)

01/05/2024  07:49 PM  .
01/05/2024  07:49 PM  ..
08/15/2023  03:29 PM  My Music
08/15/2023  03:29 PM  My Pictures
08/15/2023  03:29 PM  My Videos
   0 File(s)  0 bytes
   5 Dir(s)  480,294,719,488 bytes free

Windows is such a horrid kluge.



Re: optimizer?

2024-01-07 Thread ToddAndMargo via perl6-users

On 1/7/24 06:16, Parrot Raiser wrote:

What's the reason behind the request?

Hi Parrot,

The reason is to eliminate the Optimizer as a
source of weird problems.

I have written a wrapper for Cobian Backup/Reflector
that rotates backup sets.  Cobain has is own database
to keep track of backup sets, but it falls apart when
you are rotating several removable drives as the target.

Removable drives in rotation defeats ransomware.

So, my program locates the drive letter the target
resides on based on drive label (it will do letter too),
then counts the backup sets.  If these sets exceed
the limit of sets, I remove the oldest.

For example maximum backup sets is set to 4.  Then backup4
gets erased, backup3 gets renamed to backup4, backup2
gets renamed to backup3, backup1 gets renamed to backup2.

At this point there should be no backup1.  And I check for

sub DirectoryExists( Str $DirPath --> Bool ) { return 
"$DirPath".IO.d.Bool; }  # $Full DirPath format is `H:\MyDocsBackup`

Problem:  sometimes DirectoryExists lies about
backup1 existing.  Failure to rotate has to do with
something is the backup set that won't move.  These
are typically system files.  When I find them, I enter
into Cobian's exclude mask:


I have traced this down to the "size" of backup1 too.
A directory in backup1 that refuses to move in
a large backups, will move in a simple test task.

Now to workaround the issue of DirectoryExists not
always returning the correct answer, I have written
my own code for it:

sub IsItReallyReallyGone( Str $Path --> Bool ) {
# checks to see if a file or directory is really gone and not stuck in cache
   my Bool $Gone = False;
   my Str $RtnStr = "";

   # Note: "File Not Found" is sent to the STDERR
   $RtnStr  = RunCmd Q[dir ] ~ $Path ~ Q[ /A  2>&1];
   if $ "file not found" )  { $Gone = True ; }

   # if %CommandLine  { print "   IsItReallyReallyGone:\n   Gone 
= <$Gone>\n   Path = <$Path>\n   RtnStr = <$RtnStr>\n"; }
   if %CommandLine  { print "   IsItReallyReallyGone: $Path is 
$Gone\n"; }

   return $Gone;

And I flush a lot(same as Linux's "sync"):

sub FlushDriveCache( Str $Drive )  {
   my Str $SingleLetter = $Drive.uc.substr( 0..0 );
   my Str $RtnStr   = "";

   print "   Flushing " ~ $SingleLetter ~ "'s drive cache\n";
   $RtnStr = RunCmd "powershell.exe Write-VolumeCache " ~ 
$SingleLetter, True;


And IsItReallyReallyGone is dead nuts totally, repeatable accurate.
If backup1 is really not gone, a flush and a second delete usually
does the trick.  (I can repeat this double erase with Windows
Explorer.  First time through it coughs on certain files.
Second time through, it gets them all.)

sub FastRmdir( $DirPath )  {
   RunCmd( "del /f/s/q " ~ $DirPath ~ " > nul" );
   RunCmd( "rmdir " ~ $DirPath ~ " /S /Q" );

Now if backup1 still exists and is not rotated, Cobian keeps
writing to it and eventually the target drive fill up,
meaning no more new backups.

And that is a problem, especially with trying to get my
customer's to actually read their backup reports that
Cobian eMails them (and me, but I only check them
periodically and only as a courtesy).

So in summary, the issue is probably a cache issue with
Raku's IO commands not keeping up with what Windows
is doing.  Disabling the Optimizer would remove that
issues as a problem.  And Raku's IO commands needs
some work when dealing with large amounts of a data
in cache.

Sorry for being so long winded.


My program's help file:

>raku CobianWrapper.pl6 --help
Welcome to CobianWrapper.pl6

CobianWrapper.pl6 usage:

 Drive letter or partition label plus path to backup directory.
 for a label, surround the label with square brackets [].
 Note: the backup directory name must end in a 1
 nokill will disable the killing of programs known to be
 Shadow Copy (VSS) unfriendly

  --backup_path  [BACKUP]\MyDocsBackup\backup1
  --rotates  2
  --nokill   False
  --help False

For example
raku C:\NtUtil\CobianWrapper.pl6 --rotates 6 --backup_path 
raku C:\NtUtil\CobianWrapper.pl6 --rotates 4 --backup_path 

more? (y/n)  y

Sample Cobian Pre-backup Event call to CobianWrapper.pl6:

Note: Events, Command Line is currently not working Cobian Reflector as 
of 1.0.0

Execute and wait: "C:\Program Files\Rakudo\bin\raku.exe" 
"C:\NtUtil\CobianWrapper.pl6 --rotates 20 --backup_path 

Sample Cobian File, Destination for use with CobianWrapper.pl6:

Sample Cobian Post-backup Event call to CobianWrapper.pl6 (requires 


Re: optimizer?

2024-01-07 Thread ToddAndMargo via perl6-users

On 7 Jan 2024, at 07:09, ToddAndMargo via perl6-users  

Hi All,

Is there a switch on the command line to disable the code optimizer?

Many thanks,

On 1/7/24 04:24, Elizabeth Mattijsen wrote:
> $ raku --help
> ...
> --optimize=level use the given level of optimization (0..3)
> ...

Thank you!

Re: optimizer?

2024-01-07 Thread Parrot Raiser
What's the reason behind the request?

On Sun, Jan 7, 2024 at 7:24 AM Elizabeth Mattijsen  wrote:

> $ raku --help
> ...
> --optimize=level use the given level of optimization (0..3)
> ...
> > On 7 Jan 2024, at 07:09, ToddAndMargo via perl6-users <
>> wrote:
> >
> > Hi All,
> >
> > Is there a switch on the command line to disable the code optimizer?
> >
> > Many thanks,
> > -T

Re: optimizer?

2024-01-07 Thread Elizabeth Mattijsen
$ raku --help

--optimize=level use the given level of optimization (0..3)

> On 7 Jan 2024, at 07:09, ToddAndMargo via perl6-users  
> wrote:
> Hi All,
> Is there a switch on the command line to disable the code optimizer?
> Many thanks,
> -T

Re: Optimizer Documentation

2005-09-07 Thread Leopold Toetsch

Curtis Rawls wrote:

As part of my Summer of Code project, I wanted to improve the
documentation of the optimizer. 

Please svn add the .pod to docs/dev


Re: Optimizer Documentation

2005-09-01 Thread Brent 'Dax' Royal-Gordon
 Also, this is in plain text format.  What is the standard way to
 convert a document to POD format?

By rewriting it.  Only took me a few minutes:

=head1 NANE

optimizer.pod - About the IMCC optimizer


This document describes how the IMCC optimizer works.


The objective of the IMCC optimizer is to take a PASM function as
input and  apply code-improving transformations to it to be more
efficient, i.e. improving execution time and reducing code size.  It
must do this while preserving the same code behavior.

=head2 Data Structures

The optimizer uses a number of data structures to build a model of the
code to be optimized.

=over 4

=item IMC_Unit

The IMC_Unit structure contains all the information known about a
function.It is passed in to each optimizer method.

=item Instruction

Each instruction line has an Instruction structure.  Pointers to the
first and last Instruction are stored in IMC_Unit.  Instructions are
stored as a linked list.  To iterate through all Instructions, use:

Instruction *ins;
for (ins = unit-instructions; ins; ins = ins-next) {

=item Basic_block

Basic blocks are the most important structure for optimization.  A
basic block identifies a block of instructions that will all execute
in sequence without jumps into or out of that block.  All labels will
appear at the beginning of a block, and all conditional or
unconditional jumps will appear at the end.  Basic_block structures
are stored as an array of pointers, each with an index that denotes
their position in the array.  Block 0 is implicitly the top block.  To
iterate through all Basic_blocks, use:

int i;
for (i = 0; i  unit-n_basic_blocks; i++) {

=item Edge

Edges denote the flow of control between Basic_blocks.  Edges and
Basic_blocks together make up the basic CFG.  Each Basic_block has
*pred_list and *succ_list pointer to the first predecessor edge and
successor edge, respectively.  Each edge has a *to and *from pointer
to the  Basic_blocks it joins.  To iterate through all predecessor
Edges, use:

Edge *pred;
for (pred = to-pred_list; pred; pred=pred-pred_next) {

=item Loop_info

Loop_info structures denote the presence of loops in the instructions.
 They are found by identifying backedges, where control passes from a
tail block to the head of the loop.  Loop_info stores the header,
preheader, exit blocks, and depth of the loop.

=item Set

Set is a useful structure for defining sets of integers, which map to
indexes of structures.  This is used most often to create sets of
Basic_blocks.  Dominators, dominance frontiers, and loops use Set.  A
Set must be a defined size, and cannot grow or shrink.  Most standard
set operations are implemented: add, contains, copy, equal, union, and

=back 4

=head2 Optimizations

Optimizations are organized into an optimization loop within
imc_reg_alloc() in reg_alloc.c.  The ordering is based on the amount
of CFG information needed by each group of optimizations:
pre_optimize(), cfg_optimize(), and optimize().  Each optimization
function (group and individual) returns an int, with TRUE denoting
that an optimization has been performed and a change to the code has
been made.  The power of the optimizer is that performing one
optimization may often allow another to be performed as well.  Once
all optimizations have been run without changes, the optimizer is

The optimizer loop works as follows:

=over 4

=item 1.  Run all pre_optimize() optimizations until none make a change.

=item 2.  Build basic block info.

=item 3.  Run all cfg_optimize() optimizations.  If one makes a
change, go to step 1.

=item 4.  Build all other CFG info (dominators, loops, life analysis).

=item 5.  Run all optimize() optimizations.  If one makes a change, go
to step 1.

=back 4

Two cfg_optimize() or optimize() optimizations cannot be run in a row.
This is because most of these make the CFG information invalid when a
change is performed, and the CFG must be rebuilt.

=head3 Pre optimizer

Optimizations using only Instruction info, no CFG constructed.

=over 4

=item strength_reduce()

converts an expensive instruction to a simpler one

=item if_branch()

Convert if/branch/label constructs to a simpler form

=back 4

=head3 CFG optimizer

Optimizations using Basic_block info.  These functions invalidate the
CFG when a change is made.

=over 4

=item branch_branch()

Replaces a branch directly to another branch with a
single branch to the end of the chain

=item unused_label()

Removes unused labels

=item dead_code_remove()

Removes unreachable code

=back 4

=head3 Optimizer

=over 4

=item constant_propagation()

conservative constant propagation, i.e.
replaces 1 + 2 with 3

=item clone_remove()

remove a clone, TURNED OFF

=item used_once()

removes an instruction when the register written is only
used once (only appears in that instruction)

=item loop_optimization()