Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Sandro Cumerlato]

Try  "  char instead of  '  (char 39). I never use  '  under Windows.

Best regards
Sandro Cumerlato

On 22 Mar 2017 23:47, "James Richters" 
wrote:

> I'm trying to use copyfile and cannot get it to work,  anyone have any
> ideas?
>
> I'm using the procedure below.  My writeln showing the contents of the
> variables seems to indicate they are correct.  The souce file does exist
> and is correct, the destination file does not exitst.
> I get an error 50 which is
>
> ERROR_NOT_SUPPORTED  50 (0x32)
> The request is not supported.
>
> I don't see how copying a file I just created is not supported.  Does
> anyone have any ideas how to get this to work or what I may be doing wrong,
> or if there is a better way I should be copying my file?   I've tried using
> single quotes and double quotes around the file names.
>
> Thanks
>
> James
>
>
> Uses
>Serial,Windows,jwanative,sysutils;
>
> Procedure Copy_Backup_Rename(CBR_Filename:String);
> var
>   fileSource, fileDest: string;
>   resultsofit : Boolean;
>
> Begin
>fileSource := chr(39)+Prog_Drive+Prog_Path+'
> \'+CBR_Filename+'.tmp'+chr(39);
>fileDest := chr(39)+'C:'+Prog_Path+'\'+CBR_Filename+'.temp'+chr(39);
>writeln (filesource+''+filedest);
>resultsofit := CopyFile(PChar(fileSource), PChar(fileDest), False);
>writeln (resultsofit);
>writeln(getlasterror);
> End;
>
> Console output:
> 'C:\ProMill\BitTool.tmp''C:\ProMill\BitTool.temp'
> FALSE
> 50
>
> ___
> fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

I'm trying to use copyfile and cannot get it to work,  anyone have any ideas?

I'm using the procedure below.  My writeln showing the contents of the 
variables seems to indicate they are correct.  The souce file does exist and is 
correct, the destination file does not exitst.
I get an error 50 which is 

ERROR_NOT_SUPPORTED  50 (0x32)
The request is not supported.

I don't see how copying a file I just created is not supported.  Does anyone 
have any ideas how to get this to work or what I may be doing wrong, or if 
there is a better way I should be copying my file?   I've tried using single 
quotes and double quotes around the file names.

Thanks

James


Uses
   Serial,Windows,jwanative,sysutils;

Procedure Copy_Backup_Rename(CBR_Filename:String);
var
  fileSource, fileDest: string;
  resultsofit : Boolean;

Begin
   fileSource := chr(39)+Prog_Drive+Prog_Path+'\'+CBR_Filename+'.tmp'+chr(39);
   fileDest := chr(39)+'C:'+Prog_Path+'\'+CBR_Filename+'.temp'+chr(39);
   writeln (filesource+''+filedest);
   resultsofit := CopyFile(PChar(fileSource), PChar(fileDest), False);
   writeln (resultsofit);
   writeln(getlasterror);
End;

Console output:
'C:\ProMill\BitTool.tmp''C:\ProMill\BitTool.temp'
FALSE
50

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Klaus Hartnegg]

Am 21.03.2017 um 19:58 schrieb James Richters:

Does anyone know of a way to force critical data to be written to a SSD so it's 
not lost during a power failure?


Here is a unix-like sync command for windows:
https://technet.microsoft.com/sv-se/sysinternals/bb897438(en-us).aspx
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[José Mejuto]

El 22/03/2017 a las 19:17, James Richters escribió:


Copy the file to the hard drive with a temporary name
Flush all writes to above file
Delete previous backup file
Rename the original file to backup file name
Rename the new recently copied file to the original name


Hello,

Your problem seems to be that you are using an SSD without a big 
capacitor (or partially damaged one) to preserve writes on power loss. 
This fact combined with how TRIM works could create that effect.


You can try this steps:

1) Copy file content to .bak
2) Create new file with data (.datnew)
3) Close file (This forces SSD to write to an spare block)
4) Delete old file (This forces a TRIM sent to the SSD)
5) Rename file to original name.

In step 4, the delete should send a TRIM which could force a SSD-RAM to 
NAND-SSD write to handle the TRIM. You must not delete 0 bytes (no TRIM) 
nor files with a size less than 1024 (no TRIM in NTFS as most of them 
are stored as resident data in MFT).


--

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

Thanks for the link and information on atomic writes.   It seems like a much 
better method.   My current scheme is to copy everything to a ramdrive upon 
boot.  Here I can do what I want with my files and if the machine crashes, the 
files I started with on the hard drive have not been affected.  Now for the 
tricky part, If I change a file, and want to save the change back to the hard 
drive in a way that is either 100% successful or 0% successful.   My thinking 
is that if I create the new file on my ram drive, and my program is still 
running, I can then perform the following sequence:

Copy the file to the hard drive with a temporary name
Flush all writes to above file
Delete previous backup file
Rename the original file to backup file name
Rename the new recently copied file to the original name

At this point I should have the current data with the original name and a 
backup of the previous data in case something goes wrong before the sequence 
completes.

I would like to do this without shell execute, and it seems my best bet is to 
use windows api calls instead of pascal functions.   I have FlushFileBuffers() 
working.  For the copy should I use MoveFileEx()   Should I use DeleteFile() to 
delete the previous backup and Rename() to rename the files or is there a 
better way?  Will I need to do something for commit the rename to disk?

Thanks for the help with this

James

-Original Message-
From: fpc-pascal [mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of 
Karoly Balogh (Charlie/SGR)
Sent: Wednesday, March 22, 2017 9:13 AM
To: FPC-Pascal users discussions 
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

Hi,

On Wed, 22 Mar 2017, Karoly Balogh (Charlie/SGR) wrote:

> However, please note that transactional file handling on power loss is 
> a quite delicate scenario, so you might not be able to solve all the 
> related problems with this - what happens exactly when you get a power 
> loss during a write, might be matter of pure luck.
>
> The usual way to work this problem around on Linux systems at least, 
> is to write a new file, then do an overwriting rename to the old file name.
> There rename is an "atomic" operation, which will be either committed 
> to the disk or not, but if it was unsuccessful, you won't end up with 
> half-written files. But IIRC Windows didn't support atomic renames. 
> Maybe someone with more Windows knowledge will fix me. You definitely 
> don't want to implement a copy though, and that's anything but atomic.

Actually, this blogpost which I've just found also details the algorithm how 
you should do it, from a Windows perspective:

https://blogs.msdn.microsoft.com/adioltean/2005/12/28/how-to-do-atomic-writes-in-a-file/

Charlie
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org 
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Giuliano Colla]

Il 22/03/2017 14:21, Karoly Balogh (Charlie/SGR) ha scritto:

Hi,

On Wed, 22 Mar 2017, Giuliano Colla wrote:


Il 22/03/2017 13:20, James Richters ha scritto:

No, it is not only freepascal, but not every program either.

A wild guess. The vilain could be the journal logic.

Wild indeed. :)


You might try overwriting the old file instead of clearing it. Something
like:

Assign.
Seek to the beginning of the file. (instead of rewrite)
Write whatever you need.
Truncate.
Close.

This will give the journal a different story to cope with. Maybe it will
help.

Even if this would work (no idea), it builds completely on assumptions and
the behavior might be different depending on the underlying storage
system. It's not atomic as well, because if you get a power loss during
the above process, you could still end up with a broken file with random
contents. So it is dangerous, and could be very hard to reproduce if it
causes problems.

See my other mails about the documented way, and what Microsoft seem to
suggest.


There's no doubt that the best solution would be writing a new file and 
then renaming it, second only to using an UPS to protect against 
accidental power loss, which costs just a fraction of the time value 
spent in this discussion.


My suggestion was just a test to get an idea on how the MS Journaling 
works, and which pitfalls it may hide. If they store/execute the pending 
operations not in the order they where given, but giving somehow 
precedence to the operations faster to execute, and dropping the actions 
which are no more executable, you might end up with a renamed file which 
is still empty, because its creation and renaming are by far much faster 
than writing it!


As far as Microsoft's suggestion are related, a friend of mine with some 
decades of career as a MCSE once told me is that, according his 
experience, the best course to take is always *not* to follow Microsofts 
suggestions, but to do exactly the opposite:-)


Giuliano

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Henry Vermaak]

On Wed, Mar 22, 2017 at 01:50:45PM +0100, Karoly Balogh (Charlie/SGR) wrote:
> The usual way to work this problem around on Linux systems at least, is to
> write a new file, then do an overwriting rename to the old file name.
> There rename is an "atomic" operation, which will be either committed to
> the disk or not, but if it was unsuccessful, you won't end up with
> half-written files. But IIRC Windows didn't support atomic renames. Maybe
> someone with more Windows knowledge will fix me. You definitely don't want
> to implement a copy though, and that's anything but atomic.

I remember discussions about atomic rename for go and python (you can
still find the bug reports online).  The consensus seems to be that
MoveFileEx with MOVEFILE_REPLACE_EXISTING and MOVEFILE_WRITE_THROUGH
flags will be atomic if the file is on the same volume, and if that
volume supports changing the metadata in one transaction (which most do,
afaik).

MSDN recommends ReplaceFile() instead of TxF:

https://msdn.microsoft.com/en-us/library/windows/desktop/hh802690(v=vs.85).aspx

But I _think_ this is just a convenience function that uses
MoveFileEx().

So you can basically follow the same strategy as on POSIX, but with
MoveFileEx().

Henry
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Karoly Balogh (Charlie/SGR)]

Hi,

On Wed, 22 Mar 2017, Giuliano Colla wrote:

> Il 22/03/2017 13:20, James Richters ha scritto:
> > No, it is not only freepascal, but not every program either.
>
> A wild guess. The vilain could be the journal logic.

Wild indeed. :)

> You might try overwriting the old file instead of clearing it. Something
> like:
>
> Assign.
> Seek to the beginning of the file. (instead of rewrite)
> Write whatever you need.
> Truncate.
> Close.
>
> This will give the journal a different story to cope with. Maybe it will
> help.

Even if this would work (no idea), it builds completely on assumptions and
the behavior might be different depending on the underlying storage
system. It's not atomic as well, because if you get a power loss during
the above process, you could still end up with a broken file with random
contents. So it is dangerous, and could be very hard to reproduce if it
causes problems.

See my other mails about the documented way, and what Microsoft seem to
suggest.

Charlie
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Karoly Balogh (Charlie/SGR)]

Hi,

On Wed, 22 Mar 2017, Karoly Balogh (Charlie/SGR) wrote:

> However, please note that transactional file handling on power loss is a
> quite delicate scenario, so you might not be able to solve all the related
> problems with this - what happens exactly when you get a power loss during
> a write, might be matter of pure luck.
>
> The usual way to work this problem around on Linux systems at least, is to
> write a new file, then do an overwriting rename to the old file name.
> There rename is an "atomic" operation, which will be either committed to
> the disk or not, but if it was unsuccessful, you won't end up with
> half-written files. But IIRC Windows didn't support atomic renames. Maybe
> someone with more Windows knowledge will fix me. You definitely don't want
> to implement a copy though, and that's anything but atomic.

Actually, this blogpost which I've just found also details the algorithm
how you should do it, from a Windows perspective:

https://blogs.msdn.microsoft.com/adioltean/2005/12/28/how-to-do-atomic-writes-in-a-file/

Charlie
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Giuliano Colla]

Il 22/03/2017 13:20, James Richters ha scritto:

No, it is not only freepascal, but not every program either.


A wild guess. The vilain could be the journal logic.

The journal tells that the write operation has not been completed at 
power off, and restores the last valid version of the file, which was 
empty, because of the rewrite you performed after assigning.


You might try overwriting the old file instead of clearing it. Something 
like:


Assign.
Seek to the beginning of the file. (instead of rewrite)
Write whatever you need.
Truncate.
Close.

This will give the journal a different story to cope with. Maybe it will 
help.


Giuliano



___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Karoly Balogh (Charlie/SGR)]

Hi,

On Wed, 22 Mar 2017, James Richters wrote:

> No, it is not only freepascal, but not every program either.

I'm no Windows expert, but Windows API seems to have a way do to this
directly. CreateFile() API seems to have a flag, which has a value
FILE_FLAG_WRITE_THROUGH, which will cause the file not go through the
write cache, but gets written directly to the underlying disk, without
delays.

However, this obviously has performance implications on write (will be a
lot slower), and as the documentation notes, it's not supported on every
hardware or file system. Also, I'm not sure if you can do this somehow
together with Free Pascal's file handling API, or you have to restrict
yourself to using direct Windows API calls entirely for writing this file.

Relevant MSDN page:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx#caching_behavior

However, please note that transactional file handling on power loss is a
quite delicate scenario, so you might not be able to solve all the related
problems with this - what happens exactly when you get a power loss during
a write, might be matter of pure luck.

The usual way to work this problem around on Linux systems at least, is to
write a new file, then do an overwriting rename to the old file name.
There rename is an "atomic" operation, which will be either committed to
the disk or not, but if it was unsuccessful, you won't end up with
half-written files. But IIRC Windows didn't support atomic renames. Maybe
someone with more Windows knowledge will fix me. You definitely don't want
to implement a copy though, and that's anything but atomic.

Charlie
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

No, it is not only freepascal, but not every program either. 

I have just done some quick tests check this.  I have not done any real testing 
to prove whether these are repeatable all the time or not yet.
A file saved with notepad ++ is lost
Partial email drafts in outlook are saved
A file created with Echo "test" > test.txt is saved
Copying a file from the command line seems to end up saving both the source and 
destination.I tried saving a file with notepad++ then copying the file, and 
after the power failure, both the source and destination were complete.   But 
saving with notepad ++ will always be a string of 00s.  

I wonder if this is a 32/64 bit issue.   Notepad ++ and my free pascal programs 
are both 32bit. 

Perhaps until I can figure out what the copy command is doing to save the file 
to disk, I can just shell execute a copy command on my files and that will end 
up saving them.  I guess if I did that I could run my entire program from a ram 
drive and save some SSD activity and only copy the important files back to the 
SSD and only when they actually change.  

James





-Original Message-
From: fpc-pascal [mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of 
Santiago A.
Sent: Wednesday, March 22, 2017 4:40 AM
To: FPC-Pascal users discussions 
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

El 21/03/2017 a las 15:39, James Richters escribió:
> I should note that these systems are all using Samsung SSDs in them, 
> perhaps there is some SSD weirdness going on?
Yes, I think so.  SSD is special.

The SSD technology like any flash memories have a limited number of rewrites, 
so there are strategies to rewrite as less as possible. In the case of SSD they 
are even more aggressive, because they are expensive .Some of these strategies 
are implemented in the OS in driver level, but some in firmware of the device.  
So, it is possible that OS can't completely force the real write.

> I also notice it's not just this
> one file that is affected, but EVERY SINGLE FILE I create with 
> freepascal,
Only with freepascal?

Try it for a file generated from command line

echo "test" > filetest.txt

And do the same checks... edit notepad++... turnoff power etc
> configured a certain way.   I really need these tiny files to survive power
> failures.
Power failures are always a problem for persistent storage. Caching is a 
trade-off between speed and security. In the case of SSD there is a third 
point: Minimize the real writes on disk. So, if you want security against power 
failures, SSD is not the best idea.

By the way. What about a SAI? ;-)


--
Saludos

Santiago A.
s...@ciberpiula.net

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org 
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Santiago A.]

El 21/03/2017 a las 15:39, James Richters escribió:
> I should note that these systems are all using Samsung SSDs in them, perhaps
> there is some SSD weirdness going on?
Yes, I think so.  SSD is special.

The SSD technology like any flash memories have a limited number of
rewrites, so there are strategies to rewrite as less as possible. In the
case of SSD they are even more aggressive, because they are expensive
.Some of these strategies are implemented in the OS in driver level, but
some in firmware of the device.  So, it is possible that OS can't
completely force the real write.

> I also notice it's not just this
> one file that is affected, but EVERY SINGLE FILE I create with freepascal,
Only with freepascal?

Try it for a file generated from command line

echo "test" > filetest.txt

And do the same checks... edit notepad++... turnoff power etc
> configured a certain way.   I really need these tiny files to survive power
> failures.
Power failures are always a problem for persistent storage. Caching is a
trade-off between speed and security. In the case of SSD there is a
third point: Minimize the real writes on disk. So, if you want security
against power failures, SSD is not the best idea.

By the way. What about a SAI? ;-)


-- 
Saludos

Santiago A.
s...@ciberpiula.net

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Henry Vermaak]

On Tue, Mar 21, 2017 at 06:01:17PM -0700, Ralf Quint wrote:
> On 3/21/2017 11:58 AM, James Richters wrote:
> >I have not tried FlushFileBuffers() yet, so I just tried it now,  I am 
> >getting the same results.
> Well, that is expected, as the issue has nothing to do with your program,
> it's the caching of the operating system that is responsible for the actual
> physical write to the disk.

No, FlushFileBuffers() flushes the OS cache - see the MSDN page.  The
problem here seems to be that the SSD has its own cache (in the FTL).

Henry
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal