Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Sandro Cumerlato]

No. The "" is a shell/cmdline convention, and API calls are generally not
shell.


Thank you Marco for the clarification.

Sandro Cumerlato
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Marco van de Voort]

In our previous episode, Sandro Cumerlato said:
> ' delimiters are mandatory within this example because we are writing paths
> as string parameters directly into the source code.
> 
> I believe that I should use both ' and " in case of spaces within paths:
> 
> CopyFile('"c:\my silly path\autoexec.bat"','"c:\windows\temp\my silly
> path\autoexec.bat.backup"');

No. The "" is a shell/cmdline convention, and API calls are generally not
shell. 
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Sandro Cumerlato]

On 23 Mar 2017 11:34, "James Richters" 
wrote:

Thanks for the response, it got me on the right track and I got it working.




Happy to help you!

I originally did try “ instead of ‘ and have the same results.  I was using
‘  because of the example here: http://wiki.freepascal.org/CopyFile




> CopyFile('c:\autoexec.bat','c:\windows\temp\autoexec.bat.backup');

' delimiters are mandatory within this example because we are writing paths
as string parameters directly into the source code.

I believe that I should use both ' and " in case of spaces within paths:

CopyFile('"c:\my silly path\autoexec.bat"','"c:\windows\temp\my silly
path\autoexec.bat.backup"');

or is the CopyFile function smart enough to include delimiters
automatically?

Best regards,
Sandro Cumerlato
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

Thanks for the response, it got me on the right track and I got it working.

 

I originally did try “ instead of ‘ and have the same results.  I was using ‘  
because of the example here: http://wiki.freepascal.org/CopyFile 

 

I was able to get it to work without any ‘ or “  I even did a test with a space 
in the file name and it worked fine without  “   

It appears The function must put “ around it for you.

 

James

 

From: fpc-pascal [mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of 
Sandro Cumerlato
Sent: Thursday, March 23, 2017 1:30 AM
To: FPC-Pascal users discussions <fpc-pascal@lists.freepascal.org>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

 

Try  "  char instead of  '  (char 39). I never use  '  under Windows. 

 

Best regards

Sandro Cumerlato 

 

On 22 Mar 2017 23:47, "James Richters" <ja...@productionautomation.net 
<mailto:ja...@productionautomation.net> > wrote:

I'm trying to use copyfile and cannot get it to work,  anyone have any ideas?

I'm using the procedure below.  My writeln showing the contents of the 
variables seems to indicate they are correct.  The souce file does exist and is 
correct, the destination file does not exitst.
I get an error 50 which is

ERROR_NOT_SUPPORTED  50 (0x32)
The request is not supported.

I don't see how copying a file I just created is not supported.  Does anyone 
have any ideas how to get this to work or what I may be doing wrong, or if 
there is a better way I should be copying my file?   I've tried using single 
quotes and double quotes around the file names.

Thanks

James


Uses
   Serial,Windows,jwanative,sysutils;

Procedure Copy_Backup_Rename(CBR_Filename:String);
var
  fileSource, fileDest: string;
  resultsofit : Boolean;

Begin
   fileSource := chr(39)+Prog_Drive+Prog_Path+'\'+CBR_Filename+'.tmp'+chr(39);
   fileDest := chr(39)+'C:'+Prog_Path+'\'+CBR_Filename+'.temp'+chr(39);
   writeln (filesource+''+filedest);
   resultsofit := CopyFile(PChar(fileSource), PChar(fileDest), False);
   writeln (resultsofit);
   writeln(getlasterror);
End;

Console output:
'C:\ProMill\BitTool.tmp''C:\ProMill\BitTool.temp'
FALSE
50

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org 
<mailto:fpc-pascal@lists.freepascal.org> 
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Sandro Cumerlato]

Try  "  char instead of  '  (char 39). I never use  '  under Windows.

Best regards
Sandro Cumerlato

On 22 Mar 2017 23:47, "James Richters" 
wrote:

> I'm trying to use copyfile and cannot get it to work,  anyone have any
> ideas?
>
> I'm using the procedure below.  My writeln showing the contents of the
> variables seems to indicate they are correct.  The souce file does exist
> and is correct, the destination file does not exitst.
> I get an error 50 which is
>
> ERROR_NOT_SUPPORTED  50 (0x32)
> The request is not supported.
>
> I don't see how copying a file I just created is not supported.  Does
> anyone have any ideas how to get this to work or what I may be doing wrong,
> or if there is a better way I should be copying my file?   I've tried using
> single quotes and double quotes around the file names.
>
> Thanks
>
> James
>
>
> Uses
>Serial,Windows,jwanative,sysutils;
>
> Procedure Copy_Backup_Rename(CBR_Filename:String);
> var
>   fileSource, fileDest: string;
>   resultsofit : Boolean;
>
> Begin
>fileSource := chr(39)+Prog_Drive+Prog_Path+'
> \'+CBR_Filename+'.tmp'+chr(39);
>fileDest := chr(39)+'C:'+Prog_Path+'\'+CBR_Filename+'.temp'+chr(39);
>writeln (filesource+''+filedest);
>resultsofit := CopyFile(PChar(fileSource), PChar(fileDest), False);
>writeln (resultsofit);
>writeln(getlasterror);
> End;
>
> Console output:
> 'C:\ProMill\BitTool.tmp''C:\ProMill\BitTool.temp'
> FALSE
> 50
>
> ___
> fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
> http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

I'm trying to use copyfile and cannot get it to work,  anyone have any ideas?

I'm using the procedure below.  My writeln showing the contents of the 
variables seems to indicate they are correct.  The souce file does exist and is 
correct, the destination file does not exitst.
I get an error 50 which is 

ERROR_NOT_SUPPORTED  50 (0x32)
The request is not supported.

I don't see how copying a file I just created is not supported.  Does anyone 
have any ideas how to get this to work or what I may be doing wrong, or if 
there is a better way I should be copying my file?   I've tried using single 
quotes and double quotes around the file names.

Thanks

James


Uses
   Serial,Windows,jwanative,sysutils;

Procedure Copy_Backup_Rename(CBR_Filename:String);
var
  fileSource, fileDest: string;
  resultsofit : Boolean;

Begin
   fileSource := chr(39)+Prog_Drive+Prog_Path+'\'+CBR_Filename+'.tmp'+chr(39);
   fileDest := chr(39)+'C:'+Prog_Path+'\'+CBR_Filename+'.temp'+chr(39);
   writeln (filesource+''+filedest);
   resultsofit := CopyFile(PChar(fileSource), PChar(fileDest), False);
   writeln (resultsofit);
   writeln(getlasterror);
End;

Console output:
'C:\ProMill\BitTool.tmp''C:\ProMill\BitTool.temp'
FALSE
50

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Klaus Hartnegg]

Am 21.03.2017 um 19:58 schrieb James Richters:

Does anyone know of a way to force critical data to be written to a SSD so it's 
not lost during a power failure?


Here is a unix-like sync command for windows:
https://technet.microsoft.com/sv-se/sysinternals/bb897438(en-us).aspx
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[José Mejuto]

El 22/03/2017 a las 19:17, James Richters escribió:


Copy the file to the hard drive with a temporary name
Flush all writes to above file
Delete previous backup file
Rename the original file to backup file name
Rename the new recently copied file to the original name


Hello,

Your problem seems to be that you are using an SSD without a big 
capacitor (or partially damaged one) to preserve writes on power loss. 
This fact combined with how TRIM works could create that effect.


You can try this steps:

1) Copy file content to .bak
2) Create new file with data (.datnew)
3) Close file (This forces SSD to write to an spare block)
4) Delete old file (This forces a TRIM sent to the SSD)
5) Rename file to original name.

In step 4, the delete should send a TRIM which could force a SSD-RAM to 
NAND-SSD write to handle the TRIM. You must not delete 0 bytes (no TRIM) 
nor files with a size less than 1024 (no TRIM in NTFS as most of them 
are stored as resident data in MFT).


--

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

Thanks for the link and information on atomic writes.   It seems like a much 
better method.   My current scheme is to copy everything to a ramdrive upon 
boot.  Here I can do what I want with my files and if the machine crashes, the 
files I started with on the hard drive have not been affected.  Now for the 
tricky part, If I change a file, and want to save the change back to the hard 
drive in a way that is either 100% successful or 0% successful.   My thinking 
is that if I create the new file on my ram drive, and my program is still 
running, I can then perform the following sequence:

Copy the file to the hard drive with a temporary name
Flush all writes to above file
Delete previous backup file
Rename the original file to backup file name
Rename the new recently copied file to the original name

At this point I should have the current data with the original name and a 
backup of the previous data in case something goes wrong before the sequence 
completes.

I would like to do this without shell execute, and it seems my best bet is to 
use windows api calls instead of pascal functions.   I have FlushFileBuffers() 
working.  For the copy should I use MoveFileEx()   Should I use DeleteFile() to 
delete the previous backup and Rename() to rename the files or is there a 
better way?  Will I need to do something for commit the rename to disk?

Thanks for the help with this

James

-Original Message-
From: fpc-pascal [mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of 
Karoly Balogh (Charlie/SGR)
Sent: Wednesday, March 22, 2017 9:13 AM
To: FPC-Pascal users discussions <fpc-pascal@lists.freepascal.org>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

Hi,

On Wed, 22 Mar 2017, Karoly Balogh (Charlie/SGR) wrote:

> However, please note that transactional file handling on power loss is 
> a quite delicate scenario, so you might not be able to solve all the 
> related problems with this - what happens exactly when you get a power 
> loss during a write, might be matter of pure luck.
>
> The usual way to work this problem around on Linux systems at least, 
> is to write a new file, then do an overwriting rename to the old file name.
> There rename is an "atomic" operation, which will be either committed 
> to the disk or not, but if it was unsuccessful, you won't end up with 
> half-written files. But IIRC Windows didn't support atomic renames. 
> Maybe someone with more Windows knowledge will fix me. You definitely 
> don't want to implement a copy though, and that's anything but atomic.

Actually, this blogpost which I've just found also details the algorithm how 
you should do it, from a Windows perspective:

https://blogs.msdn.microsoft.com/adioltean/2005/12/28/how-to-do-atomic-writes-in-a-file/

Charlie
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org 
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Giuliano Colla]

Il 22/03/2017 14:21, Karoly Balogh (Charlie/SGR) ha scritto:

Hi,

On Wed, 22 Mar 2017, Giuliano Colla wrote:


Il 22/03/2017 13:20, James Richters ha scritto:

No, it is not only freepascal, but not every program either.

A wild guess. The vilain could be the journal logic.

Wild indeed. :)


You might try overwriting the old file instead of clearing it. Something
like:

Assign.
Seek to the beginning of the file. (instead of rewrite)
Write whatever you need.
Truncate.
Close.

This will give the journal a different story to cope with. Maybe it will
help.

Even if this would work (no idea), it builds completely on assumptions and
the behavior might be different depending on the underlying storage
system. It's not atomic as well, because if you get a power loss during
the above process, you could still end up with a broken file with random
contents. So it is dangerous, and could be very hard to reproduce if it
causes problems.

See my other mails about the documented way, and what Microsoft seem to
suggest.


There's no doubt that the best solution would be writing a new file and 
then renaming it, second only to using an UPS to protect against 
accidental power loss, which costs just a fraction of the time value 
spent in this discussion.


My suggestion was just a test to get an idea on how the MS Journaling 
works, and which pitfalls it may hide. If they store/execute the pending 
operations not in the order they where given, but giving somehow 
precedence to the operations faster to execute, and dropping the actions 
which are no more executable, you might end up with a renamed file which 
is still empty, because its creation and renaming are by far much faster 
than writing it!


As far as Microsoft's suggestion are related, a friend of mine with some 
decades of career as a MCSE once told me is that, according his 
experience, the best course to take is always *not* to follow Microsofts 
suggestions, but to do exactly the opposite:-)


Giuliano

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Henry Vermaak]

On Wed, Mar 22, 2017 at 01:50:45PM +0100, Karoly Balogh (Charlie/SGR) wrote:
> The usual way to work this problem around on Linux systems at least, is to
> write a new file, then do an overwriting rename to the old file name.
> There rename is an "atomic" operation, which will be either committed to
> the disk or not, but if it was unsuccessful, you won't end up with
> half-written files. But IIRC Windows didn't support atomic renames. Maybe
> someone with more Windows knowledge will fix me. You definitely don't want
> to implement a copy though, and that's anything but atomic.

I remember discussions about atomic rename for go and python (you can
still find the bug reports online).  The consensus seems to be that
MoveFileEx with MOVEFILE_REPLACE_EXISTING and MOVEFILE_WRITE_THROUGH
flags will be atomic if the file is on the same volume, and if that
volume supports changing the metadata in one transaction (which most do,
afaik).

MSDN recommends ReplaceFile() instead of TxF:

https://msdn.microsoft.com/en-us/library/windows/desktop/hh802690(v=vs.85).aspx

But I _think_ this is just a convenience function that uses
MoveFileEx().

So you can basically follow the same strategy as on POSIX, but with
MoveFileEx().

Henry
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Karoly Balogh (Charlie/SGR)]

Hi,

On Wed, 22 Mar 2017, Giuliano Colla wrote:

> Il 22/03/2017 13:20, James Richters ha scritto:
> > No, it is not only freepascal, but not every program either.
>
> A wild guess. The vilain could be the journal logic.

Wild indeed. :)

> You might try overwriting the old file instead of clearing it. Something
> like:
>
> Assign.
> Seek to the beginning of the file. (instead of rewrite)
> Write whatever you need.
> Truncate.
> Close.
>
> This will give the journal a different story to cope with. Maybe it will
> help.

Even if this would work (no idea), it builds completely on assumptions and
the behavior might be different depending on the underlying storage
system. It's not atomic as well, because if you get a power loss during
the above process, you could still end up with a broken file with random
contents. So it is dangerous, and could be very hard to reproduce if it
causes problems.

See my other mails about the documented way, and what Microsoft seem to
suggest.

Charlie
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Karoly Balogh (Charlie/SGR)]

Hi,

On Wed, 22 Mar 2017, Karoly Balogh (Charlie/SGR) wrote:

> However, please note that transactional file handling on power loss is a
> quite delicate scenario, so you might not be able to solve all the related
> problems with this - what happens exactly when you get a power loss during
> a write, might be matter of pure luck.
>
> The usual way to work this problem around on Linux systems at least, is to
> write a new file, then do an overwriting rename to the old file name.
> There rename is an "atomic" operation, which will be either committed to
> the disk or not, but if it was unsuccessful, you won't end up with
> half-written files. But IIRC Windows didn't support atomic renames. Maybe
> someone with more Windows knowledge will fix me. You definitely don't want
> to implement a copy though, and that's anything but atomic.

Actually, this blogpost which I've just found also details the algorithm
how you should do it, from a Windows perspective:

https://blogs.msdn.microsoft.com/adioltean/2005/12/28/how-to-do-atomic-writes-in-a-file/

Charlie
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Giuliano Colla]

Il 22/03/2017 13:20, James Richters ha scritto:

No, it is not only freepascal, but not every program either.


A wild guess. The vilain could be the journal logic.

The journal tells that the write operation has not been completed at 
power off, and restores the last valid version of the file, which was 
empty, because of the rewrite you performed after assigning.


You might try overwriting the old file instead of clearing it. Something 
like:


Assign.
Seek to the beginning of the file. (instead of rewrite)
Write whatever you need.
Truncate.
Close.

This will give the journal a different story to cope with. Maybe it will 
help.


Giuliano



___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Karoly Balogh (Charlie/SGR)]

Hi,

On Wed, 22 Mar 2017, James Richters wrote:

> No, it is not only freepascal, but not every program either.

I'm no Windows expert, but Windows API seems to have a way do to this
directly. CreateFile() API seems to have a flag, which has a value
FILE_FLAG_WRITE_THROUGH, which will cause the file not go through the
write cache, but gets written directly to the underlying disk, without
delays.

However, this obviously has performance implications on write (will be a
lot slower), and as the documentation notes, it's not supported on every
hardware or file system. Also, I'm not sure if you can do this somehow
together with Free Pascal's file handling API, or you have to restrict
yourself to using direct Windows API calls entirely for writing this file.

Relevant MSDN page:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx#caching_behavior

However, please note that transactional file handling on power loss is a
quite delicate scenario, so you might not be able to solve all the related
problems with this - what happens exactly when you get a power loss during
a write, might be matter of pure luck.

The usual way to work this problem around on Linux systems at least, is to
write a new file, then do an overwriting rename to the old file name.
There rename is an "atomic" operation, which will be either committed to
the disk or not, but if it was unsuccessful, you won't end up with
half-written files. But IIRC Windows didn't support atomic renames. Maybe
someone with more Windows knowledge will fix me. You definitely don't want
to implement a copy though, and that's anything but atomic.

Charlie
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

No, it is not only freepascal, but not every program either. 

I have just done some quick tests check this.  I have not done any real testing 
to prove whether these are repeatable all the time or not yet.
A file saved with notepad ++ is lost
Partial email drafts in outlook are saved
A file created with Echo "test" > test.txt is saved
Copying a file from the command line seems to end up saving both the source and 
destination.I tried saving a file with notepad++ then copying the file, and 
after the power failure, both the source and destination were complete.   But 
saving with notepad ++ will always be a string of 00s.  

I wonder if this is a 32/64 bit issue.   Notepad ++ and my free pascal programs 
are both 32bit. 

Perhaps until I can figure out what the copy command is doing to save the file 
to disk, I can just shell execute a copy command on my files and that will end 
up saving them.  I guess if I did that I could run my entire program from a ram 
drive and save some SSD activity and only copy the important files back to the 
SSD and only when they actually change.  

James





-Original Message-
From: fpc-pascal [mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of 
Santiago A.
Sent: Wednesday, March 22, 2017 4:40 AM
To: FPC-Pascal users discussions <fpc-pascal@lists.freepascal.org>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

El 21/03/2017 a las 15:39, James Richters escribió:
> I should note that these systems are all using Samsung SSDs in them, 
> perhaps there is some SSD weirdness going on?
Yes, I think so.  SSD is special.

The SSD technology like any flash memories have a limited number of rewrites, 
so there are strategies to rewrite as less as possible. In the case of SSD they 
are even more aggressive, because they are expensive .Some of these strategies 
are implemented in the OS in driver level, but some in firmware of the device.  
So, it is possible that OS can't completely force the real write.

> I also notice it's not just this
> one file that is affected, but EVERY SINGLE FILE I create with 
> freepascal,
Only with freepascal?

Try it for a file generated from command line

echo "test" > filetest.txt

And do the same checks... edit notepad++... turnoff power etc
> configured a certain way.   I really need these tiny files to survive power
> failures.
Power failures are always a problem for persistent storage. Caching is a 
trade-off between speed and security. In the case of SSD there is a third 
point: Minimize the real writes on disk. So, if you want security against power 
failures, SSD is not the best idea.

By the way. What about a SAI? ;-)


--
Saludos

Santiago A.
s...@ciberpiula.net

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org 
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Santiago A.]

El 21/03/2017 a las 15:39, James Richters escribió:
> I should note that these systems are all using Samsung SSDs in them, perhaps
> there is some SSD weirdness going on?
Yes, I think so.  SSD is special.

The SSD technology like any flash memories have a limited number of
rewrites, so there are strategies to rewrite as less as possible. In the
case of SSD they are even more aggressive, because they are expensive
.Some of these strategies are implemented in the OS in driver level, but
some in firmware of the device.  So, it is possible that OS can't
completely force the real write.

> I also notice it's not just this
> one file that is affected, but EVERY SINGLE FILE I create with freepascal,
Only with freepascal?

Try it for a file generated from command line

echo "test" > filetest.txt

And do the same checks... edit notepad++... turnoff power etc
> configured a certain way.   I really need these tiny files to survive power
> failures.
Power failures are always a problem for persistent storage. Caching is a
trade-off between speed and security. In the case of SSD there is a
third point: Minimize the real writes on disk. So, if you want security
against power failures, SSD is not the best idea.

By the way. What about a SAI? ;-)


-- 
Saludos

Santiago A.
s...@ciberpiula.net

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Henry Vermaak]

On Tue, Mar 21, 2017 at 06:01:17PM -0700, Ralf Quint wrote:
> On 3/21/2017 11:58 AM, James Richters wrote:
> >I have not tried FlushFileBuffers() yet, so I just tried it now,  I am 
> >getting the same results.
> Well, that is expected, as the issue has nothing to do with your program,
> it's the caching of the operating system that is responsible for the actual
> physical write to the disk.

No, FlushFileBuffers() flushes the OS cache - see the MSDN page.  The
problem here seems to be that the SSD has its own cache (in the FTL).

Henry
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Ralf Quint]

On 3/21/2017 11:58 AM, James Richters wrote:

I have not tried FlushFileBuffers() yet, so I just tried it now,  I am getting 
the same results.
Well, that is expected, as the issue has nothing to do with your 
program, it's the caching of the operating system that is responsible 
for the actual physical write to the disk.
I see something similar with some text files that I use while doing some 
cross-platform work that I am doing recently.


The actual test file is on a Windows 8.1 machine, while most of the 
access to that file is from either a Windows 10 laptop or a macOS Sierra 
MacBook Pro. Occasionally, more pronounced when one of these two 
machines is connected via WiFi instead of wired connection (on the same 
logical subnet though!), when reloading the file from disk, it shows up 
corrupted on the other machine, while it shows perfectly fine on the one 
that did the last write. Force a new write and in 99.9% of all cases, it 
will show up just fine on another reload...


Any chance that the file(s) you are working with are on a shared drive?

Ralf

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Henry Vermaak]

On 21 Mar 2017 18:59, "James Richters" 
wrote:

I have not tried FlushFileBuffers() yet, so I just tried it now,  I am
getting the same results.   I have also tried disabling write caching in
disk management and STILL have the same results.   I am now thinking this
is some caching being done on the SSD that is ignoring FlushFileBuffers()
Probably a 'feature' designed to help extend the life of the SSD, but
making it useless to store data that needs to survive a power failure.


Can you turn off the drive write cache in device manager? You'll have to
double click on the ssd and look around in there.

Henry
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Jonas Maebe]

On 21/03/17 20:48, Marco van de Voort wrote:

He closes the file. That should sync.


Not necessarily. On Unix there is also a difference between closing a 
file and calling fsync() on it. And even then there are sometimes 
additional things you may have to do to survive sudden power loss, such 
as first using fcntl(fd,F_FULLFSYNC) on OS X in case you want to OS to 
also instruct the disk controller to flush its internal caches (if 
supported by the disk).



Jonas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Marco van de Voort]

In our previous episode, Henry Vermaak said:
> > Writeln file
> > Writeln file ...
> > Flush file
> 
> Have you tried using the FlushFileBuffers() Windows API?  Something like
> this:
> 
> FlushFileBuffers(TextRec(AFile).Handle);
> 
> Add "windows" to the uses clause, obviously.

He closes the file. That should sync. Sounds to me he is using some dodgy FS
(network?) driver 
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

I have not tried FlushFileBuffers() yet, so I just tried it now,  I am getting 
the same results.   I have also tried disabling write caching in disk 
management and STILL have the same results.   I am now thinking this is some 
caching being done on the SSD that is ignoring FlushFileBuffers()  Probably a 
'feature' designed to help extend the life of the SSD, but making it useless to 
store data that needs to survive a power failure.

I have tried running my freepascal program on a normal mechanical hard disk and 
flushfilebuffers() seems to be working as expected, Data is getting actually 
written to the disk and my 'power failure' is not affecting the data at all, 
but on the SSD, it's still just a correct length file of 00s

Does anyone know of a way to force critical data to be written to a SSD so it's 
not lost during a power failure?  

James

-Original Message-
From: fpc-pascal [mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of 
Henry Vermaak
Sent: Tuesday, March 21, 2017 11:08 AM
To: FPC-Pascal users discussions <fpc-pascal@lists.freepascal.org>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

On Tue, Mar 21, 2017 at 10:39:23AM -0400, James Richters wrote:
> I am still having this issue.  I've managed to narrow down what is happening
> some.   The problem is my data is not actually being written all the way to
> disk.   I now have a repeatable proven method to reproduce the issue.  Here
> is the sequence.
> 
> In my freepascal console application, I create the file Assign file 
> Rewrite file Writeln file Writeln file Writeln file ...
> Flush file

Have you tried using the FlushFileBuffers() Windows API?  Something like
this:

FlushFileBuffers(TextRec(AFile).Handle);

Add "windows" to the uses clause, obviously.

Henry
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org 
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Jonas Maebe]

On 21/03/17 15:39, James Richters wrote:

If I then power off the test computer (not doing a correct windows shutdown,
this simulates an unattended power failure) when it boots back up, my file
has the correct date and time and the correct length, hovever the contents
of the file is a string of $00 with no other characters, no carriage returns
and no linefeeds, however the number of $00s is EXACTLY how many bytes the
file should have been.  It's like the file was allocated but data not
actually written to disk yet... but I thought Close (file) was supposed to
do that.


No, it does not do that. It flushes everything from the internal program 
buffers to the operating system, but the operating system will perform 
write-back caching. This is not specific to FPC programs, it does this 
almost all writes to a regular hard drive or SSD (Windows does not do it 
for USB sticks, afaik). Not doing so would slow down you system 
significantly.



Jonas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Henry Vermaak]

On Tue, Mar 21, 2017 at 10:39:23AM -0400, James Richters wrote:
> I am still having this issue.  I've managed to narrow down what is happening
> some.   The problem is my data is not actually being written all the way to
> disk.   I now have a repeatable proven method to reproduce the issue.  Here
> is the sequence.
> 
> In my freepascal console application, I create the file
> Assign file
> Rewrite file
> Writeln file 
> Writeln file
> Writeln file ...
> Flush file

Have you tried using the FlushFileBuffers() Windows API?  Something like
this:

FlushFileBuffers(TextRec(AFile).Handle);

Add "windows" to the uses clause, obviously.

Henry
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

I am still having this issue.  I've managed to narrow down what is happening
some.   The problem is my data is not actually being written all the way to
disk.   I now have a repeatable proven method to reproduce the issue.  Here
is the sequence.

In my freepascal console application, I create the file
Assign file
Rewrite file
Writeln file 
Writeln file
Writeln file ...
Flush file
Close file

I then immediately read back the file, and all is good.  I even look at the
file with notepad++ to see that the data is correct and it is fine.

If I then power off the test computer (not doing a correct windows shutdown,
this simulates an unattended power failure) when it boots back up, my file
has the correct date and time and the correct length, hovever the contents
of the file is a string of $00 with no other characters, no carriage returns
and no linefeeds, however the number of $00s is EXACTLY how many bytes the
file should have been.  It's like the file was allocated but data not
actually written to disk yet... but I thought Close (file) was supposed to
do that.I think I am dealing with some kind of disk caching going on
here, but I'm not sure how to force my files to commit all the way to disk.
I should note that these systems are all using Samsung SSDs in them, perhaps
there is some SSD weirdness going on? I also notice it's not just this
one file that is affected, but EVERY SINGLE FILE I create with freepascal,
this is the only one I notice because the other ones are all work files that
end up being re-created every time I run my program, but I checked them all
and they are all exactly the same..  correct lengths but all data is a
string of 00s   

I'm thinking of just disabling all write behind caching in windows and on
the SSD if I can figure it out,  it's not needed anyway anymore because SSDs
are way more than fast enough, and I would rather wait for my data to be
written NOW instead of it NOT getting written later!!The problem with
solving this by disabling write caching is that I can never be sure that
this was done.  So I would like to find a real solution that would force my
files all the way to disk on my own without depending on the system being
configured a certain way.   I really need these tiny files to survive power
failures.

Thanks for any advice on how to do this

James



-Original Message-
From: fpc-pascal-boun...@lists.freepascal.org
[mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of James Richters
Sent: Friday, February 24, 2017 1:53 PM
To: 'FPC-Pascal users discussions' <fpc-pascal@lists.freepascal.org>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

Thanks for the advice everyone.   It is very much appreciated.   I have been
working on a combination of ideas here.  Currently I am doing an MD5sum of a
string created from all my variables concatenated together and writing that
as the second to the last line of the file, I also am writing some fixed
text at the beginning and end of the file as suggested so I can check for
variables being overwritten in memory, great idea!   I am then reading the
file back to verify it has written properly, making a backup, checking that,
and also using ioresults as below. I will probably rename the file as well,
and exclude it from virus checkers.  When I read the file I can verify the
MD5sum and if that test fails, I'll just automatically try the backup, If
that also fails the MD5sum I'll notify the user.  

Thanks again for all the help

James
-Original Message-
From: fpc-pascal-boun...@lists.freepascal.org
[mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of Klaus Hartnegg
Sent: Thursday, February 23, 2017 6:33 AM
To: fpc-pascal@lists.freepascal.org
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

Am 21.02.2017 um 22:12 schrieb James Richters:
>  Assign(BitFile,'BitSave.pax');
{$I-}
>  ReWrite(BitFile);
if ioresult <> 0 then
   writeln ('error opening pax file');
>  WriteLn(BitFile,XADJ:1:8);
>  WriteLn(BitFile,YADJ:1:8);
>  WriteLn(BitFile,ZADJ:1:8);
>  WriteLn(BitFile,WADJ:1:8);
>  WriteLn(BitFile,AADJ:1:8);
>  WriteLn(BitFile,TADJ:1:8);
>  WriteLn(BitFile,VADJ:1:8);
>  WriteLn(BitFile,UADJ:1:8);
>  WriteLn(BitFile,CurrentTool);
>  WriteLn(Bitfile,P_Value[4]);
writeln (bitfile, 'EOF');
if ioresult <> 0 then
   writeln ('error writing pax file');
>  Close(BitFile);
if ioresult <> 0 then
   writeln ('error closing pax file');
{$I+}

This writes something at the end of the file that does not come from a
variable. Next time the error happens, you can check if that is present. 
Then your variables were overwritten in memory. Also it checks for errors.
And I would disable antivirus or at least tell it to not scan this file.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

Thanks for the advice everyone.   It is very much appreciated.   I have been
working on a combination of ideas here.  Currently I am doing an MD5sum of a
string created from all my variables concatenated together and writing that
as the second to the last line of the file, I also am writing some fixed
text at the beginning and end of the file as suggested so I can check for
variables being overwritten in memory, great idea!   I am then reading the
file back to verify it has written properly, making a backup, checking that,
and also using ioresults as below. I will probably rename the file as well,
and exclude it from virus checkers.  When I read the file I can verify the
MD5sum and if that test fails, I'll just automatically try the backup, If
that also fails the MD5sum I'll notify the user.  

Thanks again for all the help

James
-Original Message-
From: fpc-pascal-boun...@lists.freepascal.org
[mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of Klaus Hartnegg
Sent: Thursday, February 23, 2017 6:33 AM
To: fpc-pascal@lists.freepascal.org
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

Am 21.02.2017 um 22:12 schrieb James Richters:
>  Assign(BitFile,'BitSave.pax');
{$I-}
>  ReWrite(BitFile);
if ioresult <> 0 then
   writeln ('error opening pax file');
>  WriteLn(BitFile,XADJ:1:8);
>  WriteLn(BitFile,YADJ:1:8);
>  WriteLn(BitFile,ZADJ:1:8);
>  WriteLn(BitFile,WADJ:1:8);
>  WriteLn(BitFile,AADJ:1:8);
>  WriteLn(BitFile,TADJ:1:8);
>  WriteLn(BitFile,VADJ:1:8);
>  WriteLn(BitFile,UADJ:1:8);
>  WriteLn(BitFile,CurrentTool);
>  WriteLn(Bitfile,P_Value[4]);
writeln (bitfile, 'EOF');
if ioresult <> 0 then
   writeln ('error writing pax file');
>  Close(BitFile);
if ioresult <> 0 then
   writeln ('error closing pax file');
{$I+}

This writes something at the end of the file that does not come from a
variable. Next time the error happens, you can check if that is present. 
Then your variables were overwritten in memory. Also it checks for errors.
And I would disable antivirus or at least tell it to not scan this file.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Klaus Hartnegg]

Am 22.02.2017 um 21:27 schrieb James Richters:

Here is the procedure that reads the file.


Also you could always try to immediately read the file each time it has 
been modified. Append a log entry to some other file, telling that the 
pax file was modified and whether it was still correct. This would tell 
whether it happens when the file is written, or at some other time.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Klaus Hartnegg]

Am 21.02.2017 um 22:12 schrieb James Richters:

 Assign(BitFile,'BitSave.pax');

   {$I-}

 ReWrite(BitFile);

   if ioresult <> 0 then
  writeln ('error opening pax file');

 WriteLn(BitFile,XADJ:1:8);
 WriteLn(BitFile,YADJ:1:8);
 WriteLn(BitFile,ZADJ:1:8);
 WriteLn(BitFile,WADJ:1:8);
 WriteLn(BitFile,AADJ:1:8);
 WriteLn(BitFile,TADJ:1:8);
 WriteLn(BitFile,VADJ:1:8);
 WriteLn(BitFile,UADJ:1:8);
 WriteLn(BitFile,CurrentTool);
 WriteLn(Bitfile,P_Value[4]);

   writeln (bitfile, 'EOF');
   if ioresult <> 0 then
  writeln ('error writing pax file');

 Close(BitFile);

   if ioresult <> 0 then
  writeln ('error closing pax file');
   {$I+}

This writes something at the end of the file that does not come from a 
variable. Next time the error happens, you can check if that is present. 
Then your variables were overwritten in memory. Also it checks for 
errors. And I would disable antivirus or at least tell it to not scan 
this file.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Mark Morgan Lloyd]

On 22/02/17 21:00, James Richters wrote:

I suppose a virus scanner could access it, I didn't think of that.   I guess
I should have said no other program intentionally access the file.   It's
just a way for me to store some variables and get them back, it serves no
other purpose.


A virus scanner might disagree. .pax is a known type which might contain 
an archive hence a whole lot of nasty stuff, and it probably isn't wise 
to use it casually... call it .txt or use the standard .ini format.


--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

I suppose a virus scanner could access it, I didn't think of that.   I guess
I should have said no other program intentionally access the file.   It's
just a way for me to store some variables and get them back, it serves no
other purpose.   

-Original Message-
From: fpc-pascal-boun...@lists.freepascal.org
[mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of Jonas Maebe
Sent: Wednesday, February 22, 2017 3:47 PM
To: FPC-Pascal users discussions <fpc-pascal@lists.freepascal.org>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

On 22/02/17 13:17, James Richters wrote:
> No other programs on the system ever access this file

Not even virus scanners?


Jonas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Jonas Maebe]

On 22/02/17 13:17, James Richters wrote:

No other programs on the system ever access this file


Not even virus scanners?


Jonas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

Here is the procedure that reads the file. It is only read once when the 
program is started, then written to when changes are made.

I don’t have a good reason for having a different file variable name in my read 
procedure than my write procedure, but those are local varaibles to the 
procedures anyway so it shouldn’t matter what I call them.

 

Var

   locfile : text;

If FileExists('Bitsave.pax') Then

Begin

   Writeln('BitSave.pax Found, Loading... ');

   Assign(locfile,'BitSave.pax');

   reset(locfile);

   ReadLn(locfile,XAdj);

   ReadLn(locfile,YAdj);

   ReadLn(locfile,ZAdj);

   ReadLn(locfile,WAdj);

   ReadLn(locfile,AAdj);

   ReadLn(locfile,TAdj);

   ReadLn(locfile,VAdj);

   ReadLn(locfile,UAdj);

   ReadLn(locfile,CurrentTool);

   ReadLn(locfile,P_Value[4]);

   Close(locfile);

End

 

I will do as you suggest and read what I just thought I wrote back to verify 
it.   I may also incorporate some kind of checksum as well and it wouldn’t be a 
bad idea to check how many lines are in the file before I try to read it, just 
to make sure I don’t get read past end of file error.  It seems like a lot just 
to save just 10 numbers, but it’s a pain if those numbers are gone!I will 
also try flush and {$i-}  as well.  

 

Thanks to everyone for your advice and help

 

Jim

 

From: fpc-pascal-boun...@lists.freepascal.org 
[mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of Santiago A.
Sent: Wednesday, February 22, 2017 11:07 AM
To: FPC-Pascal users discussions <fpc-pascal@lists.freepascal.org>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

 

El 22/02/2017 a las 13:17, James Richters escribió:

The error I get is when trying to open the file to read it.  It’s something 
like read past end of file, because the corrupted file is just one long line, 
once I read that, the second read is past the end of file.  I can do checking 
to get the file length and avoid the error, but that doesn’t solve the real 
issue, which is that the data that is supposed to be in the file is just gone.  


Just two lines of research ;-)

A) While you are reading the file you call the procedure that writes it.

B) The is a bug, not in writing the part, but in the reading code: Somewhere 
when you read the file, you don't close it, so the file remains open. Usually 
it's not a big issue, you can open the file several times, and when you close 
the program everything is closed. But some times, when a parameter is changed 
you rewrite it while it is still open and everything gets messed. So this two 
events must happen, the program has executed the part of code that lets the 
file open and next a parameter is changed.




 

The thing is, even if windows forced a restart while my program was running 
this file should have been closed at the time, because if someone was standing 
there editing the variables, they would see the restart notification and close 
the program first, or be able to tell me they had a power failure. 

 

As a temporary measure, I’m just writing out the file twice so at least I have 
a backup, but I don’t see why the backup would not be corrupted by this same 
issue, so that’s probably pretty useless.

Yes, I think that's the way to go.

1) Before writing, read it and if it's ok make a backup of the file
2) Write the file.
3) Read again what you have written to check whether it's ok. 
4) Make a second backup.

That way, when you load parameters you have two backups to recover. And maybe a 
clue of what's going wrong.
Beside, I would add a timestamp inside the file.

In addition, you could use flush and {$I-}





-- 
Saludos
 
Santiago A.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Santiago A.]

El 22/02/2017 a las 13:17, James Richters escribió:

> The error I get is when trying to open the file to read it.  It’s
> something like read past end of file, because the corrupted file is
> just one long line, once I read that, the second read is past the end
> of file.  I can do checking to get the file length and avoid the
> error, but that doesn’t solve the real issue, which is that the data
> that is supposed to be in the file is just gone. 
>

Just two lines of research ;-)

A) While you are reading the file you call the procedure that writes it.

B) The is a bug, not in writing the part, but in the reading code:
Somewhere when you read the file, you don't close it, so the file
remains open. Usually it's not a big issue, you can open the file
several times, and when you close the program everything is closed. But
some times, when a parameter is changed you rewrite it while it is still
open and everything gets messed. So this two events must happen, the
program has executed the part of code that lets the file open and next a
parameter is changed.

>  
>
> The thing is, even if windows forced a restart while my program was
> running this file should have been closed at the time, because if
> someone was standing there editing the variables, they would see the
> restart notification and close the program first, or be able to tell
> me they had a power failure.
>
>  
>
> As a temporary measure, I’m just writing out the file twice so at
> least I have a backup, but I don’t see why the backup would not be
> corrupted by this same issue, so that’s probably pretty useless.
>
Yes, I think that's the way to go.

1) Before writing, read it and if it's ok make a backup of the file
2) Write the file.
3) Read again what you have written to check whether it's ok.
4) Make a second backup.

That way, when you load parameters you have two backups to recover. And
maybe a clue of what's going wrong.
Beside, I would add a timestamp inside the file.

In addition, you could use flush and {$I-}


-- 
Saludos

Santiago A.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Sven Barth]

Am 22.02.2017 13:12 schrieb "Jürgen Hestermann" :
> Also, I am not sure whether "Close" is equivalent to "CloseFile" and
"Assign" is equivalent to "AssignFile".
> I remenber that I had some issues using the old (TP7) notations but I am
don't remember the details.

They are the same. The ones without "File" are from TP times and were
renamed in Delphi to avoid confusing them with the Assign and Close methods
of the component hierarchy.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Peter]

On 21/02/17 21:12, James Richters wrote:


> My understanding was the Close(file); would save the file all the way to
> disk and flush any buffers that were holding it, 

I would try adding a Flush(File) before the Close, even though that
should not really be needed.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[James Richters]

Thanks for the advice on things to look at. 

 

Here is some more information.  First of all, this happens extremely rarely, 
about once a month or two on a pc here and another there, but it’s odd that 
it’s always this one file, and it should not be happening at all.I’m using 
turbo pascal compatibility mode, and it’s a console application, there is no 
multi threading going on, it’s straight inline code.  No other programs on the 
system ever access this file and there is no reason to open it with a text 
editor or anything like that.The error I get is when trying to open the 
file to read it.  It’s something like read past end of file, because the 
corrupted file is just one long line, once I read that, the second read is past 
the end of file.  I can do checking to get the file length and avoid the error, 
but that doesn’t solve the real issue, which is that the data that is supposed 
to be in the file is just gone.  

 

Bitfile is type TEXT like this: 

 

Bitfile: Text;

 

and my variables are all doubles.  

Normally what happens is, everything is fine for a month or two, but then one 
day someone will come in and notice the error on the screen, it was fine the 
night before, the program running overnight, in the morning the program is not 
running,  and when the attempt is made to run it,  The error is found and the 
cause of the error is BitSave.pax has been corrupted and is now a long string 
of [nul]   the computers this runs on are all windows 10 with Samsung solid 
state hard drives.I suspect that the majority of the time, this has 
happened due to winders doing an automatic update and restarting without 
permission as windows 10 annoyingly likes to do, thus crashing my program in 
order to do this restart that is SO important.  I have shut off windows update 
via group policy, but still had the problem since, however I cannot be certain 
there was not a power failure.   The computers are set to re-power up after 
power loss in BIOS and the program is set to run on startup, in fact the 
computers have no other function than to run this one program, they are never 
used for anything else.  The procedure that writes out this file is only ever 
used when one of the variables is changed, and they can only be changed by 
manual input, which is not happening anymore when the issue occurs.   The thing 
is, even if windows forced a restart while my program was running this file 
should have been closed at the time, because if someone was standing there 
editing the variables, they would see the restart notification and close the 
program first, or be able to tell me they had a power failure. 

 

As a temporary measure, I’m just writing out the file twice so at least I have 
a backup, but I don’t see why the backup would not be corrupted by this same 
issue, so that’s probably pretty useless.

Unfortunately this is difficult to troubleshoot because it happens so rarely. I 
have tried to force it to happen by terminating my program with task manager 
and cannot cause the file to be corrupted.   I have not yet tried to duplicate 
the error by shutting down the system while the program is running.  I could 
try that. If it does turn out to be an issue with my program running during 
shutdown,  is there some way I can detect a pending shutdown and exit normally 
before the shutdown happens?   It wouldn’t really solve the cause of the issue, 
but it would possibly help prevent some of the occurrences related to 
shutdowns, but would not help with power failures.

 

I am unable to use Try in turbo pascal compatible mode.  If I use it, I get 
identifier not found ‘Try’   If I attempt to compile in some mode other than 
turbo pascal compatible, I end up with thousands of other errors, so I would 
pretty much have to do a complete re-write.   I can’t see how the close could 
not be run, there is no way out of my procedure without running it since it’s a 
single thread console application.

 

James

 

 

From: fpc-pascal-boun...@lists.freepascal.org 
[mailto:fpc-pascal-boun...@lists.freepascal.org] On Behalf Of Santiago A.
Sent: Wednesday, February 22, 2017 3:48 AM
To: FPC-Pascal users discussions <fpc-pascal@lists.freepascal.org>
Subject: Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

 

El 21/02/2017 a las 22:12, James Richters escribió:

 

Thanks for any advice on this


My first action would be to guarantee that the file is closed with a 
try...finally.

AssignFile(BitFile,'BitSave.pax'); 

ReWrite(BitFile);

try



finally

  CloseFile(BitFile);

end;


Second:

Do you have the file opened with an editor, or things like that, while running 
the program?.

Third:

Check that all variables have valid values and that values are converted to 
string properly to be printable.

Fourth:

Do you use somewhere {$I+} {$I-}? I also had some problems with that with old 
programs.

Fourth:

What's the type BitFile? 
BitFile: TextFile;
BitFile: File;

Maybe using w

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Jürgen Hestermann]

Am 2017-02-21 um 22:12 schrieb James Richters:
> I have a freepascal Windows console application that I recently ported over 
from DOS Turbo Pascal.   I am have a small settings file that I write to disk that 
keeps getting corrupted.  It happens only occasionally, but it’s always this one 
file, no others.Do I need to do something specific to force windows to write 
the file to disk and not have it in some kind of cache or buffer?   I am using the 
following code to write the file:
>Assign(BitFile,'BitSave.pax');
>ReWrite(BitFile);
>WriteLn(BitFile,XADJ:1:8);
>WriteLn(BitFile,YADJ:1:8);
>WriteLn(BitFile,ZADJ:1:8);
>WriteLn(BitFile,WADJ:1:8);
>WriteLn(BitFile,AADJ:1:8);
>WriteLn(BitFile,TADJ:1:8);
>WriteLn(BitFile,VADJ:1:8);
>WriteLn(BitFile,UADJ:1:8);
>WriteLn(BitFile,CurrentTool);
>WriteLn(Bitfile,P_Value[4]);
>Close(BitFile);
> The above only happens when I make a change to one of the settings, and the 
system is never having any issue at that time, yet upon occasion I will have an 
error opening the file and when I inspect it, it’s just a long string of [nul].   
I am not leaving the file open, I’m assigning it, writing to it, then closing it, 
so there should be no opportunity for data to be lost.  I am having the same issue 
on about 8 different computers, most of which are running solid state hard drives. 
It’s always just this one file that is affected, nothing else.  No other process 
ever uses this file, so it must be my freepascal program that is leaving the file 
in a vulnerable state.
> My understanding was the Close(file); would save the file all the way to disk 
and flush any buffers that were holding it, but I’m wondering if that was only 
true for DOS and there is some other windows cache or buffer that also needs to be 
instructed to flush to truly save the file all the way to disk and close it.

How is "Bitfile" declared?
According to http://wiki.freepascal.org/Text
it should be either

var BitFile : TextFile;

or

var BitFile : Text;

Also, I am not sure whether "Close" is equivalent to "CloseFile" and  "Assign" is 
equivalent to "AssignFile".
I remenber that I had some issues using the old (TP7) notations but I am don't 
remember the details.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Santiago A.]

El 21/02/2017 a las 22:12, James Richters escribió:
>
>  
>
> Thanks for any advice on this
>

My first action would be to guarantee that the file is closed with a
try...finally.

AssignFile(BitFile,'BitSave.pax');

ReWrite(BitFile);

try



finally

  CloseFile(BitFile);

end;


Second:

Do you have the file opened with an editor, or things like that, while
running the program?.

Third:

Check that all variables have valid values and that values are converted
to string properly to be printable.

Fourth:

Do you use somewhere {$I+} {$I-}? I also had some problems with that
with old programs.

Fourth:

What's the type BitFile?
BitFile: TextFile;
BitFile: File;

Maybe using writeln with non-textfile files may cause problems.

-- 
Saludos

Santiago A.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Jonas Maebe]

James Richters wrote:
> I have a freepascal Windows console application that I recently ported
> over from DOS Turbo Pascal.   I am have a small settings file that I
> write to disk that keeps getting corrupted.  It happens only
> occasionally, but it’s always this one file, no others.Do I need to
> do something specific to force windows to write the file to disk and not
> have it in some kind of cache or buffer?   I am using the following code
> to write the file:

This reminds me of a bug report from a couple of years back, where
someone else had a somewhat similar weird issue under Windows:
http://bugs.freepascal.org/view.php?id=27670

Do you happen to use Flexraid as well?


Jonas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] FreePascal Windows - Force files to write to disk

[Tomas Hajny]

On Tue, February 21, 2017 22:12, James Richters wrote:


Hello James,

> I have a freepascal Windows console application that I recently ported
> over from DOS Turbo Pascal.   I am have a small settings file that I write
> to disk that keeps getting corrupted.  It happens only occasionally, but
> it’s always this one file, no others.Do I need to do something
> specific to force windows to write the file to disk and not have it in
> some kind of cache or buffer?   I am using the following code to write the
> file:
>
>Assign(BitFile,'BitSave.pax');
>ReWrite(BitFile);
>WriteLn(BitFile,XADJ:1:8);
>WriteLn(BitFile,YADJ:1:8);
>WriteLn(BitFile,ZADJ:1:8);
>WriteLn(BitFile,WADJ:1:8);
>WriteLn(BitFile,AADJ:1:8);
>WriteLn(BitFile,TADJ:1:8);
>WriteLn(BitFile,VADJ:1:8);
>WriteLn(BitFile,UADJ:1:8);
>WriteLn(BitFile,CurrentTool);
>WriteLn(Bitfile,P_Value[4]);
>Close(BitFile);
>
> The above only happens when I make a change to one of the settings, and
> the system is never having any issue at that time, yet upon occasion I
> will have an error opening the file and when I inspect it, it’s just a
> long string of [nul].   I am not leaving the file open, I’m assigning it,
> writing to it, then closing it, so there should be no opportunity for data
> to be lost.  I am having the same issue on about 8 different computers,
> most of which are running solid state hard drives. It’s always just this
> one file that is affected, nothing else.  No other process ever uses this
> file, so it must be my freepascal program that is leaving the file in a
> vulnerable state.

It's difficult to find out possible reasons without fully compilable
source. If I should guess, I would check the types of the variables used
for writing and also make sure that the content of those variables isn't
overwritten in memory (i.e. even before they're written to the file). In
particular, if your variables are strings and you compiled the source
(originally written for Turbo Pascal, i.e. using shortstrings) e.g. with
{$H+}, or in a compiler mode using ansistrings or
widestrings/unicodestrings by default _and_ the original source modified
the strings in a non-standard way, you might overwrite the content of
those variables.

BTW, what exactly is the error you get when opening the file?


> My understanding was the Close(file); would save the file all the way to
> disk and flush any buffers that were holding it, but I’m wondering if that
> was only true for DOS and there is some other windows cache or buffer that
> also needs to be instructed to flush to truly save the file all the way to
> disk and close it.

Your understanding is correct, nothing else should be necessary.

Tomas


___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal