Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message 53021f2a53.matt...@sinenomine.freeserve.co.uk
on 10 Mar 2013 Matthew Phillips wrote:
Still not having looked at the code, my impression is that the routine in
question is returning a single 32-bit word to *its* caller which attempts,
in one word, to keep the position in the directory and the subdirectory.
See the start of the thread where Alan quoted from the code:
/* After preaching the passion, now the hack : when we're going to do
the reverse suffix swapping, we will have up to two outstanding OS_GBPB
sessions : one in the main dir and another in the suffix dir.
How we're going to give an 'off_t' result in telldir describing these
two offsets in OS_GBPB ? Well, we are going to rely on unity monotonic
increase of the offsets and this up to GBPB_MAX_ENUM. If these
conditions are not fullfilled, we stop the enumeration.
Another solution is to return table index numbers which, when presented
to seekdir, will get looked up in a table giving the two internal dir
offsets. */
How exactly it relies on unity monotonic increasing, I don't know. Perhaps
I'd better download unix/dirent.c to see.
OK, take a look at the code here:
http://www.riscos.info/websvn/filedetails.php?repname=gccsdkpath=%2Ftrunk%2Fgcc4%2Frecipe%2Ffiles%2Fgcc%2Flibunixlib%2Funix%2Fdirent.c
The problems come from the way the standard Unix functions telldir and
seekdir are implemented. These rely on the use of a long int to record the
position we are up to in traversing the directory. Unixlib is pretending
that the files contained in subdirectories named c, h, o etc. are
actually files in the main directory with suffixes instead, so if you had a
directory containing (RISC OS style)
a_file
c
c.source
c.source2
h
h.header
it would return filenames (Unix style) of
a_file
source.c
source2.c
header.h
The telldir function has to return a long int which says where we currently
are in reading the directory and these special subdirectories. The seekdir
function has to take that value and turn it back into a GBPB position in the
directory and in the corresponding subdirectory if we are in the middle of
one.
I think the difficulty mainly comes from the OS_GBPB call in readdir_r
reading multiple file information at once. This means, if it encounters a
suitable subdirectory part way through what it has read, that it is
impossible for it to know what value of R4 would take it straight back to
that entry without relying on R4 increasing by one each time. It needs to be
able to recover the subdirectory name because otherwise it doesn't know
which subdirectory it's getting the files from.
I've not scoured the source thoroughly, but the only bit of arithmetic on R4
values I can see is in telldir:
long int
telldir (DIR *stream)
{
if (!__validdir (stream))
return -1;
if (stream-suffix stream-suffix-dd_off != GBPB_END_ENUM)
return (stream-dd_off - 1) + (stream-suffix-dd_off 16);
return stream-dd_off + (stream-dd_suf_off 16);
}
If OS_GBPB 10 were used to read one file at a time, it would be possible to
alter the code to store the proper R4 values and avoid this hack. But this
might be rather less efficient, speed-wise. I am not sure whether things
would be noticeably slower.
I would need to read a bit more about how these Unix functions are supposed
to behave before being able to suggest a solution.
The code also relies on R4 return values from OS_GBPB 10 being in the range
-1 to 65535, as it packs two of them into a single word. That should not be
a problem for FileCore file systems or for FAT32FS but is not in general
true.
The implementation does check that R4 values are going up by one per
directory entry, and if it finds this not to be the case, enumeration is
stopped to be on the safe side. Removing this check, which can be found in
this block of code
/* Check if we have a monotonic unit increase OS_GBPB
offset, if not, we don't do anything. */
if (stream-gbpb_off + regs[3] != regs[4])
{
stream-gbpb_off = stream-dd_off = GBPB_END_ENUM;
return 0;
}
could well actually cure the problem as far as FAT32FS goes, because that
filing system copes with reversing R4 values by a single step, but in general
it is a useful safety check which needs to be kept while the functions are
implemented in this manner.
--
Matthew Phillips
Durham
___
GCCSDK mailing list gcc@gccsdk.riscos.info
Bugzilla: http://www.riscos.info/bugzilla/index.cgi
List Info: http://www.riscos.info/mailman/listinfo/gcc
Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
Hi Matthew, The telldir function has to return a long int which says where we currently are in reading the directory and these special subdirectories. The seekdir function has to take that value and turn it back into a GBPB position in the directory and in the corresponding subdirectory if we are in the middle of one. Given the DIR that these functions use is opaque, it seems to me it could store an array of every unique (dir, sub) RISC OS position ever returned by telldir(), with telldir returning the index, and seekdir() plucks the tuple from that index. I don't see anything in the POSIX man page that suggests telldir results have an ordering, merely they can be passed back to seekdir. No clean-up of the list is possible until the whole DIR is scrapped with closedir(), but it's unlikely to be a problem in practice. Cheers, Ralph. ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message 8106242a53.matt...@sinenomine.freeserve.co.uk
on 10 Mar 2013 Matthew Phillips wrote:
The implementation does check that R4 values are going up by one per
directory entry, and if it finds this not to be the case, enumeration is
stopped to be on the safe side. Removing this check, which can be found in
this block of code
/* Check if we have a monotonic unit increase OS_GBPB
offset, if not, we don't do anything. */
if (stream-gbpb_off + regs[3] != regs[4])
{
stream-gbpb_off = stream-dd_off = GBPB_END_ENUM;
return 0;
}
could well actually cure the problem as far as FAT32FS goes, because that
filing system copes with reversing R4 values by a single step, but in
general it is a useful safety check which needs to be kept while the
functions are implemented in this manner.
Ignore that: there are plenty of cases where the current implementation would
fail on FAT32FS if the above check were removed. You only have to have had a
few things deleted from the directory for it to cause trouble, depending on
what the caller is trying to do with telldir/seekdir of course.
--
Matthew Phillips
Durham
___
GCCSDK mailing list gcc@gccsdk.riscos.info
Bugzilla: http://www.riscos.info/bugzilla/index.cgi
List Info: http://www.riscos.info/mailman/listinfo/gcc
Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
On 10 Mar 2013 Ralph Corderoy wrote: Hi Matthew, The telldir function has to return a long int which says where we currently are in reading the directory and these special subdirectories. The seekdir function has to take that value and turn it back into a GBPB position in the directory and in the corresponding subdirectory if we are in the middle of one. Given the DIR that these functions use is opaque, it seems to me it could store an array of every unique (dir, sub) RISC OS position ever returned by telldir(), with telldir returning the index, and seekdir() plucks the tuple from that index. I don't see anything in the POSIX man page that suggests telldir results have an ordering, merely they can be passed back to seekdir. No clean-up of the list is possible until the whole DIR is scrapped with closedir(), but it's unlikely to be a problem in practice. That would be a nice safe method. You still have the issue of working out what the R4 values ought to be for resuming from that position, and for that to work the code will have to be modified to read one directory entry at a time from OS_GBPB 10, rather than several as it is doing at present. There is no getting away from that. -- Matthew Phillips Durham ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message 381fea2853.mar...@blueyonder.co.uk on 7 Mar 2013 Martin Bazley wrote: The following bytes were arranged on 7 Mar 2013 by alan buckley : On Thu, 7 Mar 2013 11:23:12 Martin Bazley wrote: The following bytes were arranged on 6 Mar 2013 by Matthew Phillips : I have been wondering about this, with the caveat that I also have not looked at the UnixLib code. This is for wildcard scanning, right? Why would it need to go backwards in those circumstances? I've had a brief look at the code, and it appears to scan through the directory with a subscan of directories used for suffix swapping. So I don't think it does go backwards, just into subdirectories and then continues on when the subdirectory is finished. I don't understand. The value returned in R4 doesn't expire. You can perfectly well take a backup on the stack, recurse into the directory and continue from where you left off afterwards. There *is* a known problem with using previous values of R4 when deleting or otherwise changing the number of files in a directory at the same time as scanning it, but I sincerely hope it isn't doing that! Still not having looked at the code, my impression is that the routine in question is returning a single 32-bit word to *its* caller which attempts, in one word, to keep the position in the directory and the subdirectory. See the start of the thread where Alan quoted from the code: /* After preaching the passion, now the hack : when we're going to do the reverse suffix swapping, we will have up to two outstanding OS_GBPB sessions : one in the main dir and another in the suffix dir. How we're going to give an 'off_t' result in telldir describing these two offsets in OS_GBPB ? Well, we are going to rely on unity monotonic increase of the offsets and this up to GBPB_MAX_ENUM. If these conditions are not fullfilled, we stop the enumeration. Another solution is to return table index numbers which, when presented to seekdir, will get looked up in a table giving the two internal dir offsets. */ How exactly it relies on unity monotonic increasing, I don't know. Perhaps I'd better download unix/dirent.c to see. -- Matthew Phillips Durham ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
The following bytes were arranged on 7 Mar 2013 by alan buckley : On Thu, 7 Mar 2013 11:23:12 Martin Bazley wrote: The following bytes were arranged on 6 Mar 2013 by Matthew Phillips : I have been wondering about this, with the caveat that I also have not looked at the UnixLib code. This is for wildcard scanning, right? Why would it need to go backwards in those circumstances? I've had a brief look at the code, and it appears to scan through the directory with a subscan of directories used for suffix swapping. So I don't think it does go backwards, just into subdirectories and then continues on when the subdirectory is finished. I don't understand. The value returned in R4 doesn't expire. You can perfectly well take a backup on the stack, recurse into the directory and continue from where you left off afterwards. There *is* a known problem with using previous values of R4 when deleting or otherwise changing the number of files in a directory at the same time as scanning it, but I sincerely hope it isn't doing that! -- __^__ / _ _ \ It is written that Geeks shall inherit the Earth. ( ( |_| ) ) \_ _/ === Martin Bazley == ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message 116d532753...@hobbes.bass-software.com on 4 Mar 2013 John Tytgat wrote: In message dub120-ds7e48892a10433c7ca0983f0...@phx.gbl Alan Buckley alan_...@hotmail.com wrote: From comments when I mentioned this bug on the ROOL forum it made me think Fat32FS does not increment the value as expected by the hack. It's very much possible that this wrong assumption is the cause of your problem. However for enumeration purposes we're using 1 KByte buffer so you would get a correct result for a 'modest' number of files in your cc subdirectory. Is there anybody who could look at to confirm my findings (and possibly fix) this? I'm not using FatFS32 myself but this can be very easily tested in BASIC, just write a dir enum based on OS_GBPB (fetching at most 1 element) and see what kind of offsets this generates. From a discussion about this not long ago on comp.sys.acorn.something Jeff confirmed that Fat32FS may go up in steps of more than one. There was a failure case with a piece of software which attempted to go backwards a single step occasionally which he had allowed for in some cunning way at his end. There are other filing systems out there which do not go up in steps of one. CPMFS is another example, but I doubt anyone apart from me and J.G. Harston is using that, and we certainly wouldn't be compiling C off CP/M discs. CPMFS and Fat32FS have the same need because FAT shares some of the CP/M heritage. Basically, in a FAT directory each file or subdirectory entry takes 32 bytes. When a file is deleted, the first byte of the entry is marked to show that: the gap is not closed up. Also some entries are used not for files but for continuations of earlier entries to store long file names. The most straightforward thing for Fat32FS to return is thus an index into the list of directory entries, and while it will always ascend, it may skip some values on the way. The alternative, of ignoring deleted files in the numbering, would fail if a file earlier in the directory were deleted between calls to OS_GBPB. I think Jeff dealt with the software which subtracted one by spotting if the index pointed to an entry which was a deleted file or a filename continuation block and scanning backwards to find the previous genuine entry. I have a feeling it was FilerAction which was the culprit actually! Certainly the FilerAction module came up in the discussion. I have not looked at the unixlib code, but if it's just a matter of using a single 32-bit word (an off_t ?) to keep track of enumerating two directories independently via OS_GBPB there must be a solution which does not involve assuming an increase of one per file, even without going as far as scanning the whole directory at the outset. -- Matthew Phillips Durham ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message b7206b2753...@hobbes.bass-software.com
John Tytgat john.tyt...@aaug.net wrote:
Which recipe file for gcc do you mean ?
So there is a minimum of three changes to do.
Three ? I'm mentioning the UnixEnv$*$sfix @ !Run, you're referring to
a recipe file for gcc, but what's the third change ?
(1)
gcc4/recipe/files/gcc/gcc/config/arm/riscos-gcc.c
const char *
riscos_convert_filename (void *obstack, const char *name,
int do_exe, int do_obj ATTRIBUTE_UNUSED)
{
char tmp[1024];
/* if (do_exe)*/
return name;
/*if (!__unixify_std (name, tmp, sizeof (tmp), 0))
return name;
return obstack_copy0 ((struct obstack *) obstack, tmp, strlen (tmp));*/
}
(2)
gcc4/recipe/create-gcckit
Comment out 85-94 114 and 189
(3)
!GCC.!Run
Set UnixEnv$gcc$sfix
Comment out
|f:for:F:fpp:cc:cxx:cpp:c++:C:i:ii:rpo:c:m:h:hh:s:S:xrb:xrs:l:o:y:tcc:cmhg:adb:ads:ali
Actually, as long as you can change the !GCC library files directories
yourself, The only thing the OP would need would be the changed gcc
binary.
Note, I haven't extensively tested !GCC this way but It hasn't faltered
so far.
I have a pattern saved so !Rename can revert norcroft style source
directories.
The pattern match doesn't accept ^ as an up directory, so I use a system
variable for the same in it's place and it works ok.
Ron M.
___
GCCSDK mailing list gcc@gccsdk.riscos.info
Bugzilla: http://www.riscos.info/bugzilla/index.cgi
List Info: http://www.riscos.info/mailman/listinfo/gcc
Main Page: http://www.riscos.info/index.php/GCCSDK
[gccsdk] unixlib directory iteration doesn't work on Fat32FS
I’ve been trying to build a project stored on a Fat32FS USB pen drive and found make wasn’t locating any of the files using the wildcard function. The wildcard was looking for *.cc files so I believe it is to do with suffix swapping. After further investigation I think I’ve tracked the problem down to the unixlib directory iteration in the unix/dirent.c file. I believe the following comments from the top of the file are relevant. /* It is a common mistake to assume that the offsets returned from OS_GBPB is monotonic increasing with unity steps from the start offset 0 onwards. Accoring to the PRMs, the only thing you may be sure of is that offset 0 starts the GBPB enumeration and offset -1 returned indicates the end of the enumeration. ... /* After preaching the passion, now the hack : when we're going to do the reverse suffix swapping, we will have up to two outstanding OS_GBPB sessions : one in the main dir and another in the suffix dir. How we're going to give an 'off_t' result in telldir describing these two offsets in OS_GBPB ? Well, we are going to rely on unity monotonic increase of the offsets and this up to GBPB_MAX_ENUM. If these conditions are not fullfilled, we stop the enumeration. Another solution is to return table index numbers which, when presented to seekdir, will get looked up in a table giving the two internal dir offsets. */ From comments when I mentioned this bug on the ROOL forum it made me think Fat32FS does not increment the value as expected by the hack. Is there anybody who could look at to confirm my findings (and possibly fix) this? Regards, Alan___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message dub120-ds7e48892a10433c7ca0983f0...@phx.gbl Alan Buckley alan_...@hotmail.com wrote: snip From comments when I mentioned this bug on the ROOL forum it made me think Fat32FS does not increment the value as expected by the hack. Is there anybody who could look at to confirm my findings (and possibly fix) this? Just my own approach, but I have built a !GCC that doesn't use suffix swapping and my recent build seems to be working OK. I also seem to have got away with removing some suffixing routines from the recipe riscosify.c and unixify.c without side effects this time. It would require you to have your source directories organised in the plain way of course. (file/c file/h etc) This might show that it is the suffixing causing your problem, and you might even see some advantages of using standard file naming. I recently had a completely hands off experience where I downloaded some source, set the CSD and 'make' did the job straight off. Anyway, let me know, and I could email you my !GCC. We could use my multiparting tar app to break it down to email sized chunks. (-: Ron M. ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message dub120-ds7e48892a10433c7ca0983f0...@phx.gbl Alan Buckley alan_...@hotmail.com wrote: From comments when I mentioned this bug on the ROOL forum it made me think Fat32FS does not increment the value as expected by the hack. It's very much possible that this wrong assumption is the cause of your problem. However for enumeration purposes we're using 1 KByte buffer so you would get a correct result for a 'modest' number of files in your cc subdirectory. Is there anybody who could look at to confirm my findings (and possibly fix) this? I'm not using FatFS32 myself but this can be very easily tested in BASIC, just write a dir enum based on OS_GBPB (fetching at most 1 element) and see what kind of offsets this generates. If that's the issue, I guess the dirent() routines need to be rewritten from scratch (together with some small testqueue code) by enumerating the full RISC OS directory contents at opendir() time. That's the only way to get a bulletproof implementation. John. -- John Tytgat, in his comfy chair at home john.tyt...@aaug.net ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message f67e522753.b...@ron1954.woosh.co.nz Ron b...@woosh.co.nz wrote: Just my own approach, but I have built a !GCC that doesn't use suffix swapping and my recent build seems to be working OK. I also seem to have got away with removing some suffixing routines from the recipe riscosify.c and unixify.c without side effects this time. Isn't that way too much effort ? Doesn't this work out-of-box with a regular !GCC release where you change UnixEnv$gcc$sfix in !GCC.!Run into an empty string ? Afterall, that's the whole point of UnixEnv$*$sfix system variables. John. -- John Tytgat, in his comfy chair at home john.tyt...@aaug.net ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message b1ea532753...@hobbes.bass-software.com John Tytgat john.tyt...@aaug.net wrote: In message f67e522753.b...@ron1954.woosh.co.nz Ron b...@woosh.co.nz wrote: Just my own approach, but I have built a !GCC that doesn't use suffix swapping and my recent build seems to be working OK. I also seem to have got away with removing some suffixing routines from the recipe riscosify.c and unixify.c without side effects this time. Isn't that way too much effort ? Doesn't this work out-of-box with a regular !GCC release where you change UnixEnv$gcc$sfix in !GCC.!Run into an empty string ? Afterall, that's the whole point of UnixEnv$*$sfix system variables. John. Regarding !GCC, the recipe file for gcc has a sfix routine that I had to remove also, /only then/ UnixEnv$gcc$sfix in the !Run file effects the remaining binaries and all is well. So there is a minimum of three changes to do. Modifying riscosify.c and unixify.c is extra, but I'm hoping it will make them simpler for me to follow for any further changes I might try. I still have the problem of textfile,xyz (where xyz is a recogniseable extension) causing tar to skip it. While it doesn't make much sense to have a file like that on the RISC OS filesystem (It should have been already converted by program foo) I'm aiming to be able to copy 'all' files verbatim and this is one that catches. Setting UnixEnv$prog$sfix to is enough to turn off the sfix functioning. For programs like tar that are acessing many files, I thought it might also be be a speed advantage to remove the sfix /checking/ functions, especially as I'm never using them these days. Ron M. ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
Re: [gccsdk] unixlib directory iteration doesn't work on Fat32FS
In message e0d4592753.b...@ron1954.woosh.co.nz Ron b...@woosh.co.nz wrote: In message b1ea532753...@hobbes.bass-software.com John Tytgat john.tyt...@aaug.net wrote: In message f67e522753.b...@ron1954.woosh.co.nz Ron b...@woosh.co.nz wrote: Just my own approach, but I have built a !GCC that doesn't use suffix swapping and my recent build seems to be working OK. I also seem to have got away with removing some suffixing routines from the recipe riscosify.c and unixify.c without side effects this time. Isn't that way too much effort ? Doesn't this work out-of-box with a regular !GCC release where you change UnixEnv$gcc$sfix in !GCC.!Run into an empty string ? Afterall, that's the whole point of UnixEnv$*$sfix system variables. John. Regarding !GCC, the recipe file for gcc has a sfix routine that I had to remove also, /only then/ UnixEnv$gcc$sfix in the !Run file effects the remaining binaries and all is well. Which recipe file for gcc do you mean ? So there is a minimum of three changes to do. Three ? I'm mentioning the UnixEnv$*$sfix @ !Run, you're referring to a recipfe file for gcc, but what's the third change ? Modifying riscosify.c and unixify.c is extra, but I'm hoping it will make them simpler for me to follow for any further changes I might try. I still have the problem of textfile,xyz (where xyz is a recogniseable extension) causing tar to skip it. While it doesn't make much sense to have a file like that on the RISC OS filesystem (It should have been already converted by program foo) I'm aiming to be able to copy 'all' files verbatim and this is one that catches. If you want to make that bullet proof Unixify should somehow escape the , in the RISC OS file name and you have to teach riscosify to do the unescaping. BTW, are RISC OS files / and // properly tar'd and can they be untar'd on Unix ? :-) Setting UnixEnv$prog$sfix to is enough to turn off the sfix functioning. For programs like tar that are acessing many files, I thought it might also be be a speed advantage to remove the sfix /checking/ functions, especially as I'm never using them these days. I would be surprised this having a meaningful advantage. John. -- John Tytgat, in his comfy chair at home john.tyt...@aaug.net ___ GCCSDK mailing list gcc@gccsdk.riscos.info Bugzilla: http://www.riscos.info/bugzilla/index.cgi List Info: http://www.riscos.info/mailman/listinfo/gcc Main Page: http://www.riscos.info/index.php/GCCSDK
