Re: Has Anyone Got A Directory \\\"Walker\\\" Available

[Brian Milby via use-livecode]

I think we may be in the territory of pre-mature optimization. There is
going to be a bit more overhead creating a copy-on-write object than a
pass-by-reference object, but the real gains (over the old way) are going
to be seen with objects that are “large”. If you run a profiler and need to
improve a section of code, then this would be something to have in the
toolbox. It probably isn’t a good practice to use everywhere since then you
have to be careful about changing the value of a parameter.

It would be interesting to see that same piece of code run before the
change. When did copy on write get introduced?
On Mon, May 7, 2018 at 2:20 PM Richard Gaskin via use-livecode <
use-livecode@lists.runrev.com> wrote:

> Mark Wieder wrote:
>
>  > On 05/06/2018 02:42 PM, Richard Gaskin via use-livecode wrote:
>  >
>  >> Did copy-on-write get changed in v9, or is the scope of its effects
>  >> just more limited than I had understood it to be?
>  >
>  > I'm still at the point of not trusting copy-on-write yet, but I think
>  > you're misinterpreting your results. If I'm understanding the way in
>  > which LC has implemented copy-on-write, then whether or not you use
>  > references in parameters, the "return p-1" code will have to make a
>  > copy on the stack to return a value - you can't just return a
>  > reference. So all you're changing by removing the "@" is one level of
>  > copying.
>
> That makes sense.  Thanks.
>
> --
>   Richard Gaskin
>   Fourth World Systems
>   Software Design and Development for the Desktop, Mobile, and the Web
>   
>   ambassa...@fourthworld.comhttp://www.FourthWorld.com
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \\\"Walker\\\" Available

[Richard Gaskin via use-livecode]

Mark Wieder wrote:

> On 05/06/2018 02:42 PM, Richard Gaskin via use-livecode wrote:
>
>> Did copy-on-write get changed in v9, or is the scope of its effects
>> just more limited than I had understood it to be?
>
> I'm still at the point of not trusting copy-on-write yet, but I think
> you're misinterpreting your results. If I'm understanding the way in
> which LC has implemented copy-on-write, then whether or not you use
> references in parameters, the "return p-1" code will have to make a
> copy on the stack to return a value - you can't just return a
> reference. So all you're changing by removing the "@" is one level of
> copying.

That makes sense.  Thanks.

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \\\"Walker\\\" Available

[Mark Wieder via use-livecode]

On 05/06/2018 02:42 PM, Richard Gaskin via use-livecode wrote:

Did copy-on-write get changed in v9, or is the scope of its effects just 
more limited than I had understood it to be?


I'm still at the point of not trusting copy-on-write yet, but I think 
you're misinterpreting your results. If I'm understanding the way in 
which LC has implemented copy-on-write, then whether or not you use 
references in parameters, the "return p-1" code will have to make a copy 
on the stack to return a value - you can't just return a reference. So 
all you're changing by removing the "@" is one level of copying.


Of course, I may well be not quite understanding what's been 
implemented, in which case I welcome any correction.


--
 Mark Wieder
 ahsoftw...@gmail.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \"Walker\" Available

[Alex Tweedly via use-livecode]

I doubt if it would work out faster. A quick test of

 time ls -lR > null

gives 166 ms vs. the 179 I get from the non-recursive tree walker - so 
by the time you gather that output, and re-format back to something more 
usable, I think you'd probably come out the same or slower.


Alex.


On 06/05/2018 21:46, Malte Pfaff-Brill via use-livecode wrote:

I wonder if shelling out to DIR on Windows and LS on Linux/Mac wouldn’t be the 
fastest option…

Cheers,

Malte


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory "Walker" Available

[Alex Tweedly via use-livecode]

On 06/05/2018 20:12, Richard Gaskin via use-livecode wrote:

Ah, that makes sense.  It's not so much the recursion per se, but the 
more general advantage of inline processing vs handler calls.



Exactly.
In the test case below you know how many levels deep you're going, 
which seems a good fit for inline loops. But how do we handle cases 
like directory traversal, where we don't know in advance how deep 
we'll need to go?


In this test case I do know how many levels - but the code makes no use 
of that knowledge, it simply exits when we get down to 0.
For an example of how to handle directory recursion, see Ben's earlier 
post in this thread. Or I've included my current version below - I 
return a 2-level array of folder / file / data ... rather than a 
file-per-line.
It's at least comforting to see that while the overhead of handler 
calls is significant, it's mostly in relative comparison: If I read 
that correctly, there are 1000 iterations of 100 operations, for a 
total of 100k operations.  If my coffee is working, that means an 
overhead of just 0.00208 ms per operation when called in a separate 
handler, no? (thinking 273ms total for separate handler - 65ms for 
inline, per 100k operations).


Yep - though it gets higher with more parameters. My recursive 'walker' 
needed 4 parameters, so the overhead was higher.
Note that if I was writing a recursive tree-walker now, I'd make it a 
private function, so I could keep the parameter checks only at the top 
(i.e. public) level, and recurse without repeating them.
I might even move most of the parameters into script-local variables, 
since I know the code is interrupt-safe.
With directory traversal, I'd guess the overhead of physically 
touching each inode would outweigh that manyfold.


Yeah - though it's particularly hard to test adequately. When I was 
trying to benchmark it, I found that to get consistent results, I had to 
run the test 2 or 3 times and ignore the first, highly variable, run. 
Which basically means that I was seeing results with the disk-cache 
fully engaged, and therefore likely to underestimate the importance of 
the disk operations in any real context.
Still useful to keep in mind: when inline code can do the job clearly, 
it will be more efficient.

Below is my current non-recursive walker.
NB - it goes to some trouble to allow you to pass in a relative path 
name, and maintain that in the result. That is the opposite of what is 
commonly done (i.e. using absolute path/file names); I did this to make 
it easier to compare multiple trees. If you want the traditional result, 
simply do

   put the defaultfolder into temp -- gives the absolute path
   getArrayOfFiles temp, tMyArray
but if you want "my" twist, you'd do
   getArrayOfFiles ".", tMyArray

-- Alex.
# Produce an array of folders + files with the "full" relative paths
command getArrayOfFiles pFolder, @pA, pIgnoreDotFolders, pIgnoreDotFiles
   local tFiles, tFolders
   local tThisFolder
   if pIgnoreDotFolders is empty then
  put TRUE into pIgnoreDotFolders
   end if
   if pIgnoreDotFiles is empty then
  put TRUE into pIgnoreDotFiles
   end if

   put the defaultfolder into tThisFolder

   local tAbsFolder, tFoldersLeft, tSub
   set the defaultfolder to pFolder
   put the defaultfolder into tAbsFolder   -- changes relative into 
absolute


   put CR into tFoldersLeft
   repeat until tFoldersLeft is empty
  put line 1 of tFoldersLeft into tSub
  delete line 1 of tFoldersLeft
  set the defaultFolder to (tAbsFolder & tSub)
  if the result is not empty then
 -- skip or report as needed
 next repeat
  end if
  try
 put folders() into tFolders
  end try
  if pIgnoreDotFolders then
 filter tFolders without ".*"
  else
 filter tFolders without "."
 filter tFolders without ".."
  end if
  repeat for each line L in tFolders
 put tSub & "/" & L & CR after tFoldersLeft
  end repeat

  try
 put the detailed files into tFiles
  end try

  if pIgnoreDotFiles then
 filter tFiles without ".*"
  end if
  repeat for each line L in tFiles
 put item 2 to -1 of L into pA[pFolder & tSub][item 1 of L]
  end repeat
   end repeat
end getArrayOfFiles


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \"Walker\" Available

[Stephen Barncard via use-livecode]

there's actually a directory walker search one can use in a shell command
 on mac that is quite speedy...

let me report back.

sqb

--
Stephen Barncard - Sebastopol Ca. USA -
mixstream.org

On Sun, May 6, 2018 at 2:29 PM, Richard Gaskin via use-livecode <
use-livecode@lists.runrev.com> wrote:

> Malte Pfaff-Brill wrote:
>
> > I wonder if shelling out to DIR on Windows and LS on Linux/Mac
> > wouldn’t be the fastest option…
>
> Maybe, but there's overhead in setting up the shell session.
>
> Even if it's measurably faster, I would be surprised if it were noticeably
> faster.
>
> Anything that depends on touching the disk with each call is likely to be
> far more I/O-bound than CPU-bound.
>
> --
>  Richard Gaskin
>  Fourth World Systems
>  Software Design and Development for the Desktop, Mobile, and the Web
>  
>  ambassa...@fourthworld.comhttp://www.FourthWorld.com
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \\\"Walker\\\" Available

[Richard Gaskin via use-livecode]

Bernd wrote:
> Richard wrote:
>> Which version of LC did you test with?
>> I was under the impression that since LC switched to copy-on-write
>> for all >arguments we should no longer need to use "@" for
>> performance, only for for logic.
>
>
> I tested using LC 9 GM, what kind of results do you get?

Well, so much for copy-on-write.

As written:
31
84
109
81

After removing "@":
33
95
177
93

Looks like I'll go back to the old habit of adding otherwise-unnecessary 
"@"s in time-sensitive handlers, being as careful as I used to have to 
be not to modify anything unless that's what I truly want.


Did copy-on-write get changed in v9, or is the scope of its effects just 
more limited than I had understood it to be?


--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \"Walker\" Available

[Richard Gaskin via use-livecode]

Malte Pfaff-Brill wrote:

> I wonder if shelling out to DIR on Windows and LS on Linux/Mac
> wouldn’t be the fastest option…

Maybe, but there's overhead in setting up the shell session.

Even if it's measurably faster, I would be surprised if it were 
noticeably faster.


Anything that depends on touching the disk with each call is likely to 
be far more I/O-bound than CPU-bound.


--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \\\"Walker\\\" Available

[Niggemann, Bernd via use-livecode]

>Which version of LC did you test with?

>I was under the impression that since LC switched to copy-on-write for all 
>>arguments we should no longer need to use "@" for performance, only for for
>logic.



Richard,

I tested using LC 9 GM, what kind of results do you get?


Kind regards

Bernd
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \"Walker\" Available

[Malte Pfaff-Brill via use-livecode]

I wonder if shelling out to DIR on Windows and LS on Linux/Mac wouldn’t be the 
fastest option…

Cheers,

Malte


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \"Walker\" Available

[Richard Gaskin via use-livecode]

Bernd Niggemann wrote:

> a combination of "private" and referenced variables (@) improves the
> speed of the calls somewhat.

Which version of LC did you test with?

I was under the impression that since LC switched to copy-on-write for 
all arguments we should no longer need to use "@" for performance, only 
for for logic.


--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory \"Walker\" Available

[Niggemann, Bernd via use-livecode]

Alex,

a combination of "private" and referenced variables (@) improves the speed of 
the calls somewhat.

Kind regards
Bernd

--
on mouseup
   local t1, t2
   constant K = 1000
   
   local x
   constant KX = 100
   
   put the millisecs into t1
   repeat K times
  put KX into x
  repeat
 if x = 0 then exit repeat
 subtract 1 from x
  end repeat
   end repeat
   put the millisecs into t2
   put t2 - t1 & CR after msg
   
   put the millisecs into t1
   repeat K times
  put KX into x
  repeat
 if x = 0 then exit repeat
 put myfunc(x) into x
  end repeat
   end repeat
   put the millisecs into t2
   put t2 - t1 & CR after msg
   
   put the millisecs into t1
   local tA, tB, tC, tD
   put 1 into tA
   put 2 into tB
   put "333" into tC
   put "444" into tD
   repeat K times
  put KX into x
  repeat
 if x = 0 then exit repeat
 put manyParameters(x, tA, tB, tC, tD) into x
  end repeat
   end repeat
   put the millisecs into t2
   put t2 - t1 & CR after msg
   
   put the millisecs into t1
   repeat K times
  put KX into x
  recursive x
  put the result into x
   end repeat
   put the millisecs into t2
   put t2 - t1 & CR after msg
end mouseup

private function myfunc @p
   return p-1
end myfunc

private function manyParameters @p, @p1, @p2, @p3, @p4
   return p-1
end manyParameters

private command recursive @p
   if p=0 then return 0
   subtract 1 from p
   recursive p
end recursive
--
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory "Walker" Available

[Richard Gaskin via use-livecode]

Ah, that makes sense.  It's not so much the recursion per se, but the 
more general advantage of inline processing vs handler calls.


In the test case below you know how many levels deep you're going, which 
seems a good fit for inline loops. But how do we handle cases like 
directory traversal, where we don't know in advance how deep we'll need 
to go?


It's at least comforting to see that while the overhead of handler calls 
is significant, it's mostly in relative comparison: If I read that 
correctly, there are 1000 iterations of 100 operations, for a total of 
100k operations.  If my coffee is working, that means an overhead of 
just 0.00208 ms per operation when called in a separate handler, no? 
(thinking 273ms total for separate handler - 65ms for inline, per 100k 
operations).


With directory traversal, I'd guess the overhead of physically touching 
each inode would outweigh that manyfold.


Still useful to keep in mind: when inline code can do the job clearly, 
it will be more efficient.


--
 Richard Gaskin
 Fourth World Systems


Alex Tweedly wrote:


On 05/05/2018 01:29, Richard Gaskin via use-livecode wrote:


How does recursion impair performance so significantly?
In general, there's significant work involved in a function or handler 
call and return - you need to establish a new context for locals, copy 
or calculate, parameters, etc.  My claim that recursion is slow relative 
to a serial- or loop-based version is language-independent (though there 
might be specific exceptions like LISP :-)


Compiler writers spend considerable effort minimizing the overhead of 
function calls - but still recursion is common, and indeed recognizing 
"tail-recursion" and optimizing it away is worthwhile.


I don't know enough (i.e. I know nothing) about LC's engine, so don't 
know specifically what might be involved for LC.


But here's a minimal test case (code below)
1.  65 : serial
2. 288 : many function calls
3. 471 : same number of calls as 2, more paramters
4. 273 : same number of calls as 2, but recursive

Note : result for 2 and 4 are the same - caused by the number of calls, 
not the fact that it's recursion.

Note : 3 (same as 2 but with extra parameters) is slower.

This does point the way towards possible improvements in recursive 
solutions (and specifically in the code I used in my earlier recursive 
directory walker function). I'll try those out and see if they make a 
difference.


Anyway - here's the code that gave the above results:

on mouseup
local t1, t2
constant K = 1000

local x
constant KX = 100

put the millisecs into t1
repeat K times
   put KX into x
   repeat
  if x = 0 then exit repeat
  subtract 1 from x
   end repeat
end repeat
put the millisecs into t2
put t2 - t1 & CR after msg

put the millisecs into t1
repeat K times
   put KX into x
   repeat
  if x = 0 then exit repeat
  put myfunc(x) into x
   end repeat
end repeat
put the millisecs into t2
put t2 - t1 & CR after msg

put the millisecs into t1
repeat K times
   put KX into x
   repeat
  if x = 0 then exit repeat
  put manyParameters(x, 1, 2, "333", "444") into x
   end repeat
end repeat
put the millisecs into t2
put t2 - t1 & CR after msg

put the millisecs into t1
repeat K times
   put KX into x
   put recursive(x) into x
end repeat
put the millisecs into t2
put t2 - t1 & CR after msg

end mouseup

function myfunc p
return p-1
end myfunc

function manyParameters p, p1, p2, p3, p4
return p-1
end manyParameters

function recursive p
if p=0 then return 0
return recursive(p-1)
end recursive

-- Alex.


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory "Walker" Available

[Alex Tweedly via use-livecode]

On 05/05/2018 01:29, Richard Gaskin via use-livecode wrote:


How does recursion impair performance so significantly?
In general, there's significant work involved in a function or handler 
call and return - you need to establish a new context for locals, copy 
or calculate, parameters, etc.  My claim that recursion is slow relative 
to a serial- or loop-based version is language-independent (though there 
might be specific exceptions like LISP :-)


Compiler writers spend considerable effort minimizing the overhead of 
function calls - but still recursion is common, and indeed recognizing 
"tail-recursion" and optimizing it away is worthwhile.


I don't know enough (i.e. I know nothing) about LC's engine, so don't 
know specifically what might be involved for LC.


But here's a minimal test case (code below)
1.  65 : serial
2. 288 : many function calls
3. 471 : same number of calls as 2, more paramters
4. 273 : same number of calls as 2, but recursive

Note : result for 2 and 4 are the same - caused by the number of calls, 
not the fact that it's recursion.

Note : 3 (same as 2 but with extra parameters) is slower.

This does point the way towards possible improvements in recursive 
solutions (and specifically in the code I used in my earlier recursive 
directory walker function). I'll try those out and see if they make a 
difference.


Anyway - here's the code that gave the above results:

on mouseup
   local t1, t2
   constant K = 1000

   local x
   constant KX = 100

   put the millisecs into t1
   repeat K times
  put KX into x
  repeat
 if x = 0 then exit repeat
 subtract 1 from x
  end repeat
   end repeat
   put the millisecs into t2
   put t2 - t1 & CR after msg

   put the millisecs into t1
   repeat K times
  put KX into x
  repeat
 if x = 0 then exit repeat
 put myfunc(x) into x
  end repeat
   end repeat
   put the millisecs into t2
   put t2 - t1 & CR after msg

   put the millisecs into t1
   repeat K times
  put KX into x
  repeat
 if x = 0 then exit repeat
 put manyParameters(x, 1, 2, "333", "444") into x
  end repeat
   end repeat
   put the millisecs into t2
   put t2 - t1 & CR after msg

   put the millisecs into t1
   repeat K times
  put KX into x
  put recursive(x) into x
   end repeat
   put the millisecs into t2
   put t2 - t1 & CR after msg

end mouseup

function myfunc p
   return p-1
end myfunc

function manyParameters p, p1, p2, p3, p4
   return p-1
end manyParameters

function recursive p
   if p=0 then return 0
   return recursive(p-1)
end recursive

-- Alex.

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory "Walker" Available

[Richard Gaskin via use-livecode]

Alex Tweedly wrote:

> Avoiding recursion is (almost) always a good idea, if it can be
> done without jumping through hoops; for a file walker I timed it
> (a couple of years ago) at somewhere between 20 and 35% faster.

How does recursion impair performance so significantly?

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Has Anyone Got A Directory "Walker" Available

[Alex Tweedly via use-livecode]

Thanks Ben - that's pretty similar to what I use. Avoiding recursion is 
(almost) always a good idea, if it can be done without jumping through 
hoops; for a file walker I timed it (a couple of years ago) at somewhere 
between 20 and 35% faster.


And it's so much easier to add a good progress indicator callback in the 
non-recursive method.


But I would really urge you to add the safety check that Richard 
recommended against protected-directories - otherwise you'll find 
yourself in an infinite loop.


-- Alex.

P.S. the other addition I have is a couple of parameters for 
"ignore-dot-folders and ignore-dot-files". I rarely want to see, for 
instance, .git folders, so in my library I default to ignoring 
directories beginning with a dot.



On 03/05/2018 20:48, Ben Rubinstein via use-livecode wrote:
FWIW I don't use recursion for directory walking. My typical routine 
(typing directly into email, may contain errors) is something along 
the following lines:


-- assumes tRootFolder is the path to start reviewing
put return into tFoldersToDo
repeat until tFoldersToDo = empty

    -- process another folder
    put line 1 of tFoldersToDo into tSubFolder
    delete line 1 of tFoldersToDo
    set the defaultFolder to (tRootFolder & tSubFolder)

    -- note sub-folders
    repeat for each line f in the folders
    if char 1 of f = "." then next repeat
    put tSubFolder & "/" & f & return after tFoldersToDo
    end repeat

    -- process files
    repeat for each line f in the files
    if char 1 of f = "." then next repeat
    -- do something with f
    -- or store the path, i.e. tSubFolder & "/" & f
    end repeat

end repeat

HTH,

Ben

On 21/04/2018 09:15, Ali Lloyd via use-livecode wrote:
Ah, it will throw an error - so best to use try / catch around the 
function

calls.

On Sat, Apr 21, 2018 at 9:13 AM Ali Lloyd  
wrote:



Now that we have files(pFolder), folder(pFolder) (and files(pFolder,
"detailed"), folders(pFolder, "detailed")), directory walking code 
can be

improved somewhat by not having to set the current folder.

Actually, I haven't checked what happens when you get
files(pRestrictedFolder)


On Fri, Apr 20, 2018 at 7:49 PM Richard Gaskin via use-livecode <
use-livecode@lists.runrev.com> wrote:


You may recall the dreaded recursion errors that have cropped up in
discussions of directory walkers.  They happen not because anyone has
directory trees > 40,000 folders deep, but because a permissions
restriction can prevent going into a subdirectory, causing the current
directory to be traversed again and again until the recursion error is
thrown.

The way to avoid this is to add an error check when setting the
directory, e.g.:

   set the directory to pFolder
   if the result is not empty then
 -- skip or report as needed
   end if


--
   Richard Gaskin
   Fourth World Systems


Sannyasin Brahmanathaswami wrote:

  >  I found this in a toolbox. Sent by some ago, by a LiveCode deva
  >
  > For what it's worth :
  >
  > # Filters the strings "." and ".." from a list
  > function filterDots pList
  >    local tList
  >
  >    put pList into tList
  >    filter tList without "."
  >    filter tList without ".."
  >
  >    return tList
  > end filterDots
  > # Returns a filtered list of files in the current directory
  > function filteredFiles
  >    return filterDots(the files)
  > end filteredFiles
  >
  > # Returns a filtered list of folders in the current directory
  > function filteredFolders
  >    return filterDots(the folders)
  > end filteredFolders
  >
  > # Returns a list of files in the current directory including
  > # each file's full path.
  > function filteredFilesWithPaths
  >    local tFiles, tFilesWithPaths
  >
  >    put filteredFiles() into tFiles
  >    repeat for each line tFile in tFiles
  >   put the directory & slash & tFile & return after \
  > tFilesWithPaths
  >    end repeat
  >    delete the last char of tFilesWithPaths
  >
  >    return tFilesWithPaths
  > end filteredFilesWithPaths
  >
  >
  > # Returns a list of files in a given folder, using recursion to
  > # include files in subfolders if desired.
  > function listFiles pFolder, pRecurse
  >    local tTotalFiles, tCurrentFiles, tFolders
  >
  > set the directory to pFolder
  >    put filteredFiles() into tCurrentFiles
  >    if not pRecurse then return tCurrentFiles
  >    if tCurrentFiles is not empty then
  >   put tCurrentFiles & return after tTotalFiles
  >    end if
  >    put filteredFolders() into tFolders
  >
  > repeat for each line tFolder in tFolders
  >   put listFiles((pFolder & slash & tFolder), pRecurse) into \
  >   tCurrentFiles
  >   if tCurrentFiles is not empty then
  >  put tCurrentFiles & return after tTotalFiles
  >   end if
  >    end repeat
  > delete the last char of tTotalFiles
  >
  >    return tTotalFiles
  > end 

Re: Has Anyone Got A Directory "Walker" Available

[Ben Rubinstein via use-livecode]

FWIW I don't use recursion for directory walking. My typical routine (typing 
directly into email, may contain errors) is something along the following lines:


-- assumes tRootFolder is the path to start reviewing
put return into tFoldersToDo
repeat until tFoldersToDo = empty

-- process another folder
put line 1 of tFoldersToDo into tSubFolder
delete line 1 of tFoldersToDo
set the defaultFolder to (tRootFolder & tSubFolder)

-- note sub-folders
repeat for each line f in the folders
if char 1 of f = "." then next repeat
put tSubFolder & "/" & f & return after tFoldersToDo
end repeat

-- process files
repeat for each line f in the files
if char 1 of f = "." then next repeat
-- do something with f
-- or store the path, i.e. tSubFolder & "/" & f
end repeat

end repeat

HTH,

Ben

On 21/04/2018 09:15, Ali Lloyd via use-livecode wrote:

Ah, it will throw an error - so best to use try / catch around the function
calls.

On Sat, Apr 21, 2018 at 9:13 AM Ali Lloyd  wrote:


Now that we have files(pFolder), folder(pFolder) (and files(pFolder,
"detailed"), folders(pFolder, "detailed")), directory walking code can be
improved somewhat by not having to set the current folder.

Actually, I haven't checked what happens when you get
files(pRestrictedFolder)


On Fri, Apr 20, 2018 at 7:49 PM Richard Gaskin via use-livecode <
use-livecode@lists.runrev.com> wrote:


You may recall the dreaded recursion errors that have cropped up in
discussions of directory walkers.  They happen not because anyone has
directory trees > 40,000 folders deep, but because a permissions
restriction can prevent going into a subdirectory, causing the current
directory to be traversed again and again until the recursion error is
thrown.

The way to avoid this is to add an error check when setting the
directory, e.g.:

   set the directory to pFolder
   if the result is not empty then
 -- skip or report as needed
   end if


--
   Richard Gaskin
   Fourth World Systems


Sannyasin Brahmanathaswami wrote:

  >  I found this in a toolbox. Sent by some ago, by a LiveCode deva
  >
  > For what it's worth :
  >
  > # Filters the strings "." and ".." from a list
  > function filterDots pList
  >local tList
  >
  >put pList into tList
  >filter tList without "."
  >filter tList without ".."
  >
  >return tList
  > end filterDots
  > # Returns a filtered list of files in the current directory
  > function filteredFiles
  >return filterDots(the files)
  > end filteredFiles
  >
  > # Returns a filtered list of folders in the current directory
  > function filteredFolders
  >return filterDots(the folders)
  > end filteredFolders
  >
  > # Returns a list of files in the current directory including
  > # each file's full path.
  > function filteredFilesWithPaths
  >local tFiles, tFilesWithPaths
  >
  >put filteredFiles() into tFiles
  >repeat for each line tFile in tFiles
  >   put the directory & slash & tFile & return after \
  > tFilesWithPaths
  >end repeat
  >delete the last char of tFilesWithPaths
  >
  >return tFilesWithPaths
  > end filteredFilesWithPaths
  >
  >
  > # Returns a list of files in a given folder, using recursion to
  > # include files in subfolders if desired.
  > function listFiles pFolder, pRecurse
  >local tTotalFiles, tCurrentFiles, tFolders
  >
  > set the directory to pFolder
  >put filteredFiles() into tCurrentFiles
  >if not pRecurse then return tCurrentFiles
  >if tCurrentFiles is not empty then
  >   put tCurrentFiles & return after tTotalFiles
  >end if
  >put filteredFolders() into tFolders
  >
  > repeat for each line tFolder in tFolders
  >   put listFiles((pFolder & slash & tFolder), pRecurse) into \
  >   tCurrentFiles
  >   if tCurrentFiles is not empty then
  >  put tCurrentFiles & return after tTotalFiles
  >   end if
  >end repeat
  > delete the last char of tTotalFiles
  >
  >return tTotalFiles
  > end listFiles
  >
  > # Returns a list of files with their containing folders, using
  > # recursion
  > # if desired to descend into sub folders.
  > function listFilesWithFolders pFolder, pRecurse
  >local tTotalFiles, tCurrentFiles, tFolders
  >
  > set the directory to pFolder
  >put filteredFiles() into tCurrentFiles
  >if not pRecurse then return pFolder & return & "--" & return \
  >& tCurrentFiles
  >if tCurrentFiles is not empty then
  > put pFolder & return & "--" & return after tTotalFiles
  >   put tCurrentFiles & return & return after tTotalFiles
  >end if
  > put filteredFolders() into tFolders
  >repeat 

Re: Has Anyone Got A Directory "Walker" Available

[Ali Lloyd via use-livecode]

Ah, it will throw an error - so best to use try / catch around the function
calls.

On Sat, Apr 21, 2018 at 9:13 AM Ali Lloyd  wrote:

> Now that we have files(pFolder), folder(pFolder) (and files(pFolder,
> "detailed"), folders(pFolder, "detailed")), directory walking code can be
> improved somewhat by not having to set the current folder.
>
> Actually, I haven't checked what happens when you get
> files(pRestrictedFolder)
>
>
> On Fri, Apr 20, 2018 at 7:49 PM Richard Gaskin via use-livecode <
> use-livecode@lists.runrev.com> wrote:
>
>> You may recall the dreaded recursion errors that have cropped up in
>> discussions of directory walkers.  They happen not because anyone has
>> directory trees > 40,000 folders deep, but because a permissions
>> restriction can prevent going into a subdirectory, causing the current
>> directory to be traversed again and again until the recursion error is
>> thrown.
>>
>> The way to avoid this is to add an error check when setting the
>> directory, e.g.:
>>
>>   set the directory to pFolder
>>   if the result is not empty then
>> -- skip or report as needed
>>   end if
>>
>>
>> --
>>   Richard Gaskin
>>   Fourth World Systems
>>
>>
>> Sannyasin Brahmanathaswami wrote:
>>
>>  >  I found this in a toolbox. Sent by some ago, by a LiveCode deva
>>  >
>>  > For what it's worth :
>>  >
>>  > # Filters the strings "." and ".." from a list
>>  > function filterDots pList
>>  >local tList
>>  >
>>  >put pList into tList
>>  >filter tList without "."
>>  >filter tList without ".."
>>  >
>>  >return tList
>>  > end filterDots
>>  > # Returns a filtered list of files in the current directory
>>  > function filteredFiles
>>  >return filterDots(the files)
>>  > end filteredFiles
>>  >
>>  > # Returns a filtered list of folders in the current directory
>>  > function filteredFolders
>>  >return filterDots(the folders)
>>  > end filteredFolders
>>  >
>>  > # Returns a list of files in the current directory including
>>  > # each file's full path.
>>  > function filteredFilesWithPaths
>>  >local tFiles, tFilesWithPaths
>>  >
>>  >put filteredFiles() into tFiles
>>  >repeat for each line tFile in tFiles
>>  >   put the directory & slash & tFile & return after \
>>  > tFilesWithPaths
>>  >end repeat
>>  >delete the last char of tFilesWithPaths
>>  >
>>  >return tFilesWithPaths
>>  > end filteredFilesWithPaths
>>  >
>>  >
>>  > # Returns a list of files in a given folder, using recursion to
>>  > # include files in subfolders if desired.
>>  > function listFiles pFolder, pRecurse
>>  >local tTotalFiles, tCurrentFiles, tFolders
>>  >
>>  > set the directory to pFolder
>>  >put filteredFiles() into tCurrentFiles
>>  >if not pRecurse then return tCurrentFiles
>>  >if tCurrentFiles is not empty then
>>  >   put tCurrentFiles & return after tTotalFiles
>>  >end if
>>  >put filteredFolders() into tFolders
>>  >
>>  > repeat for each line tFolder in tFolders
>>  >   put listFiles((pFolder & slash & tFolder), pRecurse) into \
>>  >   tCurrentFiles
>>  >   if tCurrentFiles is not empty then
>>  >  put tCurrentFiles & return after tTotalFiles
>>  >   end if
>>  >end repeat
>>  > delete the last char of tTotalFiles
>>  >
>>  >return tTotalFiles
>>  > end listFiles
>>  >
>>  > # Returns a list of files with their containing folders, using
>>  > # recursion
>>  > # if desired to descend into sub folders.
>>  > function listFilesWithFolders pFolder, pRecurse
>>  >local tTotalFiles, tCurrentFiles, tFolders
>>  >
>>  > set the directory to pFolder
>>  >put filteredFiles() into tCurrentFiles
>>  >if not pRecurse then return pFolder & return & "--" & return \
>>  >& tCurrentFiles
>>  >if tCurrentFiles is not empty then
>>  > put pFolder & return & "--" & return after tTotalFiles
>>  >   put tCurrentFiles & return & return after tTotalFiles
>>  >end if
>>  > put filteredFolders() into tFolders
>>  >repeat for each line tFolder in tFolders
>>  >   put listFilesWithFolders((pFolder & slash & tFolder), \
>>  >   pRecurse) into tCurrentFiles
>>  >   if tCurrentFiles is not empty then put tCurrentFiles & \
>>  >   return after tTotalFiles
>>  > end repeat
>>  >delete the last char of tTotalFiles
>>  >
>>  >return tTotalFiles
>>  > end listFilesWithFolders
>>  >
>>  > # Returns a list of files with the full paths
>>  > function listFilesWithPaths pFolder, pRecurse
>>  >local tTotalFiles, tCurrentFiles, tFolders
>>  >
>>  >set the directory to pFolder
>>  >put filteredFilesWithPaths() into tCurrentFiles
>>  >if not pRecurse then return tCurrentFiles
>>  > if tCurrentFiles is not empty then put tCurrentFiles & \
>>  >return after tTotalFiles
>>  >put filteredFolders() into tFolders
>>  >repeat for each line tFolder in tFolders
>>  >   put listFilesWithPaths((pFolder & slash & 

Re: Has Anyone Got A Directory "Walker" Available

[Ali Lloyd via use-livecode]

Now that we have files(pFolder), folder(pFolder) (and files(pFolder,
"detailed"), folders(pFolder, "detailed")), directory walking code can be
improved somewhat by not having to set the current folder.

Actually, I haven't checked what happens when you get
files(pRestrictedFolder)

On Fri, Apr 20, 2018 at 7:49 PM Richard Gaskin via use-livecode <
use-livecode@lists.runrev.com> wrote:

> You may recall the dreaded recursion errors that have cropped up in
> discussions of directory walkers.  They happen not because anyone has
> directory trees > 40,000 folders deep, but because a permissions
> restriction can prevent going into a subdirectory, causing the current
> directory to be traversed again and again until the recursion error is
> thrown.
>
> The way to avoid this is to add an error check when setting the
> directory, e.g.:
>
>   set the directory to pFolder
>   if the result is not empty then
> -- skip or report as needed
>   end if
>
>
> --
>   Richard Gaskin
>   Fourth World Systems
>
>
> Sannyasin Brahmanathaswami wrote:
>
>  >  I found this in a toolbox. Sent by some ago, by a LiveCode deva
>  >
>  > For what it's worth :
>  >
>  > # Filters the strings "." and ".." from a list
>  > function filterDots pList
>  >local tList
>  >
>  >put pList into tList
>  >filter tList without "."
>  >filter tList without ".."
>  >
>  >return tList
>  > end filterDots
>  > # Returns a filtered list of files in the current directory
>  > function filteredFiles
>  >return filterDots(the files)
>  > end filteredFiles
>  >
>  > # Returns a filtered list of folders in the current directory
>  > function filteredFolders
>  >return filterDots(the folders)
>  > end filteredFolders
>  >
>  > # Returns a list of files in the current directory including
>  > # each file's full path.
>  > function filteredFilesWithPaths
>  >local tFiles, tFilesWithPaths
>  >
>  >put filteredFiles() into tFiles
>  >repeat for each line tFile in tFiles
>  >   put the directory & slash & tFile & return after \
>  > tFilesWithPaths
>  >end repeat
>  >delete the last char of tFilesWithPaths
>  >
>  >return tFilesWithPaths
>  > end filteredFilesWithPaths
>  >
>  >
>  > # Returns a list of files in a given folder, using recursion to
>  > # include files in subfolders if desired.
>  > function listFiles pFolder, pRecurse
>  >local tTotalFiles, tCurrentFiles, tFolders
>  >
>  > set the directory to pFolder
>  >put filteredFiles() into tCurrentFiles
>  >if not pRecurse then return tCurrentFiles
>  >if tCurrentFiles is not empty then
>  >   put tCurrentFiles & return after tTotalFiles
>  >end if
>  >put filteredFolders() into tFolders
>  >
>  > repeat for each line tFolder in tFolders
>  >   put listFiles((pFolder & slash & tFolder), pRecurse) into \
>  >   tCurrentFiles
>  >   if tCurrentFiles is not empty then
>  >  put tCurrentFiles & return after tTotalFiles
>  >   end if
>  >end repeat
>  > delete the last char of tTotalFiles
>  >
>  >return tTotalFiles
>  > end listFiles
>  >
>  > # Returns a list of files with their containing folders, using
>  > # recursion
>  > # if desired to descend into sub folders.
>  > function listFilesWithFolders pFolder, pRecurse
>  >local tTotalFiles, tCurrentFiles, tFolders
>  >
>  > set the directory to pFolder
>  >put filteredFiles() into tCurrentFiles
>  >if not pRecurse then return pFolder & return & "--" & return \
>  >& tCurrentFiles
>  >if tCurrentFiles is not empty then
>  > put pFolder & return & "--" & return after tTotalFiles
>  >   put tCurrentFiles & return & return after tTotalFiles
>  >end if
>  > put filteredFolders() into tFolders
>  >repeat for each line tFolder in tFolders
>  >   put listFilesWithFolders((pFolder & slash & tFolder), \
>  >   pRecurse) into tCurrentFiles
>  >   if tCurrentFiles is not empty then put tCurrentFiles & \
>  >   return after tTotalFiles
>  > end repeat
>  >delete the last char of tTotalFiles
>  >
>  >return tTotalFiles
>  > end listFilesWithFolders
>  >
>  > # Returns a list of files with the full paths
>  > function listFilesWithPaths pFolder, pRecurse
>  >local tTotalFiles, tCurrentFiles, tFolders
>  >
>  >set the directory to pFolder
>  >put filteredFilesWithPaths() into tCurrentFiles
>  >if not pRecurse then return tCurrentFiles
>  > if tCurrentFiles is not empty then put tCurrentFiles & \
>  >return after tTotalFiles
>  >put filteredFolders() into tFolders
>  >repeat for each line tFolder in tFolders
>  >   put listFilesWithPaths((pFolder & slash & tFolder), \
>  >  pRecurse) into tCurrentFiles
>  >   if tCurrentFiles is not empty then put tCurrentFiles & \
>  >   return after tTotalFiles
>  > end repeat
>  >delete the last char of tTotalFiles
>  >
>  >return tTotalFiles
>  > end listFilesWithPaths
>  >
>  > # Returns a sorted list 

Re: Has Anyone Got A Directory "Walker" Available

[Richard Gaskin via use-livecode]

You may recall the dreaded recursion errors that have cropped up in 
discussions of directory walkers.  They happen not because anyone has 
directory trees > 40,000 folders deep, but because a permissions 
restriction can prevent going into a subdirectory, causing the current 
directory to be traversed again and again until the recursion error is 
thrown.


The way to avoid this is to add an error check when setting the 
directory, e.g.:


 set the directory to pFolder
 if the result is not empty then
   -- skip or report as needed
 end if


--
 Richard Gaskin
 Fourth World Systems


Sannyasin Brahmanathaswami wrote:

>  I found this in a toolbox. Sent by some ago, by a LiveCode deva
>
> For what it's worth :
>
> # Filters the strings "." and ".." from a list
> function filterDots pList
>local tList
>
>put pList into tList
>filter tList without "."
>filter tList without ".."
>
>return tList
> end filterDots
> # Returns a filtered list of files in the current directory
> function filteredFiles
>return filterDots(the files)
> end filteredFiles
>
> # Returns a filtered list of folders in the current directory
> function filteredFolders
>return filterDots(the folders)
> end filteredFolders
>
> # Returns a list of files in the current directory including
> # each file's full path.
> function filteredFilesWithPaths
>local tFiles, tFilesWithPaths
>
>put filteredFiles() into tFiles
>repeat for each line tFile in tFiles
>   put the directory & slash & tFile & return after \
> tFilesWithPaths
>end repeat
>delete the last char of tFilesWithPaths
>
>return tFilesWithPaths
> end filteredFilesWithPaths
>
>
> # Returns a list of files in a given folder, using recursion to
> # include files in subfolders if desired.
> function listFiles pFolder, pRecurse
>local tTotalFiles, tCurrentFiles, tFolders
>
> set the directory to pFolder
>put filteredFiles() into tCurrentFiles
>if not pRecurse then return tCurrentFiles
>if tCurrentFiles is not empty then
>   put tCurrentFiles & return after tTotalFiles
>end if
>put filteredFolders() into tFolders
>
> repeat for each line tFolder in tFolders
>   put listFiles((pFolder & slash & tFolder), pRecurse) into \
>   tCurrentFiles
>   if tCurrentFiles is not empty then
>  put tCurrentFiles & return after tTotalFiles
>   end if
>end repeat
> delete the last char of tTotalFiles
>
>return tTotalFiles
> end listFiles
>
> # Returns a list of files with their containing folders, using
> # recursion
> # if desired to descend into sub folders.
> function listFilesWithFolders pFolder, pRecurse
>local tTotalFiles, tCurrentFiles, tFolders
>
> set the directory to pFolder
>put filteredFiles() into tCurrentFiles
>if not pRecurse then return pFolder & return & "--" & return \
>& tCurrentFiles
>if tCurrentFiles is not empty then
> put pFolder & return & "--" & return after tTotalFiles
>   put tCurrentFiles & return & return after tTotalFiles
>end if
> put filteredFolders() into tFolders
>repeat for each line tFolder in tFolders
>   put listFilesWithFolders((pFolder & slash & tFolder), \
>   pRecurse) into tCurrentFiles
>   if tCurrentFiles is not empty then put tCurrentFiles & \
>   return after tTotalFiles
> end repeat
>delete the last char of tTotalFiles
>
>return tTotalFiles
> end listFilesWithFolders
>
> # Returns a list of files with the full paths
> function listFilesWithPaths pFolder, pRecurse
>local tTotalFiles, tCurrentFiles, tFolders
>
>set the directory to pFolder
>put filteredFilesWithPaths() into tCurrentFiles
>if not pRecurse then return tCurrentFiles
> if tCurrentFiles is not empty then put tCurrentFiles & \
>return after tTotalFiles
>put filteredFolders() into tFolders
>repeat for each line tFolder in tFolders
>   put listFilesWithPaths((pFolder & slash & tFolder), \
>  pRecurse) into tCurrentFiles
>   if tCurrentFiles is not empty then put tCurrentFiles & \
>   return after tTotalFiles
> end repeat
>delete the last char of tTotalFiles
>
>return tTotalFiles
> end listFilesWithPaths
>
> # Returns a sorted list of the files in pFolder. Again using
> # recursion if
> # required.
> function listSortedFiles pFolder, pRecurse
>local tFiles
>
>put listFiles(pFolder, pRecurse) into tFiles
>sort lines of tFiles by listSortedFilesSortKey(each)
>
>return tFiles
> end listSortedFiles
>
> # Used by the listSortedFiles() function. Returns a sort key
> # from each file's name to
> # allow the sorting to be fully customized.
> function listSortedFilesSortKey pFile
># Use this value for a normal text sort
>return pFile
> end listSortedFilesSortKey
>
>
> on doSomething pFile
>put pFile & return after msg
> end doSomething
>
> --This will achieve similar results to listing the files except that
> it will be slower because text has to be drawn to the 

Re: Has Anyone Got A Directory "Walker" Available

[Paul Hibbert via use-livecode]

> On 19 Apr 2018, at 11:20, Sannyasin Brahmanathaswami via use-livecode 
> > wrote:
> 
> Never mind…
> 
> I found this in a toolbox. Sent by some ago, by a LiveCode deva 


That looks very much like the code supplied under the “Resources” menu, from 
Tutorials>General>Files and Folders.

I’ve used it on a few projects n the past and it has worked pretty well for me.

Paul
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Has Anyone Got A Directory "Walker" Available

[Sannyasin Brahmanathaswami via use-livecode]

I have directory available this days of Ken Ray, in the on system of set the 
default folder and get the the files, registered that path. descend recursively 
to the level… etc.

Do anyone has a new one handy they could share using the folder and file 
function?

BR


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode