Re: [Unicon-group] Walk of file directory
in the process of tracking the > open() problem. I committed a fix for that leak. This is only affecting > Windows. > > Short term solution: flush your TMP folder. > Long term: we will into ways to improve our tmp file strategy to overcome > the shortcoming of Windows API. This will come in a later date! :) > > Cheers, > Jafar > > > On Thu, Jan 22, 2015 at 4:43 AM, Sergey Logichev > wrote: > > Jafar, > > You've provided very interesting version of walk directory algorithm. > Communication with active threads' is a great thing! > I have checked your program under Windows 7. I was confused the fact that > execution time is negligibly depended on number of concurrent threads. I > dug into and discovered that the first operation open(s) takes near ALL > execution time! 95% at least. Check it yourself when you slightly edit > getdirs(): > ... > if ( stat(s).mode ? ="d" ) & ( tm := &time, d := open(s) ) then { > if n=1 then write(s," : ",&time-tm) > ... > > > So, if first open() is so long then all other enhancements have no sense. > Please clarify if I am wrong. > > Best regards, > Sergey > > 22.01.2015, 00:58, "Jafar Al-Gharaibeh" : > > Here is a slightly tweaked/reformatted version. It now by default > auto-detect the number of available cores in the machine and launch twice > as many threads. > > --Jafar > > On Wed, Jan 21, 2015 at 12:17 PM, Jafar Al-Gharaibeh > wrote: > > David, > > I added a threaded solution @ > http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon >Please review/edit as you see fit. (The source file is attached). > Combining recursion with thread might not be the best solution for this > problem. If I were to put this in real use I'd go with an iterative > approach using master/workers model. Anyway, this is a excellent > demonstration on how to use threads!. The key features are: > >1- How to create threads, limit their numbers, self-load balanced (new > threads are spawned at the time/place where needed. One they are done, > they vanish allowing new threads to pop up in new places in the directory > structure) >2- pass data and collect results to/from the threads using the new > language features. > > > Here is some sample output from my desktop machine (quad-core with > mechanical HDD. I will try another machine with an SSD and see if more > threads scale better). > > the first argument to the program is the target directory. The second is > the maximum number of concurrent threads to use at any given moment. (soft > limit! my counters are "unmutexed", so the actual number might deviate). > Note that this is different from the actual number of threads used during > the run which is reported at the end. The program can create/destroy > threads as needed, but cannot use more than "max" # of threads at any > given moment, and again "max" is "soft". :) > > Cheers, > Jafar > > c:\proj>tdir c:\ 1 > 39708 directories in 99867 ms using 1 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 6 ms using 4 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 87650 ms using 4 threads > > c:\proj>tdir c:\ 1 > 39708 directories in 92525 ms using 1 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 95655 ms using 4 threads > > c:\proj>tdir c:\ 16 > 39708 directories in 66138 ms using 21 threads > > c:\proj>tdir c:\ 8 > 39708 directories in 69307 ms using 8 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 70539 ms using 4 threads > > c:\proj>tdir c:\ 16 > 39708 directories in 76392 ms using 32 threads > > > > On Sun, Jan 11, 2015 at 1:25 PM, David Gamey > wrote: > > Sergey, > > I am responsible for much of the Rosetta code contributions (thanks also > to Steve, Andrew, Matt, Peter, and about 4 others) and this one in > particular dating from 2010. As I recall this was before the > multi-threading versions were widely available. I think multi-threading is > underrepresented in Rosetta/Unicon. > > If you come up with a multi-threading version, we should add it to the > post as an alternative version. If you don't feel comfortable doing this, > post the code and I can add it. > > David > > > -- > *From:* Sergey Logichev > *To:* Jafar Al-Gharaibeh > *Cc:* Unicon group > *Sent:* Sunday, January 11, 2015 1:16 AM > *Subject:* Re: [Unicon-group] Walk of file directory > > Jafar, > > Thank you for a whole bundle of advices and suggestions! Threads are worth > to try. The thought of search b
Re: [Unicon-group] Walk of file directory
ent threads. I dug into and discovered that the first operation open(s) takes near ALL execution time! 95% at least. Check it yourself when you slightly edit getdirs():...if ( stat(s).mode ? ="d" ) & ( tm := &time, d := open(s) ) then { if n=1 then write(s," : ",&time-tm)... So, if first open() is so long then all other enhancements have no sense. Please clarify if I am wrong. Best regards,Sergey 22.01.2015, 00:58, "Jafar Al-Gharaibeh" <to.ja...@gmail.com>:Here is a slightly tweaked/reformatted version. It now by default auto-detect the number of available cores in the machine and launch twice as many threads. --JafarOn Wed, Jan 21, 2015 at 12:17 PM, Jafar Al-Gharaibeh <to.ja...@gmail.com> wrote:David, I added a threaded solution @ http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon Please review/edit as you see fit. (The source file is attached). Combining recursion with thread might not be the best solution for this problem. If I were to put this in real use I'd go with an iterative approach using master/workers model. Anyway, this is a excellent demonstration on how to use threads!. The key features are: 1- How to create threads, limit their numbers, self-load balanced (new threads are spawned at the time/place where needed. One they are done, they vanish allowing new threads to pop up in new places in the directory structure) 2- pass data and collect results to/from the threads using the new language features. Here is some sample output from my desktop machine (quad-core with mechanical HDD. I will try another machine with an SSD and see if more threads scale better). the first argument to the program is the target directory. The second is the maximum number of concurrent threads to use at any given moment. (soft limit! my counters are "unmutexed", so the actual number might deviate). Note that this is different from the actual number of threads used during the run which is reported at the end. The program can create/destroy threads as needed, but cannot use more than "max" # of threads at any given moment, and again "max" is "soft". :) Cheers,Jafar c:\proj>tdir c:\ 139708 directories in 99867 ms using 1 threads c:\proj>tdir c:\ 439708 directories in 6 ms using 4 threads c:\proj>tdir c:\ 439708 directories in 87650 ms using 4 threads c:\proj>tdir c:\ 139708 directories in 92525 ms using 1 threads c:\proj>tdir c:\ 439708 directories in 95655 ms using 4 threads c:\proj>tdir c:\ 1639708 directories in 66138 ms using 21 threads c:\proj>tdir c:\ 839708 directories in 69307 ms using 8 threads c:\proj>tdir c:\ 439708 directories in 70539 ms using 4 threads c:\proj>tdir c:\ 1639708 directories in 76392 ms using 32 threads On Sun, Jan 11, 2015 at 1:25 PM, David Gamey <david.ga...@rogers.com> wrote:Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev <slogic...@yandex.ru> To: Jafar Al-Gharaibeh <to.ja...@gmail.com> Cc: Unicon group <unicon-group@lists.sourceforge.net> Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" <to.ja...@gmail.com>:Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this: L := open("*.icn") to get a list of all of Unicon source files in t
Re: [Unicon-group] Walk of file directory
ows API. This will come in a later date! :) >> >> Cheers, >> Jafar >> >> >> On Thu, Jan 22, 2015 at 4:43 AM, Sergey Logichev >> wrote: >> >> Jafar, >> >> You've provided very interesting version of walk directory algorithm. >> Communication with active threads' is a great thing! >> I have checked your program under Windows 7. I was confused the fact that >> execution time is negligibly depended on number of concurrent threads. I >> dug into and discovered that the first operation open(s) takes near ALL >> execution time! 95% at least. Check it yourself when you slightly edit >> getdirs(): >> ... >> if ( stat(s).mode ? ="d" ) & ( tm := &time, d := open(s) ) then { >> if n=1 then write(s," : ",&time-tm) >> ... >> >> >> So, if first open() is so long then all other enhancements have no sense. >> Please clarify if I am wrong. >> >> Best regards, >> Sergey >> >> 22.01.2015, 00:58, "Jafar Al-Gharaibeh" : >> >> Here is a slightly tweaked/reformatted version. It now by default >> auto-detect the number of available cores in the machine and launch twice >> as many threads. >> >> --Jafar >> >> On Wed, Jan 21, 2015 at 12:17 PM, Jafar Al-Gharaibeh >> wrote: >> >> David, >> >> I added a threaded solution @ >> http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon >>Please review/edit as you see fit. (The source file is attached). >> Combining recursion with thread might not be the best solution for this >> problem. If I were to put this in real use I'd go with an iterative >> approach using master/workers model. Anyway, this is a excellent >> demonstration on how to use threads!. The key features are: >> >>1- How to create threads, limit their numbers, self-load balanced (new >> threads are spawned at the time/place where needed. One they are done, >> they vanish allowing new threads to pop up in new places in the directory >> structure) >>2- pass data and collect results to/from the threads using the new >> language features. >> >> >> Here is some sample output from my desktop machine (quad-core with >> mechanical HDD. I will try another machine with an SSD and see if more >> threads scale better). >> >> the first argument to the program is the target directory. The second is >> the maximum number of concurrent threads to use at any given moment. (soft >> limit! my counters are "unmutexed", so the actual number might deviate). >> Note that this is different from the actual number of threads used during >> the run which is reported at the end. The program can create/destroy >> threads as needed, but cannot use more than "max" # of threads at any >> given moment, and again "max" is "soft". :) >> >> Cheers, >> Jafar >> >> c:\proj>tdir c:\ 1 >> 39708 directories in 99867 ms using 1 threads >> >> c:\proj>tdir c:\ 4 >> 39708 directories in 6 ms using 4 threads >> >> c:\proj>tdir c:\ 4 >> 39708 directories in 87650 ms using 4 threads >> >> c:\proj>tdir c:\ 1 >> 39708 directories in 92525 ms using 1 threads >> >> c:\proj>tdir c:\ 4 >> 39708 directories in 95655 ms using 4 threads >> >> c:\proj>tdir c:\ 16 >> 39708 directories in 66138 ms using 21 threads >> >> c:\proj>tdir c:\ 8 >> 39708 directories in 69307 ms using 8 threads >> >> c:\proj>tdir c:\ 4 >> 39708 directories in 70539 ms using 4 threads >> >> c:\proj>tdir c:\ 16 >> 39708 directories in 76392 ms using 32 threads >> >> >> >> On Sun, Jan 11, 2015 at 1:25 PM, David Gamey >> wrote: >> >> Sergey, >> >> I am responsible for much of the Rosetta code contributions (thanks also >> to Steve, Andrew, Matt, Peter, and about 4 others) and this one in >> particular dating from 2010. As I recall this was before the >> multi-threading versions were widely available. I think multi-threading is >> underrepresented in Rosetta/Unicon. >> >> If you come up with a multi-threading version, we should add it to the >> post as an alternative version. If you don't feel comfortable doing this, >> post the code and I can add it. >> >> David >> >> >> -- >> *From:* Sergey Logichev >> *To:* Jafar Al-Gharaibeh >> *Cc:* Unicon group >> *Sent:* Sunday, Jan
Re: [Unicon-group] Walk of file directory
as you see fit. (The source file is attached). Combining recursion with thread might not be >>>>>>>the best solution for this problem. If I were to put this in real use I'd go with an iterative approach using master/>>>>>>>workers model. Anyway, this is a excellent demonstration on how to use threads!. The key features are: 1- How to create threads, limit their numbers, self-load balanced (new threads are spawned at the time/place >>>>>>>where needed. One they are done, they vanish allowing new threads to pop up in new places in the directory >>>>>>>structure) 2- pass data and collect results to/from the threads using the new language features. Here is some sample output from my desktop machine (quad-core with mechanical HDD. I will try another >>>>>>>machine with an SSD and see if more threads scale better).the first argument to the program is the target directory. The second is the maximum number of concurrent >>>>>>>threads to use at any given moment. (soft limit! my counters are "unmutexed", so the actual number might >>>>>>>deviate). Note that this is different from the actual number of threads used during the run which is reported at >>>>>>>the end. The program can create/destroy threads as needed, but cannot use more than "max" # of threads at >>>>>>>any given moment, and again "max" is "soft". :) Cheers, Jafar c:\proj>tdir c:\ 1 39708 directories in 99867 ms using 1 threads c:\proj>tdir c:\ 4 39708 directories in 6 ms using 4 threads c:\proj>tdir c:\ 4 39708 directories in 87650 ms using 4 threads c:\proj>tdir c:\ 1 39708 directories in 92525 ms using 1 threads c:\proj>tdir c:\ 4 39708 directories in 95655 ms using 4 threads c:\proj>tdir c:\ 16 39708 directories in 66138 ms using 21 threads c:\proj>tdir c:\ 8 39708 directories in 69307 ms using 8 threads c:\proj>tdir c:\ 4 39708 directories in 70539 ms using 4 threads c:\proj>tdir c:\ 16 39708 directories in 76392 ms using 32 threads On Sun, Jan 11, 2015 at 1:25 PM, David Gamey wrote: Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, >>>>>>>>Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the >>>>>>>>multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/>>>>>>>>Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. >>>>>>>>If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev To: Jafar Al-Gharaibeh Cc: Unicon group Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth >>>>>>>>>to try. The thought of search by file attributes is very useful too. Your >>>>>>>>>suggestion about slow I/O partly is right. For UNIX I tried the program on >>>>>>>>>Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for >>>>>>>>>Windows it was quite fast HDD. It would be interesting to compare >>>>>>>>>performance of the program on Windows with classic approach based on >>>>>>>>>Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus >>>>>>>>>implementations of this algorithm. Feel that it will be faster but in which >>>>>>>>>degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : Sergey, There are so many things that came to mind when I saw your >>>>>>>>>>program. 1- At the end of your email, sourceforge ad says "Go Parallel", >>>>>>>>>>Which is not a bad idea for this highly parallel application.There is a similar program "wordcount" listed in my dissertation >>>>>>>>>>(available on unicon.org) that go through directories and count >>>>>>>>>>words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would >>>>>>>>>>greatly (I believe) speedup your program. For example you can >>>>>>>>>>do this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would b
Re: [Unicon-group] Walk of file directory
hreads, limit their numbers, self-load balanced (new > threads are spawned at the time/place where needed. One they are done, > they vanish allowing new threads to pop up in new places in the directory > structure) >2- pass data and collect results to/from the threads using the new > language features. > > > Here is some sample output from my desktop machine (quad-core with > mechanical HDD. I will try another machine with an SSD and see if more > threads scale better). > > the first argument to the program is the target directory. The second is > the maximum number of concurrent threads to use at any given moment. (soft > limit! my counters are "unmutexed", so the actual number might deviate). > Note that this is different from the actual number of threads used during > the run which is reported at the end. The program can create/destroy > threads as needed, but cannot use more than "max" # of threads at any > given moment, and again "max" is "soft". :) > > Cheers, > Jafar > > c:\proj>tdir c:\ 1 > 39708 directories in 99867 ms using 1 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 6 ms using 4 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 87650 ms using 4 threads > > c:\proj>tdir c:\ 1 > 39708 directories in 92525 ms using 1 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 95655 ms using 4 threads > > c:\proj>tdir c:\ 16 > 39708 directories in 66138 ms using 21 threads > > c:\proj>tdir c:\ 8 > 39708 directories in 69307 ms using 8 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 70539 ms using 4 threads > > c:\proj>tdir c:\ 16 > 39708 directories in 76392 ms using 32 threads > > > > On Sun, Jan 11, 2015 at 1:25 PM, David Gamey > wrote: > > Sergey, > > I am responsible for much of the Rosetta code contributions (thanks also > to Steve, Andrew, Matt, Peter, and about 4 others) and this one in > particular dating from 2010. As I recall this was before the > multi-threading versions were widely available. I think multi-threading is > underrepresented in Rosetta/Unicon. > > If you come up with a multi-threading version, we should add it to the > post as an alternative version. If you don't feel comfortable doing this, > post the code and I can add it. > > David > > > -- > *From:* Sergey Logichev > *To:* Jafar Al-Gharaibeh > *Cc:* Unicon group > *Sent:* Sunday, January 11, 2015 1:16 AM > *Subject:* Re: [Unicon-group] Walk of file directory > > Jafar, > > Thank you for a whole bundle of advices and suggestions! Threads are worth > to try. The thought of search by file attributes is very useful too. Your > suggestion about slow I/O partly is right. For UNIX I tried the program on > Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for > Windows it was quite fast HDD. It would be interesting to compare > performance of the program on Windows with classic approach based on Win32 > _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus > implementations of this algorithm. Feel that it will be faster but in which > degree? > > Sergey > > 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : > > > Sergey, > > There are so many things that came to mind when I saw your program. > > 1- At the end of your email, sourceforge ad says "Go Parallel", Which is > not a bad idea for this highly parallel application. > > There is a similar program "wordcount" listed in my dissertation > (available on unicon.org) that go through directories and count words in > every file using threads (Chapter 7, page 107) > > 2- Unicon open() already supports " pattern matching that would greatly (I > believe) speedup your program. For example you can do this: > L := open("*.icn") > >to get a list of all of Unicon source files in the current directory. > > Note: It would be nice if there were a way to tell open() to return > files not only based on a pattern, but also on file attribute to allow > something like "get me all directories in the current directory", or "get > me all read only file". There are a lot of situations where filtering > directory names for example is very useful - like this program > > 3- The program on Rosetta Code is not optimized for speed. You can > minimize the number of lists created and put() by careful rewriting of the > code. > > 4- Depending on how deep the directory tree is, there might be a lot of > I/O going on. A slow disk might limit how fast you can go regardless of how > optimized your code is. > > I will sh
Re: [Unicon-group] Walk of file directory
d is the maximum number of concurrent threads >>>>>to use at any given moment. (soft limit! my counters are "unmutexed", so the actual number might deviate). Note >>>>>that this is different from the actual number of threads used during the run which is reported at the end. The >>>>>program can create/destroy threads as needed, but cannot use more than "max" # of threads at any given moment, >>>>>and again "max" is "soft". :) Cheers, Jafar c:\proj>tdir c:\ 1 39708 directories in 99867 ms using 1 threads c:\proj>tdir c:\ 4 39708 directories in 6 ms using 4 threads c:\proj>tdir c:\ 4 39708 directories in 87650 ms using 4 threads c:\proj>tdir c:\ 1 39708 directories in 92525 ms using 1 threads c:\proj>tdir c:\ 4 39708 directories in 95655 ms using 4 threads c:\proj>tdir c:\ 16 39708 directories in 66138 ms using 21 threads c:\proj>tdir c:\ 8 39708 directories in 69307 ms using 8 threads c:\proj>tdir c:\ 4 39708 directories in 70539 ms using 4 threads c:\proj>tdir c:\ 16 39708 directories in 76392 ms using 32 threads On Sun, Jan 11, 2015 at 1:25 PM, David Gamey wrote: Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, >>>>>>and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi->>>>>>threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If >>>>>>you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev To: Jafar Al-Gharaibeh Cc: Unicon group Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to >>>>>>>try. The thought of search by file attributes is very useful too. Your suggestion >>>>>>>about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with >>>>>>>6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast >>>>>>>HDD. It would be interesting to compare performance of the program on >>>>>>>Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT >>>>>>>functions. I have threaded Delphi/Lazarus implementations of this algorithm. >>>>>>>Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : Sergey, There are so many things that came to mind when I saw your >>>>>>>>program. 1- At the end of your email, sourceforge ad says "Go Parallel", >>>>>>>>Which is not a bad idea for this highly parallel application.There is a similar program "wordcount" listed in my dissertation >>>>>>>>(available on unicon.org) that go through directories and count >>>>>>>>words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would >>>>>>>>greatly (I believe) speedup your program. For example you can do >>>>>>>>this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would be nice if there were a way to tell open() to return files >>>>>>>>not only based on a pattern, but also on file attribute to allow >>>>>>>>something like "get me all directories in the current directory", or >>>>>>>>"get me all read only file". There are a lot of situations where filtering >>>>>>>>directory names for example is very useful - like this program 3- The program on Rosetta Code is not optimized for speed. You >>>>>>>>can minimize the number of lists created and put() by careful >>>>>>>>rewriting of the code. 4- Depending on how deep the directory tree is, there might be a lot >>>>>>>>of I/O going on. A slow disk might limit how fast you can go >>>>>>>>regardless of how optimized your code is. I will share results if get around trying any of these options. Cheers, Jafar On Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichev >>>>>>>> wrote: Hello all! Now I investigate the best approach to get list of files in specified >>>>>>>>>directory and beneath in
Re: [Unicon-group] Walk of file directory
\ 139708 directories in 99867 ms using 1 threads c:\proj>tdir c:\ 439708 directories in 6 ms using 4 threads c:\proj>tdir c:\ 439708 directories in 87650 ms using 4 threads c:\proj>tdir c:\ 139708 directories in 92525 ms using 1 threads c:\proj>tdir c:\ 439708 directories in 95655 ms using 4 threads c:\proj>tdir c:\ 1639708 directories in 66138 ms using 21 threads c:\proj>tdir c:\ 839708 directories in 69307 ms using 8 threads c:\proj>tdir c:\ 439708 directories in 70539 ms using 4 threads c:\proj>tdir c:\ 1639708 directories in 76392 ms using 32 threads On Sun, Jan 11, 2015 at 1:25 PM, David Gamey <david.ga...@rogers.com> wrote:Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev <slogic...@yandex.ru> To: Jafar Al-Gharaibeh <to.ja...@gmail.com> Cc: Unicon group <unicon-group@lists.sourceforge.net> Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" <to.ja...@gmail.com>:Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would be nice if there were a way to tell open() to return files not only based on a pattern, but also on file attribute to allow something like "get me all directories in the current directory", or "get me all read only file". There are a lot of situations where filtering directory names for example is very useful - like this program 3- The program on Rosetta Code is not optimized for speed. You can minimize the number of lists created and put() by careful rewriting of the code. 4- Depending on how deep the directory tree is, there might be a lot of I/O going on. A slow disk might limit how fast you can go regardless of how optimized your code is. I will share results if get around trying any of these options. Cheers,Jafar On Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichev <slogic...@yandex.ru> wrote:Hello all! Now I investigate the best approach to get list of files in specified directory and beneath in Unicon.I found excellent example at rosettacode.org: http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon I reconstructed this one to implement matching of filenames to specified pattern (regular _expression_). My program recursively walks a directory and prints appropriate filenames. The same as dir (ls) does. All working fine except performance. If directory has a lot of subdirs the search may took 10-20 seconds before starting output. Could you provide some advices how to enchance the performance? Some notes how to make and use. Unpack content of udir.zip to your local directory. Define which environment you use in env.icn file - uncomment line "$define _UNIX 1" in the case of UNIX. Nothing to do in the case of Windows.Make udir program:unicon -c futils.icnunicon -c options.icnunicon -c regexp.icnunicon udir.icn Usage: udir -ffor example: udir -f*.icnshall list of icn files in the current dir and all its subdirectories. Best regards,Sergey Logichev-- Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from
Re: [Unicon-group] Walk of file directory
eads > > c:\proj>tdir c:\ 16 > 39708 directories in 76392 ms using 32 threads > > > > On Sun, Jan 11, 2015 at 1:25 PM, David Gamey > wrote: > > Sergey, > > I am responsible for much of the Rosetta code contributions (thanks also > to Steve, Andrew, Matt, Peter, and about 4 others) and this one in > particular dating from 2010. As I recall this was before the > multi-threading versions were widely available. I think multi-threading is > underrepresented in Rosetta/Unicon. > > If you come up with a multi-threading version, we should add it to the > post as an alternative version. If you don't feel comfortable doing this, > post the code and I can add it. > > David > > > -- > *From:* Sergey Logichev > *To:* Jafar Al-Gharaibeh > *Cc:* Unicon group > *Sent:* Sunday, January 11, 2015 1:16 AM > *Subject:* Re: [Unicon-group] Walk of file directory > > Jafar, > > Thank you for a whole bundle of advices and suggestions! Threads are worth > to try. The thought of search by file attributes is very useful too. Your > suggestion about slow I/O partly is right. For UNIX I tried the program on > Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for > Windows it was quite fast HDD. It would be interesting to compare > performance of the program on Windows with classic approach based on Win32 > _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus > implementations of this algorithm. Feel that it will be faster but in which > degree? > > Sergey > > 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : > > > Sergey, > > There are so many things that came to mind when I saw your program. > > 1- At the end of your email, sourceforge ad says "Go Parallel", Which is > not a bad idea for this highly parallel application. > > There is a similar program "wordcount" listed in my dissertation > (available on unicon.org) that go through directories and count words in > every file using threads (Chapter 7, page 107) > > 2- Unicon open() already supports " pattern matching that would greatly (I > believe) speedup your program. For example you can do this: > L := open("*.icn") > >to get a list of all of Unicon source files in the current directory. > > Note: It would be nice if there were a way to tell open() to return > files not only based on a pattern, but also on file attribute to allow > something like "get me all directories in the current directory", or "get > me all read only file". There are a lot of situations where filtering > directory names for example is very useful - like this program > > 3- The program on Rosetta Code is not optimized for speed. You can > minimize the number of lists created and put() by careful rewriting of the > code. > > 4- Depending on how deep the directory tree is, there might be a lot of > I/O going on. A slow disk might limit how fast you can go regardless of how > optimized your code is. > > I will share results if get around trying any of these options. > > Cheers, > Jafar > > > > On Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichev > wrote: > > Hello all! > > Now I investigate the best approach to get list of files in specified > directory and beneath in Unicon. > I found excellent example at rosettacode.org: > http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon > > I reconstructed this one to implement matching of filenames to specified > pattern (regular expression). My program recursively walks a directory and > prints appropriate filenames. The same as dir (ls) does. All working fine > except performance. If directory has a lot of subdirs the search may took > 10-20 seconds before starting output. Could you provide some advices how to > enchance the performance? > > Some notes how to make and use. Unpack content of udir.zip to your local > directory. Define which environment you use in env.icn file - uncomment > line "$define _UNIX 1" in the case of UNIX. Nothing to do in the case of > Windows. > Make udir program: > unicon -c futils.icn > unicon -c options.icn > unicon -c regexp.icn > unicon udir.icn > > Usage: udir -f > for example: udir -f*.icn > shall list of icn files in the current dir and all its subdirectories. > > Best regards, > Sergey Logichev > > > -- > Dive into the World of Parallel Programming! The Go Parallel Website, > sponsored by Intel and developed in partnership with Slashdot Media, is > your > hub for all things parallel software development, from weekly thoug
Re: [Unicon-group] Walk of file directory
Jafar, You've provided very interesting version of walk directory algorithm. Communication with active threads' is a great thing!I have checked your program under Windows 7. I was confused the fact that execution time is negligibly depended on number of concurrent threads. I dug into and discovered that the first operation open(s) takes near ALL execution time! 95% at least. Check it yourself when you slightly edit getdirs():...if ( stat(s).mode ? ="d" ) & ( tm := &time, d := open(s) ) then { if n=1 then write(s," : ",&time-tm)... So, if first open() is so long then all other enhancements have no sense. Please clarify if I am wrong. Best regards,Sergey 22.01.2015, 00:58, "Jafar Al-Gharaibeh" :Here is a slightly tweaked/reformatted version. It now by default auto-detect the number of available cores in the machine and launch twice as many threads. --JafarOn Wed, Jan 21, 2015 at 12:17 PM, Jafar Al-Gharaibeh <to.ja...@gmail.com> wrote:David, I added a threaded solution @ http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon Please review/edit as you see fit. (The source file is attached). Combining recursion with thread might not be the best solution for this problem. If I were to put this in real use I'd go with an iterative approach using master/workers model. Anyway, this is a excellent demonstration on how to use threads!. The key features are: 1- How to create threads, limit their numbers, self-load balanced (new threads are spawned at the time/place where needed. One they are done, they vanish allowing new threads to pop up in new places in the directory structure) 2- pass data and collect results to/from the threads using the new language features. Here is some sample output from my desktop machine (quad-core with mechanical HDD. I will try another machine with an SSD and see if more threads scale better). the first argument to the program is the target directory. The second is the maximum number of concurrent threads to use at any given moment. (soft limit! my counters are "unmutexed", so the actual number might deviate). Note that this is different from the actual number of threads used during the run which is reported at the end. The program can create/destroy threads as needed, but cannot use more than "max" # of threads at any given moment, and again "max" is "soft". :) Cheers,Jafar c:\proj>tdir c:\ 139708 directories in 99867 ms using 1 threads c:\proj>tdir c:\ 439708 directories in 6 ms using 4 threads c:\proj>tdir c:\ 439708 directories in 87650 ms using 4 threads c:\proj>tdir c:\ 139708 directories in 92525 ms using 1 threads c:\proj>tdir c:\ 439708 directories in 95655 ms using 4 threads c:\proj>tdir c:\ 1639708 directories in 66138 ms using 21 threads c:\proj>tdir c:\ 839708 directories in 69307 ms using 8 threads c:\proj>tdir c:\ 439708 directories in 70539 ms using 4 threads c:\proj>tdir c:\ 1639708 directories in 76392 ms using 32 threads On Sun, Jan 11, 2015 at 1:25 PM, David Gamey <david.ga...@rogers.com> wrote:Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev <slogic...@yandex.ru> To: Jafar Al-Gharaibeh <to.ja...@gmail.com> Cc: Unicon group <unicon-group@lists.sourceforge.net> Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" <to.ja...@gmail.com>:Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using thread
Re: [Unicon-group] Walk of file directory
Very cool. I likely wont get to this for a few days. From: Jafar Al-Gharaibeh To: David Gamey Cc: Unicon group Sent: Wednesday, January 21, 2015 4:58 PM Subject: Re: [Unicon-group] Walk of file directory Here is a slightly tweaked/reformatted version. It now by default auto-detect the number of available cores in the machine and launch twice as many threads. --Jafar On Wed, Jan 21, 2015 at 12:17 PM, Jafar Al-Gharaibeh wrote: David, I added a threaded solution @ http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon Please review/edit as you see fit. (The source file is attached). Combining recursion with thread might not be the best solution for this problem. If I were to put this in real use I'd go with an iterative approach using master/workers model. Anyway, this is a excellent demonstration on how to use threads!. The key features are: 1- How to create threads, limit their numbers, self-load balanced (new threads are spawned at the time/place where needed. One they are done, they vanish allowing new threads to pop up in new places in the directory structure) 2- pass data and collect results to/from the threads using the new language features. Here is some sample output from my desktop machine (quad-core with mechanical HDD. I will try another machine with an SSD and see if more threads scale better). the first argument to the program is the target directory. The second is the maximum number of concurrent threads to use at any given moment. (soft limit! my counters are "unmutexed", so the actual number might deviate). Note that this is different from the actual number of threads used during the run which is reported at the end. The program can create/destroy threads as needed, but cannot use more than "max" # of threads at any given moment, and again "max" is "soft". :) Cheers,Jafar c:\proj>tdir c:\ 139708 directories in 99867 ms using 1 threads c:\proj>tdir c:\ 439708 directories in 6 ms using 4 threads c:\proj>tdir c:\ 4 39708 directories in 87650 ms using 4 threads c:\proj>tdir c:\ 139708 directories in 92525 ms using 1 threads c:\proj>tdir c:\ 439708 directories in 95655 ms using 4 threads c:\proj>tdir c:\ 1639708 directories in 66138 ms using 21 threads c:\proj>tdir c:\ 839708 directories in 69307 ms using 8 threads c:\proj>tdir c:\ 439708 directories in 70539 ms using 4 threads c:\proj>tdir c:\ 1639708 directories in 76392 ms using 32 threads On Sun, Jan 11, 2015 at 1:25 PM, David Gamey wrote: Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev To: Jafar Al-Gharaibeh Cc: Unicon group Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would be nice if there were a way to tell open() to return files not only based on a pattern, but also on file attribute to allow something like "get me all directories in the current directory", or "get me all read only file". There are a lot of situations where filtering directory names for example is very useful - like this program 3
Re: [Unicon-group] Walk of file directory
Here is a slightly tweaked/reformatted version. It now by default auto-detect the number of available cores in the machine and launch twice as many threads. --Jafar On Wed, Jan 21, 2015 at 12:17 PM, Jafar Al-Gharaibeh wrote: > David, > > I added a threaded solution @ > http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon >Please review/edit as you see fit. (The source file is attached). > Combining recursion with thread might not be the best solution for this > problem. If I were to put this in real use I'd go with an iterative > approach using master/workers model. Anyway, this is a excellent > demonstration on how to use threads!. The key features are: > >1- How to create threads, limit their numbers, self-load balanced (new > threads are spawned at the time/place where needed. One they are done, > they vanish allowing new threads to pop up in new places in the directory > structure) >2- pass data and collect results to/from the threads using the new > language features. > > > Here is some sample output from my desktop machine (quad-core with > mechanical HDD. I will try another machine with an SSD and see if more > threads scale better). > > the first argument to the program is the target directory. The second is > the maximum number of concurrent threads to use at any given moment. (soft > limit! my counters are "unmutexed", so the actual number might deviate). > Note that this is different from the actual number of threads used during > the run which is reported at the end. The program can create/destroy > threads as needed, but cannot use more than "max" # of threads at any > given moment, and again "max" is "soft". :) > > Cheers, > Jafar > > c:\proj>tdir c:\ 1 > 39708 directories in 99867 ms using 1 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 6 ms using 4 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 87650 ms using 4 threads > > c:\proj>tdir c:\ 1 > 39708 directories in 92525 ms using 1 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 95655 ms using 4 threads > > c:\proj>tdir c:\ 16 > 39708 directories in 66138 ms using 21 threads > > c:\proj>tdir c:\ 8 > 39708 directories in 69307 ms using 8 threads > > c:\proj>tdir c:\ 4 > 39708 directories in 70539 ms using 4 threads > > c:\proj>tdir c:\ 16 > 39708 directories in 76392 ms using 32 threads > > > > On Sun, Jan 11, 2015 at 1:25 PM, David Gamey > wrote: > >> Sergey, >> >> I am responsible for much of the Rosetta code contributions (thanks also >> to Steve, Andrew, Matt, Peter, and about 4 others) and this one in >> particular dating from 2010. As I recall this was before the >> multi-threading versions were widely available. I think multi-threading is >> underrepresented in Rosetta/Unicon. >> >> If you come up with a multi-threading version, we should add it to the >> post as an alternative version. If you don't feel comfortable doing this, >> post the code and I can add it. >> >> David >> >> -- >> *From:* Sergey Logichev >> *To:* Jafar Al-Gharaibeh >> *Cc:* Unicon group >> *Sent:* Sunday, January 11, 2015 1:16 AM >> *Subject:* Re: [Unicon-group] Walk of file directory >> >> Jafar, >> >> Thank you for a whole bundle of advices and suggestions! Threads are >> worth to try. The thought of search by file attributes is very useful too. >> Your suggestion about slow I/O partly is right. For UNIX I tried the >> program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But >> for Windows it was quite fast HDD. It would be interesting to compare >> performance of the program on Windows with classic approach based on Win32 >> _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus >> implementations of this algorithm. Feel that it will be faster but in which >> degree? >> >> Sergey >> >> 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : >> >> >> Sergey, >> >> There are so many things that came to mind when I saw your program. >> >> 1- At the end of your email, sourceforge ad says "Go Parallel", Which is >> not a bad idea for this highly parallel application. >> >> There is a similar program "wordcount" listed in my dissertation >> (available on unicon.org) that go through directories and count words in >> every file using threads (Chapter 7, page 107) >> >> 2- Unicon open() already supports " pattern matching that would great
Re: [Unicon-group] Walk of file directory
David, I added a threaded solution @ http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon Please review/edit as you see fit. (The source file is attached). Combining recursion with thread might not be the best solution for this problem. If I were to put this in real use I'd go with an iterative approach using master/workers model. Anyway, this is a excellent demonstration on how to use threads!. The key features are: 1- How to create threads, limit their numbers, self-load balanced (new threads are spawned at the time/place where needed. One they are done, they vanish allowing new threads to pop up in new places in the directory structure) 2- pass data and collect results to/from the threads using the new language features. Here is some sample output from my desktop machine (quad-core with mechanical HDD. I will try another machine with an SSD and see if more threads scale better). the first argument to the program is the target directory. The second is the maximum number of concurrent threads to use at any given moment. (soft limit! my counters are "unmutexed", so the actual number might deviate). Note that this is different from the actual number of threads used during the run which is reported at the end. The program can create/destroy threads as needed, but cannot use more than "max" # of threads at any given moment, and again "max" is "soft". :) Cheers, Jafar c:\proj>tdir c:\ 1 39708 directories in 99867 ms using 1 threads c:\proj>tdir c:\ 4 39708 directories in 6 ms using 4 threads c:\proj>tdir c:\ 4 39708 directories in 87650 ms using 4 threads c:\proj>tdir c:\ 1 39708 directories in 92525 ms using 1 threads c:\proj>tdir c:\ 4 39708 directories in 95655 ms using 4 threads c:\proj>tdir c:\ 16 39708 directories in 66138 ms using 21 threads c:\proj>tdir c:\ 8 39708 directories in 69307 ms using 8 threads c:\proj>tdir c:\ 4 39708 directories in 70539 ms using 4 threads c:\proj>tdir c:\ 16 39708 directories in 76392 ms using 32 threads On Sun, Jan 11, 2015 at 1:25 PM, David Gamey wrote: > Sergey, > > I am responsible for much of the Rosetta code contributions (thanks also > to Steve, Andrew, Matt, Peter, and about 4 others) and this one in > particular dating from 2010. As I recall this was before the > multi-threading versions were widely available. I think multi-threading is > underrepresented in Rosetta/Unicon. > > If you come up with a multi-threading version, we should add it to the > post as an alternative version. If you don't feel comfortable doing this, > post the code and I can add it. > > David > > -- > *From:* Sergey Logichev > *To:* Jafar Al-Gharaibeh > *Cc:* Unicon group > *Sent:* Sunday, January 11, 2015 1:16 AM > *Subject:* Re: [Unicon-group] Walk of file directory > > Jafar, > > Thank you for a whole bundle of advices and suggestions! Threads are worth > to try. The thought of search by file attributes is very useful too. Your > suggestion about slow I/O partly is right. For UNIX I tried the program on > Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for > Windows it was quite fast HDD. It would be interesting to compare > performance of the program on Windows with classic approach based on Win32 > _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus > implementations of this algorithm. Feel that it will be faster but in which > degree? > > Sergey > > 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : > > > Sergey, > > There are so many things that came to mind when I saw your program. > > 1- At the end of your email, sourceforge ad says "Go Parallel", Which is > not a bad idea for this highly parallel application. > > There is a similar program "wordcount" listed in my dissertation > (available on unicon.org) that go through directories and count words in > every file using threads (Chapter 7, page 107) > > 2- Unicon open() already supports " pattern matching that would greatly (I > believe) speedup your program. For example you can do this: > L := open("*.icn") > >to get a list of all of Unicon source files in the current directory. > > Note: It would be nice if there were a way to tell open() to return > files not only based on a pattern, but also on file attribute to allow > something like "get me all directories in the current directory", or "get > me all read only file". There are a lot of situations where filtering > directory names for example is very useful - like this program > > 3- The program on Rosetta Code is not optimized for speed. You can > minimize the number of lists created and put() by careful rewriting of
Re: [Unicon-group] Walk of file directory
Jafar, I like your suggestion about printing thread. It would really improve usability of this procedure. Why we need to wait till the end of search? For user it's looking as hanging.As for extending functionality of open() - to filter only directories or files with some specific attributes - I think it can be done with help of passing to open some structure as input variable. It may be a record, which specifies filename mask, start directory, flag to search recursively and so on. How do you think about it? The same approach is used in the module, which I use for file search in Lazarus (Delphi) applications (attached).It would be interesting to seek an example of recursively file search in Unicon book as Clinton mentioned. Unfortunately, I didn't see such examples in books which I read. Best regards,Sergey 13.01.2015, 00:57, "Jafar Al-Gharaibeh" :I have to add that, in his initial description, Sergey pointed out the search takes 10-20 seconds before the output can be displayed. He was looking for ways to improve performance. If he cares about "performance" in terms of "latency" then a second thread that starts printing results as soon as they are available might be all he needs! :) Cheers,Jafar On Mon, Jan 12, 2015 at 3:48 PM, Jafar Al-Gharaibeh <to.ja...@gmail.com> wrote:Well, stat() works fine, it is the one used by the example code at RosettaCode which Sergey used. We are just spoiled by the the ability of open() to filter based on names with wild cards. To get all of Unicon source files in a given directory I suggested open("*.icn") But if you have to do it recursively for sub-directories you still have to loop through all files (traversing .icn again) to get directory names, which kind of defeats the purpose of the initial open("*.icn") in this case. That is why open("directories only") would be useful here. I have seen a lot of situation where such feature is useful. Of course it can all be done using stat(), but this whole discussion started on the basis of performance where the current implementation of the problem was not "fast enough". open()'s built-in filtering might help a little but it doesn't provide everything needed to completely rely on it for filtering. Cheers,Jafar On Mon, Jan 12, 2015 at 2:22 PM, Jeffery, Clint (jeffe...@uidaho.edu) <jeffe...@uidaho.edu> wrote:Hi, I am all for implementing a library procedure, or class, to do this as part of the Unicon distribution, if one is not there already. I am wondering why I haven't noticed someone chime in with the stat() function, it is the building block one uses to read file attributes, determine if something is a directory, etc. I do know the Unicon book has an example program that walks through directories recursively. Cheers,ClintFrom: Sergey Logichev <slogic...@yandex.ru> Sent: Monday, January 12, 2015 3:05 AM To: David Gamey; Jafar Al-Gharaibeh Cc: Unicon group Subject: Re: [Unicon-group] Walk of file directory David, Ok, I will try to implement it myself at first. If I will be in trouble I ask you help, sure. In any case I will inform Unicon group about it.It would be great to implement file search directly in Unicon. As Jafar mentioned, we already may do it within current directory with help of open() function. For example, open("*.icn") does filtering of icn files in the directory. Next step would be logical to do search recursively in all subdirectories. For example, something like open("rootdir/*.icn","recursively"). Sergey 11.01.2015, 22:25, "David Gamey" <david.ga...@rogers.com>:Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev <slogic...@yandex.ru> To: Jafar Al-Gharaibeh <to.ja...@gmail.com> Cc: Unicon group <unicon-group@lists.sourceforge.net> Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm.
Re: [Unicon-group] Walk of file directory
I have to add that, in his initial description, Sergey pointed out the search takes 10-20 seconds before the output can be displayed. He was looking for ways to improve performance. If he cares about "performance" in terms of "latency" then a second thread that starts printing results as soon as they are available might be all he needs! :) Cheers, Jafar On Mon, Jan 12, 2015 at 3:48 PM, Jafar Al-Gharaibeh wrote: > Well, stat() works fine, it is the one used by the example code at > RosettaCode which Sergey used. We are just spoiled by the the ability of > open() to filter based on names with wild cards. To get all of Unicon > source files in a given directory I suggested > > open("*.icn") > > But if you have to do it recursively for sub-directories you still have > to loop through all files (traversing .icn again) to get directory names, > which kind of defeats the purpose of the initial open("*.icn") in this > case. That is why open("directories only") would be useful here. I have > seen a lot of situation where such feature is useful. Of course it can all > be done using stat(), but this whole discussion started on the basis of > performance where the current implementation of the problem was not "fast > enough". open()'s built-in filtering might help a little but it doesn't > provide everything needed to completely rely on it for filtering. > > Cheers, > Jafar > > > On Mon, Jan 12, 2015 at 2:22 PM, Jeffery, Clint (jeffe...@uidaho.edu) < > jeffe...@uidaho.edu> wrote: > >> Hi, >> >> >> I am all for implementing a library procedure, or class, to do this as >> part of the Unicon distribution, if one is not there already. I am >> wondering why I haven't noticed someone chime in with the stat() function, >> it is the building block one uses to read file attributes, determine if >> something is a directory, etc. I do know the Unicon book has an example >> program that walks through directories recursively. >> >> >> Cheers, >> >> Clint >> -- >> *From:* Sergey Logichev >> *Sent:* Monday, January 12, 2015 3:05 AM >> *To:* David Gamey; Jafar Al-Gharaibeh >> *Cc:* Unicon group >> >> *Subject:* Re: [Unicon-group] Walk of file directory >> >> David, >> >> Ok, I will try to implement it myself at first. If I will be in trouble I >> ask you help, sure. In any case I will inform Unicon group about it. >> It would be great to implement file search directly in Unicon. As Jafar >> mentioned, we already may do it within current directory with help of >> open() function. For example, open("*.icn") does filtering of icn files in >> the directory. Next step would be logical to do search recursively in all >> subdirectories. For example, something like >> open("rootdir/*.icn","recursively"). >> >> Sergey >> >> 11.01.2015, 22:25, "David Gamey" : >> >> Sergey, >> >> I am responsible for much of the Rosetta code contributions (thanks also >> to Steve, Andrew, Matt, Peter, and about 4 others) and this one in >> particular dating from 2010. As I recall this was before the >> multi-threading versions were widely available. I think multi-threading is >> underrepresented in Rosetta/Unicon. >> >> If you come up with a multi-threading version, we should add it to the >> post as an alternative version. If you don't feel comfortable doing this, >> post the code and I can add it. >> >> David >> >> >> -- >> *From:* Sergey Logichev >> *To:* Jafar Al-Gharaibeh >> *Cc:* Unicon group >> *Sent:* Sunday, January 11, 2015 1:16 AM >> *Subject:* Re: [Unicon-group] Walk of file directory >> >> Jafar, >> >> Thank you for a whole bundle of advices and suggestions! Threads are >> worth to try. The thought of search by file attributes is very useful too. >> Your suggestion about slow I/O partly is right. For UNIX I tried the >> program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But >> for Windows it was quite fast HDD. It would be interesting to compare >> performance of the program on Windows with classic approach based on Win32 >> _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus >> implementations of this algorithm. Feel that it will be faster but in which >> degree? >> >> Sergey >> >> 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : >> >> >> Sergey, >> >> There are
Re: [Unicon-group] Walk of file directory
Well, stat() works fine, it is the one used by the example code at RosettaCode which Sergey used. We are just spoiled by the the ability of open() to filter based on names with wild cards. To get all of Unicon source files in a given directory I suggested open("*.icn") But if you have to do it recursively for sub-directories you still have to loop through all files (traversing .icn again) to get directory names, which kind of defeats the purpose of the initial open("*.icn") in this case. That is why open("directories only") would be useful here. I have seen a lot of situation where such feature is useful. Of course it can all be done using stat(), but this whole discussion started on the basis of performance where the current implementation of the problem was not "fast enough". open()'s built-in filtering might help a little but it doesn't provide everything needed to completely rely on it for filtering. Cheers, Jafar On Mon, Jan 12, 2015 at 2:22 PM, Jeffery, Clint (jeffe...@uidaho.edu) < jeffe...@uidaho.edu> wrote: > Hi, > > > I am all for implementing a library procedure, or class, to do this as > part of the Unicon distribution, if one is not there already. I am > wondering why I haven't noticed someone chime in with the stat() function, > it is the building block one uses to read file attributes, determine if > something is a directory, etc. I do know the Unicon book has an example > program that walks through directories recursively. > > > Cheers, > > Clint > -- > *From:* Sergey Logichev > *Sent:* Monday, January 12, 2015 3:05 AM > *To:* David Gamey; Jafar Al-Gharaibeh > *Cc:* Unicon group > > *Subject:* Re: [Unicon-group] Walk of file directory > > David, > > Ok, I will try to implement it myself at first. If I will be in trouble I > ask you help, sure. In any case I will inform Unicon group about it. > It would be great to implement file search directly in Unicon. As Jafar > mentioned, we already may do it within current directory with help of > open() function. For example, open("*.icn") does filtering of icn files in > the directory. Next step would be logical to do search recursively in all > subdirectories. For example, something like > open("rootdir/*.icn","recursively"). > > Sergey > > 11.01.2015, 22:25, "David Gamey" : > > Sergey, > > I am responsible for much of the Rosetta code contributions (thanks also > to Steve, Andrew, Matt, Peter, and about 4 others) and this one in > particular dating from 2010. As I recall this was before the > multi-threading versions were widely available. I think multi-threading is > underrepresented in Rosetta/Unicon. > > If you come up with a multi-threading version, we should add it to the > post as an alternative version. If you don't feel comfortable doing this, > post the code and I can add it. > > David > > > -- > *From:* Sergey Logichev > *To:* Jafar Al-Gharaibeh > *Cc:* Unicon group > *Sent:* Sunday, January 11, 2015 1:16 AM > *Subject:* Re: [Unicon-group] Walk of file directory > > Jafar, > > Thank you for a whole bundle of advices and suggestions! Threads are worth > to try. The thought of search by file attributes is very useful too. Your > suggestion about slow I/O partly is right. For UNIX I tried the program on > Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for > Windows it was quite fast HDD. It would be interesting to compare > performance of the program on Windows with classic approach based on Win32 > _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus > implementations of this algorithm. Feel that it will be faster but in which > degree? > > Sergey > > 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : > > > Sergey, > > There are so many things that came to mind when I saw your program. > > 1- At the end of your email, sourceforge ad says "Go Parallel", Which is > not a bad idea for this highly parallel application. > > There is a similar program "wordcount" listed in my dissertation > (available on unicon.org) that go through directories and count words in > every file using threads (Chapter 7, page 107) > > 2- Unicon open() already supports " pattern matching that would greatly (I > believe) speedup your program. For example you can do this: > L := open("*.icn") > >to get a list of all of Unicon source files in the current directory. > > Note: It would be nice if there were a way to tell open() to return > files not only based on a pattern, but also on file attribute to allow > some
Re: [Unicon-group] Walk of file directory
Hi, I am all for implementing a library procedure, or class, to do this as part of the Unicon distribution, if one is not there already. I am wondering why I haven't noticed someone chime in with the stat() function, it is the building block one uses to read file attributes, determine if something is a directory, etc. I do know the Unicon book has an example program that walks through directories recursively. Cheers, Clint From: Sergey Logichev Sent: Monday, January 12, 2015 3:05 AM To: David Gamey; Jafar Al-Gharaibeh Cc: Unicon group Subject: Re: [Unicon-group] Walk of file directory David, Ok, I will try to implement it myself at first. If I will be in trouble I ask you help, sure. In any case I will inform Unicon group about it. It would be great to implement file search directly in Unicon. As Jafar mentioned, we already may do it within current directory with help of open() function. For example, open("*.icn") does filtering of icn files in the directory. Next step would be logical to do search recursively in all subdirectories. For example, something like open("rootdir/*.icn","recursively"). Sergey 11.01.2015, 22:25, "David Gamey" : Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev mailto:slogic...@yandex.ru>> To: Jafar Al-Gharaibeh mailto:to.ja...@gmail.com>> Cc: Unicon group mailto:unicon-group@lists.sourceforge.net>> Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" mailto:to.ja...@gmail.com>>: Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org<http://unicon.org/>) that go through directories and count words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would be nice if there were a way to tell open() to return files not only based on a pattern, but also on file attribute to allow something like "get me all directories in the current directory", or "get me all read only file". There are a lot of situations where filtering directory names for example is very useful - like this program 3- The program on Rosetta Code is not optimized for speed. You can minimize the number of lists created and put() by careful rewriting of the code. 4- Depending on how deep the directory tree is, there might be a lot of I/O going on. A slow disk might limit how fast you can go regardless of how optimized your code is. I will share results if get around trying any of these options. Cheers, Jafar On Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichev mailto:slogic...@yandex.ru>> wrote: Hello all! Now I investigate the best approach to get list of files in specified directory and beneath in Unicon. I found excellent example at rosettacode.org<http://rosettacode.org/>: http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon I reconstructed this one to implement matching of filenames to specified pattern (regular expression). My program recursively walks a directory and prints appropriate filenames. The same as dir (ls) does. All working fine except performance. If directory has a lot of subdirs the search may took 10-20 seconds before starting output. Could you provide some advices how to
Re: [Unicon-group] Walk of file directory
David, Ok, I will try to implement it myself at first. If I will be in trouble I ask you help, sure. In any case I will inform Unicon group about it.It would be great to implement file search directly in Unicon. As Jafar mentioned, we already may do it within current directory with help of open() function. For example, open("*.icn") does filtering of icn files in the directory. Next step would be logical to do search recursively in all subdirectories. For example, something like open("rootdir/*.icn","recursively"). Sergey 11.01.2015, 22:25, "David Gamey" :Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev <slogic...@yandex.ru> To: Jafar Al-Gharaibeh <to.ja...@gmail.com> Cc: Unicon group <unicon-group@lists.sourceforge.net> Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" <to.ja...@gmail.com>:Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would be nice if there were a way to tell open() to return files not only based on a pattern, but also on file attribute to allow something like "get me all directories in the current directory", or "get me all read only file". There are a lot of situations where filtering directory names for example is very useful - like this program 3- The program on Rosetta Code is not optimized for speed. You can minimize the number of lists created and put() by careful rewriting of the code. 4- Depending on how deep the directory tree is, there might be a lot of I/O going on. A slow disk might limit how fast you can go regardless of how optimized your code is. I will share results if get around trying any of these options. Cheers,Jafar On Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichev <slogic...@yandex.ru> wrote:Hello all! Now I investigate the best approach to get list of files in specified directory and beneath in Unicon.I found excellent example at rosettacode.org: http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon I reconstructed this one to implement matching of filenames to specified pattern (regular _expression_). My program recursively walks a directory and prints appropriate filenames. The same as dir (ls) does. All working fine except performance. If directory has a lot of subdirs the search may took 10-20 seconds before starting output. Could you provide some advices how to enchance the performance? Some notes how to make and use. Unpack content of udir.zip to your local directory. Define which environment you use in env.icn file - uncomment line "$define _UNIX 1" in the case of UNIX. Nothing to do in the case of Windows.Make udir program:unicon -c futils.icnunicon -c options.icnunicon -c regexp.icnunicon udir.icn Usage: udir -ffor example: udir -f*.icnshall list of icn files in the current dir and all its subdirectories. Best regards,Sergey Logichev-- Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, t
Re: [Unicon-group] Walk of file directory
Sergey, I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon. If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it. David From: Sergey Logichev To: Jafar Al-Gharaibeh Cc: Unicon group Sent: Sunday, January 11, 2015 1:16 AM Subject: Re: [Unicon-group] Walk of file directory Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" : Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would be nice if there were a way to tell open() to return files not only based on a pattern, but also on file attribute to allow something like "get me all directories in the current directory", or "get me all read only file". There are a lot of situations where filtering directory names for example is very useful - like this program 3- The program on Rosetta Code is not optimized for speed. You can minimize the number of lists created and put() by careful rewriting of the code. 4- Depending on how deep the directory tree is, there might be a lot of I/O going on. A slow disk might limit how fast you can go regardless of how optimized your code is. I will share results if get around trying any of these options. Cheers,Jafar On Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichev wrote: Hello all! Now I investigate the best approach to get list of files in specified directory and beneath in Unicon.I found excellent example at rosettacode.org: http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon I reconstructed this one to implement matching of filenames to specified pattern (regular expression). My program recursively walks a directory and prints appropriate filenames. The same as dir (ls) does. All working fine except performance. If directory has a lot of subdirs the search may took 10-20 seconds before starting output. Could you provide some advices how to enchance the performance? Some notes how to make and use. Unpack content of udir.zip to your local directory. Define which environment you use in env.icn file - uncomment line "$define _UNIX 1" in the case of UNIX. Nothing to do in the case of Windows.Make udir program:unicon -c futils.icnunicon -c options.icnunicon -c regexp.icnunicon udir.icn Usage: udir -ffor example: udir -f*.icnshall list of icn files in the current dir and all its subdirectories. Best regards,Sergey Logichev -- Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net ___ Unicon-group mailing list Unicon-group@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/unicon-group -- Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. htt
Re: [Unicon-group] Walk of file directory
Jafar, Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree? Sergey 10.01.2015, 21:50, "Jafar Al-Gharaibeh" :Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would be nice if there were a way to tell open() to return files not only based on a pattern, but also on file attribute to allow something like "get me all directories in the current directory", or "get me all read only file". There are a lot of situations where filtering directory names for example is very useful - like this program 3- The program on Rosetta Code is not optimized for speed. You can minimize the number of lists created and put() by careful rewriting of the code. 4- Depending on how deep the directory tree is, there might be a lot of I/O going on. A slow disk might limit how fast you can go regardless of how optimized your code is. I will share results if get around trying any of these options. Cheers,Jafar On Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichevwrote:Hello all! Now I investigate the best approach to get list of files in specified directory and beneath in Unicon.I found excellent example at rosettacode.org: http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon I reconstructed this one to implement matching of filenames to specified pattern (regular _expression_). My program recursively walks a directory and prints appropriate filenames. The same as dir (ls) does. All working fine except performance. If directory has a lot of subdirs the search may took 10-20 seconds before starting output. Could you provide some advices how to enchance the performance? Some notes how to make and use. Unpack content of udir.zip to your local directory. Define which environment you use in env.icn file - uncomment line "$define _UNIX 1" in the case of UNIX. Nothing to do in the case of Windows.Make udir program:unicon -c futils.icnunicon -c options.icnunicon -c regexp.icnunicon udir.icn Usage: udir -ffor example: udir -f*.icnshall list of icn files in the current dir and all its subdirectories. Best regards,Sergey Logichev-- Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net___ Unicon-group mailing list Unicon-group@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/unicon-group -- Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net___ Unicon-group mailing list Unicon-group@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/unicon-group
Re: [Unicon-group] Walk of file directory
Sergey, There are so many things that came to mind when I saw your program. 1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application. There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using threads (Chapter 7, page 107) 2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this: L := open("*.icn") to get a list of all of Unicon source files in the current directory. Note: It would be nice if there were a way to tell open() to return files not only based on a pattern, but also on file attribute to allow something like "get me all directories in the current directory", or "get me all read only file". There are a lot of situations where filtering directory names for example is very useful - like this program 3- The program on Rosetta Code is not optimized for speed. You can minimize the number of lists created and put() by careful rewriting of the code. 4- Depending on how deep the directory tree is, there might be a lot of I/O going on. A slow disk might limit how fast you can go regardless of how optimized your code is. I will share results if get around trying any of these options. Cheers, Jafar On Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichev wrote: > Hello all! > > Now I investigate the best approach to get list of files in specified > directory and beneath in Unicon. > I found excellent example at rosettacode.org: > http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_Unicon > > I reconstructed this one to implement matching of filenames to specified > pattern (regular expression). My program recursively walks a directory and > prints appropriate filenames. The same as dir (ls) does. All working fine > except performance. If directory has a lot of subdirs the search may took > 10-20 seconds before starting output. Could you provide some advices how to > enchance the performance? > > Some notes how to make and use. Unpack content of udir.zip to your local > directory. Define which environment you use in env.icn file - uncomment > line "$define _UNIX 1" in the case of UNIX. Nothing to do in the case of > Windows. > Make udir program: > unicon -c futils.icn > unicon -c options.icn > unicon -c regexp.icn > unicon udir.icn > > Usage: udir -f > for example: udir -f*.icn > shall list of icn files in the current dir and all its subdirectories. > > Best regards, > Sergey Logichev > > > -- > Dive into the World of Parallel Programming! The Go Parallel Website, > sponsored by Intel and developed in partnership with Slashdot Media, is > your > hub for all things parallel software development, from weekly thought > leadership blogs to news, videos, case studies, tutorials and more. Take a > look and join the conversation now. http://goparallel.sourceforge.net > ___ > Unicon-group mailing list > Unicon-group@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/unicon-group > > -- Dive into the World of Parallel Programming! The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net___ Unicon-group mailing list Unicon-group@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/unicon-group