Jafar,
I like your suggestion about printing thread. It would really improve usability of this procedure. Why we need to wait till the end of search? For user it's looking as hanging.
As for extending functionality of open() - to filter only directories or files with some specific attributes - I think it can be done with help of passing to open some structure as input variable. It may be a record, which specifies filename mask, start directory, flag to search recursively and so on. How do you think about it? The same approach is used in the module, which I use for file search in Lazarus (Delphi) applications (attached).
It would be interesting to seek an example of recursively file search in Unicon book as Clinton mentioned. Unfortunately, I didn't see such examples in books which I read.
Best regards,
Sergey
13.01.2015, 00:57, "Jafar Al-Gharaibeh" <[email protected]>:
I have to add that, in his initial description, Sergey pointed out the search takes 10-20 seconds before the output can be displayed. He was looking for ways to improve performance. If he cares about "performance" in terms of "latency" then a second thread that starts printing results as soon as they are available might be all he needs! :)Cheers,JafarOn Mon, Jan 12, 2015 at 3:48 PM, Jafar Al-Gharaibeh <[email protected]> wrote:Well, stat() works fine, it is the one used by the example code at RosettaCode which Sergey used. We are just spoiled by the the ability of open() to filter based on names with wild cards. To get all of Unicon source files in a given directory I suggestedopen("*.icn")But if you have to do it recursively for sub-directories you still have to loop through all files (traversing .icn again) to get directory names, which kind of defeats the purpose of the initial open("*.icn") in this case. That is why open("directories only") would be useful here. I have seen a lot of situation where such feature is useful. Of course it can all be done using stat(), but this whole discussion started on the basis of performance where the current implementation of the problem was not "fast enough". open()'s built-in filtering might help a little but it doesn't provide everything needed to completely rely on it for filtering.Cheers,JafarOn Mon, Jan 12, 2015 at 2:22 PM, Jeffery, Clint ([email protected]) <[email protected]> wrote:Hi,
I am all for implementing a library procedure, or class, to do this as part of the Unicon distribution, if one is not there already. I am wondering why I haven't noticed someone chime in with the stat() function, it is the building block one uses to read file attributes, determine if something is a directory, etc. I do know the Unicon book has an example program that walks through directories recursively.
Cheers,
Clint
From: Sergey Logichev <[email protected]>
Sent: Monday, January 12, 2015 3:05 AM
To: David Gamey; Jafar Al-Gharaibeh
Cc: Unicon group
Subject: Re: [Unicon-group] Walk of file directoryDavid,Ok, I will try to implement it myself at first. If I will be in trouble I ask you help, sure. In any case I will inform Unicon group about it.It would be great to implement file search directly in Unicon. As Jafar mentioned, we already may do it within current directory with help of open() function. For example, open("*.icn") does filtering of icn files in the directory. Next step would be logical to do search recursively in all subdirectories. For example, something like open("rootdir/*.icn","recursively").Sergey11.01.2015, 22:25, "David Gamey" <[email protected]>:Sergey,I am responsible for much of the Rosetta code contributions (thanks also to Steve, Andrew, Matt, Peter, and about 4 others) and this one in particular dating from 2010. As I recall this was before the multi-threading versions were widely available. I think multi-threading is underrepresented in Rosetta/Unicon.If you come up with a multi-threading version, we should add it to the post as an alternative version. If you don't feel comfortable doing this, post the code and I can add it.David
From: Sergey Logichev <[email protected]>
To: Jafar Al-Gharaibeh <[email protected]>
Cc: Unicon group <[email protected]>
Sent: Sunday, January 11, 2015 1:16 AM
Subject: Re: [Unicon-group] Walk of file directory
Jafar,Thank you for a whole bundle of advices and suggestions! Threads are worth to try. The thought of search by file attributes is very useful too. Your suggestion about slow I/O partly is right. For UNIX I tried the program on Raspberry Pi with 6 Class microSD as HDD (it's slow, agree). But for Windows it was quite fast HDD. It would be interesting to compare performance of the program on Windows with classic approach based on Win32 _FINDFIRST, _FINDNEXT functions. I have threaded Delphi/Lazarus implementations of this algorithm. Feel that it will be faster but in which degree?Sergey10.01.2015, 21:50, "Jafar Al-Gharaibeh" <[email protected]>:
Sergey,There are so many things that came to mind when I saw your program.1- At the end of your email, sourceforge ad says "Go Parallel", Which is not a bad idea for this highly parallel application.There is a similar program "wordcount" listed in my dissertation (available on unicon.org) that go through directories and count words in every file using threads (Chapter 7, page 107)2- Unicon open() already supports " pattern matching that would greatly (I believe) speedup your program. For example you can do this:L := open("*.icn")to get a list of all of Unicon source files in the current directory.Note: It would be nice if there were a way to tell open() to return files not only based on a pattern, but also on file attribute to allow something like "get me all directories in the current directory", or "get me all read only file". There are a lot of situations where filtering directory names for example is very useful - like this program3- The program on Rosetta Code is not optimized for speed. You can minimize the number of lists created and put() by careful rewriting of the code.4- Depending on how deep the directory tree is, there might be a lot of I/O going on. A slow disk might limit how fast you can go regardless of how optimized your code is.I will share results if get around trying any of these options.Cheers,JafarOn Sat, Jan 10, 2015 at 5:51 AM, Sergey Logichev <[email protected]> wrote:Hello all!Now I investigate the best approach to get list of files in specified directory and beneath in Unicon.I found excellent example at rosettacode.org: http://rosettacode.org/wiki/Walk_a_directory/Recursively#Icon_and_UniconI reconstructed this one to implement matching of filenames to specified pattern (regular _expression_). My program recursively walks a directory and prints appropriate filenames. The same as dir (ls) does. All working fine except performance. If directory has a lot of subdirs the search may took 10-20 seconds before starting output. Could you provide some advices how to enchance the performance?Some notes how to make and use. Unpack content of udir.zip to your local directory. Define which environment you use in env.icn file - uncomment line "$define _UNIX 1" in the case of UNIX. Nothing to do in the case of Windows.Make udir program:unicon -c futils.icnunicon -c options.icnunicon -c regexp.icnunicon udir.icnUsage: udir -f<filemask>for example: udir -f*.icnshall list of icn files in the current dir and all its subdirectories.Best regards,Sergey Logichev
------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Unicon-group mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/unicon-group------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net_______________________________________________
Unicon-group mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/unicon-group
{
----------------------------------------------------------
MAS-CompMaker was used to generate this code
MAS-CompMaker, 2000-2001® MASoft, Mats Asplund
----------------------------------------------------------
Component Name: TmFileScan
Author: MASoft, Mats Asplund
Creation: 2002-10-10
Version: 3.2
Description: A component that scans a whole device or part of it,
and outputs the fullpath filenames, corresponding to
searchfilter settings. Multiple filters can be set.
Credit:
E-mail: [email protected]
Site: http://hem.passagen.se/matasp
Legal issues: All rights reserved 2002® by MASoft, Mats Asplund
Usage: This software is provided 'as-is', without any express or
implied warranty. In no event will the author be held liable
for any damages arising from the use of this software.
Permission is granted to anyone to use this software for any
purpose, including commercial applications, and to alter it
and redistribute it freely, subject to the following
restrictions:
1. The origin of this software must not be misrepresented,
you must not claim that you wrote the original software.
If you use this software in a product, an acknowledgment
in the product documentation would be appreciated but is
not required.
2. Altered source versions must be plainly marked as such, and
must not be misrepresented as being the original software.
3. This notice may not be removed or altered from any source
distribution.
4. If you decide to use this software in any of your applications.
Send me an EMail and tell me about it.
Quick Reference:
TmFileScan inherits from TComponent.
SearchResult: StringList with the resulting files.
Key-Properties:
Paths: Which filepaths to scan (TStringList). Multiple paths can be
used.
Filters: The filesearch roules. Use wildcard. Multiple filters can be
used.
Ex.
mFileScan1.Paths.Clear;
mFileScan1.Filters.Clear;
mFileScan1.Paths.Add('c:\');
mFileScan1.Paths.Add('c:\tmp');
mFileScan1.SubDirs:= false;
mFileScan1.Filters.Add('a*.exe');
mFileScan1.Filters.Add('*a*.txt');
mFileScan1.Start;
...will search on root-dir and c:\tmp for exe-files
with filenames beginning with the letter 'a' and all
text-files with an 'a' in the filename.
SubDirs: If true, subdirectories will be scanned.
Key-Events:
OnFileFound: Use this event if you want the resulting filelist to be
updated each time a file is found.
OnReady: Use this event if you want the whole filelist to be
updated when the scan is ready.
Key-Methods:
Start: Starts the filescan.
Stop: Aborts the filescan.
Pause: Pauses the filescan. Searchthread is suspended.
Resume: Resumes the filescan. Searchthread is resumed.
New in version 3.2:
Stop method.
Aborted in the OnReady event.
--------------------------------------------------------------------------------
}
unit mFileScan;
{$MODE Delphi}
interface
uses
SysUtils, Classes, fileUtil;
//Controls, Forms, Graphics, Dialogs, StdCtrls, ExtCtrls, Messages,
type
TmFileScan = class;
TOnFileFoundEvent =
procedure(Sender: TObject; FileName: string) of object;
TOnReadyEvent =
procedure(Sender: TObject; Aborted: boolean) of object;
// procedure(Sender: TObject; Aborted: boolean; SearchResult: TStringList) of object;
TSearchThread = class(TThread)
private
ffOwner: TmFileScan;
ffSubDir: Boolean;
ffFilters: TStrings;
ffPaths: TStrings;
ffFileFound: TFileName;
ffOnFileFound: TOnFileFoundEvent;
ffOnReady: TOnReadyEvent;
// ffList: TStringList;
ffAbort: boolean;
procedure FileFound;
procedure Ready;
protected
procedure Execute; override;
public
constructor Create(Owner: TmFileScan; SubDir, Started: Boolean;
FilePaths, Filter: TStrings; fOnFileFound: TOnFileFoundEvent;
fOnReady: TOnReadyEvent);
end;
TmFileScan = class(TComponent)
private
SearchThread: TSearchThread;
fOnFileFound: TOnFileFoundEvent;
fOnReady: TOnReadyEvent;
fFilters: TStrings;
fSubDir: Boolean;
fAbout: string;
fPaths: TStrings;
fPaused: Boolean;
fStarted: Boolean;
procedure SetAbout(Value: string);
procedure SetFilters(const Value: TStrings);
procedure SetPaths(const Value: TStrings);
{ Private declarations }
public
SearchResult: TStringList;
constructor Create(AOwner: TComponent); override;
destructor Destroy; override;
procedure Start;
procedure Stop;
procedure Pause;
procedure Resume;
property Started: Boolean read fStarted;
{ Private declarations }
published
property Paths: TStrings read fPaths write SetPaths;
property Filters: TStrings read fFilters write SetFilters;
property SubDirs: Boolean read fSubDir write fSubDir;
property OnFileFound: TOnFileFoundEvent read fOnFileFound write
fOnFileFound;
property OnReady: TOnReadyEvent read fOnReady write fOnReady;
property About: string read fAbout write SetAbout;
{ Published declarations }
end;
procedure Register;
implementation
constructor TmFileScan.Create(AOwner: TComponent);
begin
inherited Create(AOwner);
fFilters:= TStringList.Create;
fPaths:= TStringList.Create;
// SearchResult:= TStringList.Create;
fSubDir:= true;
fPaused:= false;
fAbout:= 'Version 3.1, 2002® MASoft';
end;
destructor TmFileScan.Destroy;
begin
fFilters.Free;
fPaths.Free;
// SearchResult.Free;
inherited Destroy;
end;
procedure TmFileScan.SetAbout(Value: string);
begin
Exit;
end;
procedure TmFileScan.SetFilters(const Value: TStrings);
begin
fFilters.Assign(Value);
end;
procedure TmFileScan.Start;
begin
if not fStarted then
begin
// SearchResult.Clear;
SearchThread:=
TSearchThread.Create(Self, fSubDir, fStarted, fPaths, fFilters,
fOnFileFound, fOnReady);
end;
fStarted:= true;
end;
procedure TmFileScan.Stop;
begin
if fPaused then
begin
fPaused:= false;
{ SearchThread.Resume; }
SearchThread.Suspended := true;
end;
if fStarted then
SearchThread.ffAbort:= true;
end;
procedure TmFileScan.Pause;
begin
if not fPaused then
begin
fPaused:= true;
SearchThread.Suspended := true;
end;
end;
procedure TmFileScan.Resume;
begin
if fPaused then
begin
fPaused:= false;
{ SearchThread.Resume; }
SearchThread.Suspended := false;
end;
end;
procedure TmFileScan.SetPaths(const Value: TStrings);
begin
fPaths.Assign(Value);
end;
{ TSearchThread }
constructor TSearchThread.Create(Owner: TmFileScan; SubDir, Started: Boolean;
FilePaths, Filter: TStrings; fOnFileFound: TOnFileFoundEvent;
fOnReady: TOnReadyEvent);
begin
inherited Create(true);
ffOwner:= Owner;
ffPaths:= TStringList.Create;
ffFilters:= TStringList.Create;
ffSubDir:= SubDir;
ffPaths.Text:= FilePaths.Text;
ffFilters.Text:= Filter.Text;
ffOnFileFound:= fOnFileFound;
ffOnReady:= fOnReady;
ffAbort:= false;
FreeOnTerminate:= true;
{ Resume; }
Suspended := false;
end;
procedure TSearchThread.Execute;
var
Spec: string;
n, q: Integer;
function BackSlashFix(Folder: string): string;
begin
if Copy(Folder, Length(Folder), 1) = '\' then
begin
Result:= Folder;
Exit;
end
else
Result:= Folder + '\';
end;
procedure RFindFile(const Folder: string);
var
SearchRec: TSearchRec;
begin
if FindFirstUTF8(Folder + Spec, faAnyFile, SearchRec) { *Converted from FindFirst* } = 0 then
begin
try
repeat
if (SearchRec.Attr and faDirectory = 0) and
(SearchRec.Name <> '.') and (SearchRec.Name <> '..') then
begin
ffFileFound:= Folder + SearchRec.Name;
// ffList.Add(ffFileFound);
Synchronize(FileFound);
end;
until (FindNextUTF8(SearchRec) { *Converted from FindNext* } <> 0) or ffAbort;
except
FindCloseUTF8(SearchRec); { *Converted from FindClose* }
raise;
end;
FindCloseUTF8(SearchRec); { *Converted from FindClose* }
end;
if ffSubDir then
begin
if FindFirstUTF8(Folder + '*', faAnyFile, SearchRec) { *Converted from FindFirst* } = 0 then
begin
try
repeat
if ((SearchRec.Attr and faDirectory) <> 0) and
(SearchRec.Name <> '.') and (SearchRec.Name <> '..') then
RFindFile(Folder + SearchRec.Name + '\');
until (FindNextUTF8(SearchRec) { *Converted from FindNext* } <> 0) or ffAbort;
except
FindCloseUTF8(SearchRec); { *Converted from FindClose* }
raise;
end;
FindCloseUTF8(SearchRec); { *Converted from FindClose* }
end;
end;
end; // procedure RFindFile inside of FindFile
begin // function FindFile
// ffList:= TStringList.Create;
try
while not Terminated do
begin
for q:= 0 to ffPaths.Count - 1 do
begin
for n:= 0 to ffFilters.Count - 1 do
begin
Spec:= ffFilters[n];
RFindFile(BackSlashFix(ffPaths[q]));
end;
end;
Synchronize(Ready);
Terminate;
end;
finally
// ffList.Free;
end;
end;
procedure TSearchThread.FileFound;
begin
// ffOwner.SearchResult.Add(ffFileFound);
if Assigned(ffOnFileFound) then
ffOnFileFound(Self, ffFileFound);
end;
procedure TSearchThread.Ready;
begin
ffOwner.fStarted:= false;
if Assigned(ffOnReady) then
ffOnReady(Self, ffAbort);
// ffOnReady(Self, ffAbort, ffList);
end;
procedure Register;
begin
RegisterComponents('MASoft', [TmFileScan]);
end;
end.
------------------------------------------------------------------------------ New Year. New Location. New Benefits. New Data Center in Ashburn, VA. GigeNET is offering a free month of service with a new server in Ashburn. Choose from 2 high performing configs, both with 100TB of bandwidth. Higher redundancy.Lower latency.Increased capacity.Completely compliant. http://p.sf.net/sfu/gigenet
_______________________________________________ Unicon-group mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/unicon-group
