Thank you for the suggestions. Yes, we are using CloseIdentifyFile immediately after the IdentifyFile, but nevertheless we have a memory leak. If we remove the IdentifyFile the memory remains stable.
We are now trying to find a way to achieve the same goal using another method. Greetings, Natalia -----Mensaje original----- De: John P. Rouillard [mailto:rou...@cs.umb.edu] Enviado el: martes, 27 de mayo de 2014 2:33 Para: simple-evcorr-users@lists.sourceforge.net Asunto: Re: [Simple-evcorr-users] Handling of Windows files in SEC In message <cagfjscobvfj6lfwbnjjmqsdiuildzsceds4ygmgt_omlh1m...@mail.gmail.com> , Risto Vaarandi writes: >2014-05-26 19:06 GMT+03:00 Natalia Iglesias ><nigles...@lookwisesolutions.com>>: >> We think we have found out the reason of the memory leak, there is an >> external PERL library being called that contains a call to the >> IdentifyFile function to obtain the inode of windows files. Whenever >> this function is called, a memory leak occurs. If we disable this >> call, the memory usage is stable. I assume you mean: Win32::IdentifyFile from: http://search.cpan.org/~clive/Win32-IdentifyFile-1.01/lib/Win32/IdentifyFile .pm >> So we are thinking of a way to bypass this problem, either by using >> another function or assessing if it's still needed. For that we would >> like to ask you a couple of questions about SEC internal functioning >> when reading (Windows) files from the file system: >> >> 1. Does SEC detect that an open file is idle, without any writing >> activity? And if it does, does SEC close the file after a while, or >> does the file remain open until it's explicitly closed by a process? >> > >Do you mean input files which are specified with --input command line >option(s)? If so, they are never closed by SEC (unless a critical IO >error is observed). While keeping the input file open, each input file >is frequently checked for new data, and if the check was not >successful, SEC also checks whether the file has been truncated or >recreated (the latter check is relevant to UNIX only, since UNIX >filesystems allow for recreating an open file, so that the open file >would lose its name in the directory tree). If the file truncation or >recreation is observed, SEC will reopen the file. Can you tell if Win32::IdentifyFile is being called as a result of a stat() on the input file? The web page I referenced aabove says: It is important to call CloseIdentifyFile() in order to avoid a resource leak. Only one call is required, regardless of the number of times IdentifyFile() is called. If you can tell when IdentifyFile is being called, maybe you can manually run the CloseIdentifyFile with a perl sub every now and again (or worst case after every line). If Strawberry Perl is using that library internally as part of it's stat() call or something, I would report this as a bug as Strawberry perl shouldn't be leaking memory. Another possibility is to use something like cygwin perl to run SEC, although it will probably be a slower than a native windows Perl. -- -- rouilj John Rouillard =========================================================================== My employers don't acknowledge my existence much less my opinions. ---------------------------------------------------------------------------- -- The best possible search technologies are now affordable for all companies. Download your FREE open source Enterprise Search Engine today! Our experts will assist you in its installation for $59/mo, no commitment. Test it for FREE on our Cloud platform anytime! http://pubads.g.doubleclick.net/gampad/clk?id=145328191&iu=/4140/ostg.clktrk _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users ------------------------------------------------------------------------------ The best possible search technologies are now affordable for all companies. Download your FREE open source Enterprise Search Engine today! Our experts will assist you in its installation for $59/mo, no commitment. Test it for FREE on our Cloud platform anytime! http://pubads.g.doubleclick.net/gampad/clk?id=145328191&iu=/4140/ostg.clktrk _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
