Hello Edward,
I'm not an expert with this module, but I can sniff out the problem. The SD
file handle gets closed using the close
<https://metacpan.org/source/CHM/PDL-2.007/IO/HDF/SD/SD.pd#L1578> method,
which is usually called by DESTROY
<https://metacpan.org/source/CHM/PDL-2.007/IO/HDF/SD/SD.pd#L1586>. However,
if the new <https://metacpan.org/source/CHM/PDL-2.007/IO/HDF/SD/SD.pd#L405>
method fails before blessing the reference
<https://metacpan.org/source/CHM/PDL-2.007/IO/HDF/SD/SD.pd#L579>, then the
destroy method is never called. Quite a bit happens before $self is
blessed, and that is probably the thing creating the leak. This clearly
seems to be the case for you.
I think the easiest solution is to bless the reference as soon as it's
created, and to add logic to the DESTROY and/or close method to not attempt
to close the SDID if there is none. Based on the comment just before the
close method, I'm not 100% sure that'll solve the problem, though. Do you
know how to implement this idea and see if it fixes the problem, or would
you like some guidance?
David
On Mon, May 11, 2015 at 2:04 PM, Hyer, Dr. Edward <
[email protected]> wrote:
> Hello PDL wizards,
>
> My code churns through hundreds of thousands of HDF files, and some
> fraction
> of them are corrupted (bit-rot, as far as I can tell).
>
> Because PDL::IO::HDF::SD does not fail gracefully, I have to catch these
> files like this:
> # Open HDF file for input
> my $hdfopen_status = eval{$hdfobj =
> PDL::IO::HDF::SD->new("$hdf_file");};
> unless($hdfobj){
> warn "$routine: Cannot open $hdf_file : $@\n";
> print "$routine: Cannot open $hdf_file : $@\n";
> #my $hdfclose_status=$hdfobj->close(); # can't do this because
> $hdfobj doesn't exist!
> return -2;};
>
> This accomplishes my goals of putting information into the log about bad
> files (the parent routine then deals with the files), and moving on.
>
> However, lately I have been running a long reprocessing job, and at a
> certain point it stops with the "Too many open files" error, always right
> after encountering a corrupted file. My suspicion is that
> PDL::IO::HDF::SD->new() is leaving an open filehandle when it exits
> uncleanly.
>
> I don't know the precise number of corrupted files that accumulates to
> generate this error, but it's on the order of 1000.
>
> Any ideas?
>
> Thanks,
>
> --Edward H.
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
> _______________________________________________
> pdl-general mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/pdl-general
>
>
--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y
_______________________________________________
pdl-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/pdl-general
