Thank you, Elena. I was able to reproduce the hanging outside the
python script, with just IDL and HDF5.
I strace'ed and found that it was in the IDL, FFT code. We did more
research after that and found that IDL's FFT command takes several
hundred times more CPU time for odd arrays than even. Example: x[719947]
takes an about hour, while x[719948] takes about 30 sec.
More detail from: http://www.harrisgeospatial.com/docs/fft.html
"For a one-dimensional FFT, running time is roughly proportional to the
total number of points in Array times the sum of its prime factors."
I think we are past the problem for now. Thank you,
Kellee
On 04/18/2016 10:00 AM, [email protected] wrote:
Date: Sun, 17 Apr 2016 21:16:49 +0000
From: Elena Pourmal <[email protected]>
To: HDF Users Discussion List <[email protected]>
Subject: Re: [Hdf-forum] H5D_READ hanging an IDL process
Message-ID: <[email protected]>
Content-Type: text/plain; charset="windows-1252"
Hi Kellee,
There is not much information to suggest what could be going wrong. (And you
are using a pretty old version of HDF5 trunk in the first place!)
A few questions that come to mind?.
Would it be possible for you to try an officially released version of HDF5
(e.g., 1.8.16 or 1.10.0)? Can you reproduce the problem outside python script,
i.e., just with IDL and HDF5? Or opening and reading the file without IDL,
i.e., just python script and C program? Do you need to use thread-safe HDF5?
May be other people will have other suggestions for what to try...
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Apr 14, 2016, at 6:59 PM, Kellee Summers
<[email protected]<mailto:[email protected]>> wrote:
Hello,
Has anyone seen a problem like this? We are writing data at 1KHz that is set
up like this:
HDF5 "201604121356.ovms.opd_estimation_mode2.h5" {
DATASET "opd_estimation_mode2_01" {
DATATYPE H5T_COMPOUND {
H5T_STD_I64LE "time_stamp";
H5T_STD_I32LE "tai_offset";
H5T_IEEE_F32LE "sx_x_fp";
H5T_IEEE_F32LE "sx_y_fp";
H5T_IEEE_F32LE "dx_x_fp";
H5T_IEEE_F32LE "dx_y_fp";
H5T_IEEE_F32LE "diff_z_fp";
H5T_IEEE_F32LE "predict_horiz";
}
In a CentOS environment using hdf5 1.9.131 and IDL 1.8. We have scripts for reading the
HDF5 files. Typically a python script identifies the .h5 files to be read, and then
spawns an IDL process to read them and make a plot. Typically reading a 100M .h5
telemetry files take about 30 sec. The problem is that every 20th time or so, the IDL
process is getting "stuck" and uses 100% CPU until we kill it. This only
happens for IDL reading .h5 files, and never for the cases where the python script makes
a large .csv which is read by IDL. It doesn't matter if the .h5 file is being actively
written to, or if the file is weeks old.
When running from the IDL command line, to see the print statements, it seems
to hang on the H5D_READ call. Sometimes it will hang repeatedly on the same
data file, sometimes it works on the same file, without hanging.
I am going to try to attach to it with strace the next time I see it, but I was
wondering if someone else has seen something similar?
Thanks
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]<mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20160417/ea209d45/attachment-0001.html>
------------------------------
Subject: Digest Footer
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
------------------------------
End of Hdf-forum Digest, Vol 82, Issue 23
*****************************************
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
