Hi Nitya, Looks like you are in a Linux 3.0 system. Can you find out what distribution it is? E.g., Redhat, Fedora, Suse, …. There is no one way to find this information but you often can find a file like /etc/<XYZ>-release. If you can locate such a file, please send its content.
-Albert Cheng On Jul 4, 2013, at 4:31 AM, Nitya <[email protected]> wrote: > Hi Albert, Dana, > > Thanks for your replies. I give the output of uname below > > Linux 3.0.74-0.6.10-default #1 SMP x86_64 x86_64 x86_64 GNU/Linux > > I was earlier using v1.8.9 and then tried using v1.8.11. I have given > the libhdf5.settings file for both versions below. I downloaded the > library files of v1.8.11 from the HDF5 website, so haven't compiled > anything. v1.8.9 has been compiled by the system administrators. > > ------------------- > SUMMARY OF THE HDF5 CONFIGURATION > ================================= > > General Information: > ------------------- > HDF5 Version: 1.8.9 > Configured on: Thu Nov 29 11:11:42 CET 2012 > Configured by: wwn@qa01 > Configure mode: production > Host system: x86_64-unknown-linux-gnu > Uname information: Linux qa01 3.0.42-0.7-default #1 SMP > Tue Oct 9 11:58:45 UTC 2012 (a8dc443) x86_64 x86_64 x86_64 GNU/Linux > Byte sex: little-endian > Libraries: > Installation point: > /afs/rzg/common/soft/hdf5/1.8.9/amd64_sles11/intel/12.1/serial > > Compiling Options: > ------------------ > Compilation Mode: production > C Compiler: > /afs/@cell/common/soft/intel/ics/2011.0.013/12.1/bin/intel64/bin/icc ( > Intel(R) C Intel(R) 64 Compiler Version 12.1 Build 20120928) > CFLAGS: -fPIC > H5_CFLAGS: -std=c99 -O3 > AM_CFLAGS: > CPPFLAGS: > H5_CPPFLAGS: -D_POSIX_C_SOURCE=199506L -DNDEBUG > -UH5_DEBUG_API > AM_CPPFLAGS: -D_LARGEFILE_SOURCE > -D_LARGEFILE64_SOURCE -D_BSD_SOURCE > Shared C Library: yes > Static C Library: yes > Statically Linked Executables: no > LDFLAGS: > H5_LDFLAGS: > AM_LDFLAGS: > Extra libraries: -lz -lrt -lm > Archiver: ar > Ranlib: ranlib > Debugged Packages: > API Tracing: no > > Languages: > ---------- > Fortran: yes > Fortran Compiler: > /afs/@cell/common/soft/intel/ics/2011.0.013/12.1/bin/intel64/bin/ifort ( > Intel(R) Fortran Intel(R) 64 Compiler Version 12.1 Build 20120928) > Fortran 2003 Compiler: yes > Fortran Flags: -fPIC > H5 Fortran Flags: -O3 > AM Fortran Flags: > Shared Fortran Library: yes > Static Fortran Library: yes > C++: yes > C++ Compiler: > /afs/@cell/common/soft/intel/ics/2011.0.013/12.1/bin/intel64/bin/icpc ( > Intel(R) C++ Intel(R) 64 Compiler Version 12.1 Build 20120928) > C++ Flags: > H5 C++ Flags: > AM C++ Flags: > Shared C++ Library: yes > Static C++ Library: yes > > Features: > --------- > Parallel HDF5: no > High Level library: yes > Threadsafety: no > Default API Mapping: v18 > With Deprecated Public Symbols: yes > I/O filters (external): deflate(zlib) > I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset > MPE: no > Direct VFD: no > dmalloc: no > Clear file buffers before write: yes > Using memory checker: no > Function Stack Tracing: no > GPFS: no > Strict File Format Checks: no > Optimization Instrumentation: no > Large File Support (LFS): yes > ----------------------------------------------- > > > v1.8.11 > *************************************** > SUMMARY OF THE HDF5 CONFIGURATION > ================================= > > General Information: > ------------------- > HDF5 Version: 1.8.11 > Configured on: Wed May 8 16:20:56 CDT 2013 > Configured by: hdftest@koala > Configure mode: production > Host system: x86_64-unknown-linux-gnu > Uname information: Linux koala 2.6.18-348.1.1.el5 #1 SMP > Tue Jan 22 16:19:19 EST 2013 x86_64 x86_64 x86_64 GNU/Linux > Byte sex: little-endian > Libraries: static, shared > Installation point: > /mnt/scr1/pre-release/hdf5/v1811/thg-builds/koala > > Compiling Options: > ------------------ > Compilation Mode: production > C Compiler: /usr/bin/gcc ( gcc (GCC) 4.1.2 20080704 ) > CFLAGS: > H5_CFLAGS: -std=c99 -pedantic -Wall -Wextra > -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual > -Wcast-align -Wwrite-strings -Wconversion -Wag > gregate-return -Wstrict-prototypes -Wmissing-prototypes > -Wmissing-declarations -Wredundant-decls -Wnested-externs -Winline > -Wno-long-long -Wfloat-equal -Wmissing-format-attribut > e -Wmissing-noreturn -Wpacked -Wdisabled-optimization -Wformat=2 > -Wunreachable-code -Wendif-labels -Wdeclaration-after-statement > -Wold-style-definition -Winvalid-pch -Wvariadic- > macros -Wnonnull -Winit-self -Wmissing-include-dirs -Wswitch-default > -Wswitch-enum -Wunused-macros -Wunsafe-loop-optimizations -Wc++-compat > -Wvolatile-register-var -O3 -fomit-fr > ame-pointer -finline-functions > AM_CFLAGS: > CPPFLAGS: > H5_CPPFLAGS: -D_POSIX_C_SOURCE=199506L -DNDEBUG > -UH5_DEBUG_API > AM_CPPFLAGS: > -I/mnt/hdf/packages/szip/shared/encoder/Linux2.6-x86_64-gcc/include > -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_BSD_SOURCE > Shared C Library: yes > Static C Library: yes > Statically Linked Executables: yes > LDFLAGS: > H5_LDFLAGS: > AM_LDFLAGS: > -L/mnt/hdf/packages/szip/shared/encoder/Linux2.6-x86_64-gcc/lib > Extra libraries: -lsz -lz -lrt -ldl -lm > Archiver: ar > Ranlib: ranlib > Debugged Packages: > API Tracing: no > Languages: > ---------- > Fortran: yes > Fortran Compiler: /usr/bin/gfortran ( GNU Fortran (GCC) > 4.1.2 20080704 ) > Fortran 2003 Compiler: no > Fortran Flags: > H5 Fortran Flags: > AM Fortran Flags: > Shared Fortran Library: yes > Static Fortran Library: yes > > C++: yes > C++ Compiler: /usr/bin/g++ ( g++ (GCC) 4.1.2 20080704 ) > C++ Flags: > H5 C++ Flags: > AM C++ Flags: > Shared C++ Library: yes > Static C++ Library: yes > > Features: > --------- > Parallel HDF5: no > High Level library: yes > Threadsafety: no > Default API Mapping: v18 > With Deprecated Public Symbols: yes > I/O filters (external): deflate(zlib),szip(encoder) > I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset > MPE: no > Direct VFD: no > dmalloc: no > Clear file buffers before write: yes > Using memory checker: no > Function Stack Tracing: no > GPFS: no > Strict File Format Checks: no > Optimization Instrumentation: no > Large File Support (LFS): yes > **************************************** > > I had another query. What value does H5P_DEFAULT contain and where is > the value for this variable set. I was not able to find it from the > v1.8.9 source, please let me know. I ask, since the code I am working > with, sets the file access to H5P_DEFAULT if MPI is not being used, else > its set to H5Pset_fapl_mpio. > > Please let me know if you need more information. > > Thanks > > Nitya > > ----------------------------- > > Message: 4 > Date: Tue, 2 Jul 2013 14:33:07 -0500 > From: Albert Cheng <[email protected]> > To: HDF Users Discussion List <[email protected]> > Subject: Re: [Hdf-forum] HDF5 - reading files larger than 2GB > Message-ID: <[email protected]> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Nitya, > > Can you also give more information about the machine in which you ran your > application > that got the failures? E.g., > > 1. OS version ("uname -a" usually shows it.) > 1.1. For Mac, try "sw_vers". > > 2. Did you build the HDF5 library from source? > 2.1. if yes, please send a copy of the output of configure. > 2.2. If the output of configure is not available, please send a copy of > libhdf5.settings. > You can usually find "libhdf5.settings" in the same directory where > you find the > HDF5 library, libhdf5.a or libhdf5.so. > > This will help us to try to repeat your problems in the appropriate platform. > Thanks. > > -Albert Change > THG Staff > > On Jul 2, 2013, at 11:20 AM, Dana Robinson <[email protected]> wrote: > >> Hi Nitya, >> >> I'll take a look into this. It's a holiday weekend here, so I probably >> won't get to this until next week. >> >> Dana >> >> >> On Mon, Jul 1, 2013 at 1:37 PM, Nitya Hariharan <[email protected]> >> wrote: >> Hi, >> >> I am having some trouble reading in a HDF5 file that is ~5GB. I was >> initially having some problems even writing out a file this large and >> looked at some postings in the HDF5 forum related to this. >> >> http://hdf-forum.184993.n3.nabble.com/Trouble-writing-2GB-dataset-from-single-task-with-HDF5-1-8-10-td4025821.html >> >> I tried using the serial version of the HDF5 library, v1.8.9 and was >> able to write out a file of size ~5GB. However, when I try to read it >> back in, I get the following error. >> >> ---------------- >> #000: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dio.c line >> 182 in H5Dread(): can't read data >> major: Dataset >> minor: Read failed >> #001: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dio.c line >> 550 in H5D__read(): can't read data >> major: Dataset >> minor: Read failed >> #002: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dcontig.c >> line 543 in H5D__contig_read(): contiguous read failed >> major: Dataset >> minor: Read failed >> #003: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dselect.c >> line 278 in H5D__select_read(): read error >> major: Dataspace >> minor: Read failed >> #004: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dselect.c >> line 213 in H5D__select_io(): read error >> major: Dataspace >> minor: Read failed >> #005: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dcontig.c >> line 873 in H5D__contig_readvv(): can't perform vectorized sieve buffer read >> major: Dataset >> minor: Can't operate on object >> #006: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5V.c line >> 1457 in H5V_opvv(): can't perform operation >> major: Internal error (too specific to document in detail) >> minor: Can't operate on object >> #007: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dcontig.c >> line 674 in H5D__contig_readvv_sieve_cb(): block read failed >> major: Dataset >> minor: Read failed >> #008: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Fio.c line >> 113 in H5F_block_read(): read through metadata accumulator failed >> major: Low-level I/O >> minor: Read failed >> #009: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Faccum.c >> line 258 in H5F_accum_read(): driver read request failed >> major: Low-level I/O >> minor: Read failed >> #010: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5FDint.c >> line 142 in H5FD_read(): driver read request failed >> major: Virtual File Layer >> minor: Read failed >> #011: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5FDsec2.c >> line 725 in H5FD_sec2_read(): file read failed: time = Mon Jul 1 >> 20:21:57 2013 >> , filename = '/tmp/file.hdf5', file descriptor = 5, errno = 14, error >> message = 'Bad address', buf = 0x2ae9619ea010, total read size = >> 4677466176, bytes this sub-read = 4677466176, bytes actually read = >> 18446744073709551615, offset = 744468544 >> ----------------- >> >> I looked at the forum again and saw this posting, which mentioned that >> there was a bug fix done for POSIX issues. >> >> http://mail.lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2012-December/006348.html >> >> I was using v1.8.9, but thought of using the latest 1.8.11 to rule out >> any issues with the HDF5 version I was using. However, I still get the >> above error. >> >> Please could someone provide some feedback on why this is happening. If >> I am able to write out such a large file, I should be able to read it in >> as well? Ofcourse, small sized files work perfectly fine in my application. >> >> Since the library I am using, that in turn calls the HDF5 routines, >> needs the interface of v1.6 of the HDF5 library, I use the flag >> H5_USE_16_API while compiling. Would this in anyway cause the problem. >> >> Thanks in advance. >> >> -- >> Regards >> >> Nitya >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > > > > > > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
