Hi Peter Thanks for all the help so far, I've added code to add the last elements, if you want to have it i can past it in a new email to you. One more question, we need to compress the data I've now tried like this, within createDataset(...)
H5.H5Pset_layout(plist, HDF5Constants.H5D_CHUNKED); H5.H5Pset_chunk(plist, RANK, chunkSize); H5.H5Pset_deflate(plist, 9); I'm not sure what is the most efficient way, tried to exchange the H5Pset_deflate(plist, 9) with H5.H5Pset_szip(plist, HDF5Constants.H5_SZIP_NN_OPTION_MASK, 8); but did not see any diffrens. I read the szip would maybe be better. If I don't use deflate the hdf file is 1.5 gb with deflate it's 1.3 gb. So my hopes is that it can be further decreased in size. cheers, Håkon On 24 March 2010 17:25, Peter Cao <[email protected]> wrote: > Hi Håkon, > > Glad to know it work for you. Also you need to take care of the case that > the last block does not have the size of BLOCK_SIZE. This will happen > if the total size (25M) is not divided by BLOCK_SIZE. For better > performance, > make sure that BLOCK_SIZE is divided by CHUNK_SIZE. > > > Thanks > --pc > > > Håkon Sagehaug wrote: > >> Hi Peter, >> >> Thanks so much for the code, seems to work very well, the only thing I >> found was that when the index for next index to write in the hdf array, I >> had to add 1 to it, so instead of >> >> start_idx = i; >> >> I now have >> >> start_idx = i + 1; >> >> cheers, Håkon >> >> >> >> >> On 24 March 2010 01:19, Peter Cao <[email protected] <mailto: >> [email protected]>> wrote: >> >> Hi Håkon, >> >> Below is the program that you can start with. I am using variable >> length strings. >> For fixed length strings, there are some extra work. You may have >> to make the >> strings to the same length. >> >> You may try different chunk sizes and block sizes to have the best >> performance. >> >> ======================= >> import ncsa.hdf.hdf5lib.H5; >> import ncsa.hdf.hdf5lib.HDF5Constants; >> import ncsa.hdf.hdf5lib.exceptions.HDF5Exception; >> >> public class CreateStrings { >> >> private final static String H5_FILE = "G:\\temp\\strings.h5"; >> private final static String DNAME = "/strs"; >> private final static int RANK = 1; >> private final static long[] DIMS = { 25000000 }; >> private final static long[] MAX_DIMS = { >> HDF5Constants.H5S_UNLIMITED }; >> private final static long[] CHUNK_SIZE = { 25000 }; >> private final static int BLOCK_SIZE = 250000; >> >> private void createDataset(int fid) throws Exception { >> int did = -1, tid = -1, sid = -1, plist = -1; >> >> try { >> >> tid = H5.H5Tcopy(HDF5Constants.H5T_C_S1); >> // use variable length to save space >> H5.H5Tset_size(tid, HDF5Constants.H5T_VARIABLE); >> sid = H5.H5Screate_simple(RANK, DIMS, MAX_DIMS); >> >> // figure out creation properties >> >> plist = H5.H5Pcreate(HDF5Constants.H5P_DATASET_CREATE); >> H5.H5Pset_layout(plist, HDF5Constants.H5D_CHUNKED); >> H5.H5Pset_chunk(plist, RANK, CHUNK_SIZE); >> >> did = H5.H5Dcreate(fid, DNAME, tid, sid, plist); >> } finally { >> try { >> H5.H5Pclose(plist); >> } catch (HDF5Exception ex) { >> } >> try { >> H5.H5Sclose(sid); >> } catch (HDF5Exception ex) { >> } >> try { >> H5.H5Dclose(did); >> } catch (HDF5Exception ex) { >> } >> } >> } >> >> private void writeData(int fid) throws Exception { >> int did = -1, tid = -1, msid = -1, fsid = -1; >> long[] count = { BLOCK_SIZE }; >> >> try { >> did = H5.H5Dopen(fid, DNAME); >> tid = H5.H5Dget_type(did); >> fsid = H5.H5Dget_space(did); >> msid = H5.H5Screate_simple(RANK, count, null); >> String[] strs = new String[BLOCK_SIZE]; >> >> int idx = 0, block_indx = 0, start_idx = 0; >> long t0 = 0, t1 = 0; >> t0 = System.currentTimeMillis(); >> System.out.println("Total number of blocks = " >> + (DIMS[0] / BLOCK_SIZE)); >> for (int i = 0; i < DIMS[0]; i++) { >> strs[idx++] = "str" + i; >> if (idx == BLOCK_SIZE) { // operator % is very expensive >> idx = 0; >> H5.H5Sselect_hyperslab(fsid, >> HDF5Constants.H5S_SELECT_SET, >> new long[] { start_idx }, null, count, >> null); >> H5.H5Dwrite(did, tid, msid, fsid, >> HDF5Constants.H5P_DEFAULT, strs); >> >> if (block_indx == 10) { >> t1 = System.currentTimeMillis(); >> System.out.println("Total time (minutes) = " >> + ((t1 - t0) * (DIMS[0] / >> BLOCK_SIZE)) / 1000 >> / 600); >> } >> >> block_indx++; >> start_idx = i; >> } >> >> } >> >> } finally { >> try { >> H5.H5Sclose(fsid); >> } catch (HDF5Exception ex) { >> } >> try { >> H5.H5Sclose(msid); >> } catch (HDF5Exception ex) { >> } >> try { >> H5.H5Dclose(did); >> } catch (HDF5Exception ex) { >> } >> } >> } >> >> private void createFile() throws Exception { >> int fid = -1; >> >> fid = H5.H5Fcreate(H5_FILE, HDF5Constants.H5F_ACC_TRUNC, >> >> HDF5Constants.H5P_DEFAULT, HDF5Constants.H5P_DEFAULT); >> >> if (fid < 0) >> return; >> >> try { >> createDataset(fid); >> writeData(fid); >> } finally { >> H5.H5Fclose(fid); >> } >> } >> >> /** >> * @param args >> */ >> public static void main(String[] args) { >> try { >> (new CreateStrings()).createFile(); >> } catch (Exception ex) { >> ex.printStackTrace(); >> } >> } >> >> } >> ========================= >> >> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] <mailto:[email protected]> >> >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> >> >> >> ------------------------------------------------------------------------ >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org >> >> > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org > -- Håkon Sagehaug, Scientific Programmer Parallab, Uni BCCS/Uni Research [email protected], phone +47 55584125
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
