Hi David, Thanks for the well written message!
>From a brief review of what was written, it looks like everything could be implemented in a backwards-compatible and without duplicating any code by modifying the existing classes. From what I understand, the existing HDF ImageIO can be slightly improved and a new flag could be added to ImageFileReader that disables TestFileExistanceAndReadability if the intention is to use ImageFileReader for pipes. Thanks, Matt On Tue, Oct 1, 2013 at 5:43 AM, David Froger <[email protected]> wrote: > Dear all, > > We are looking for a proper way to integrate in ITK reading/writing > in standard input/output HDF5 images. > > We write a "proove concept" that works, but involves a lot of code > duplication that should be avoided. The rest of this long mail is divided > into these sections: > - Motivations > - HDF5 and unix pipe > - Building ITK with recent HDF5 version > - itkHDF5unixpipeImageIO module > - ImageUnixPipeReader module > - Using the new modules > - Questions > - References > > We can create topic branch as described in the Git workflow [2] if need. > > Thanks for reading! > > Best regards, > David Froger > > > Motivations > ----------- > > We are rewriting command line tools based on the INRimage [1] image > format/library so that they are now based ITK. > > A particularity of these command line tools is that they can be piped with > unix > pipe. For example, the 'so' command subtracts the pixel values of two images, > while the 'ical' command computes min/max/average pixel values of a image. To > test if two images are equals, we pipe the commands as follow: > so image1.inr image2.inr | ical > > While we are aware of the ITK pipeline and that we can use it to pipe ITK > sources/mappers/filters into one processus (as suggested in [3] for example), > we want to keep the possibility of using unix pipe in order to keep a backward > compatibilty because we use numerous bash and perl scripts. > > HDF5 and unix pipe > ------------------ > > The new file image operations library [4], introduced in HDF5 at the 1.8.9 > version > on May 2012 makes easy to read/write HDF5 images on standard input/output. > > An example of writting an HDF5 image to standard output is: > > #include <stdio.h> > #include "hdf5.h" > #include "H5Cpp.h" > #include "H5LTpublic.h" > > // C array -> HDF5 image file -> buffer -> stdout > > #define NDATA 5 > > int main() > { > // Create C array. > int data[NDATA] = {2, 4, 8, 16, 32}; > > // Open HDF5 image file. > hbool_t backing_store = 0; > H5::FileAccPropList fapl; > fapl.setCore((size_t)1,backing_store); > H5::H5File > file("nosuch.h5",H5F_ACC_TRUNC,H5::FileCreatPropList::DEFAULT,fapl); > > // Write C array in HDF5 image file. > hsize_t dims[1]= {NDATA}; > H5::DataSpace space(1,dims); > H5::DataSet dset = > file.createDataSet("pow2",H5::PredType::NATIVE_INT,space); > dset.write(data, H5::PredType::NATIVE_INT); > > // Get HDF5 image file buffer. > file.flush(H5F_SCOPE_GLOBAL); > hid_t fileid = file.getId(); > size_t size = H5Fget_file_image(fileid, NULL, 0); > char* buffer = new char[size]; > H5Fget_file_image(fileid, buffer, size); > > // Write buffer to standard output. > fwrite(buffer,1,size,stdout); > } > > Compile and run: > > g++ -o write_h5_stdout write_h5_stdout.cxx -lhdf5_hl -lhdf5 -lhdf5_cpp > ./write_h5_stdout > f.hdf5 > > A example of reading this f.hdf5 file from the standard input is: > > #include <stdio.h> > #include <stdlib.h> > #include "hdf5.h" > #include "H5Cpp.h" > #include "H5LTpublic.h" > > #define NDATA 5 > #define READ_SIZE 512 > #define CHUNCK_SIZE 8192 > > // stdin -> buffer -> HDF5 image file -> C array > > main() > { > // size of buffer is buffer_size=CHUNCK_SIZE*nchunck, size of h5data > is > // h5data_size <= buffer_size > if (CHUNCK_SIZE < READ_SIZE) { > fprintf(stderr, "ERROR: CHUNCK_SIZE < READ_SIZE\n"); > exit(EXIT_FAILURE); > } > > // Read stdin into buffer. > size_t nchunck = 1, n; > size_t buffer_size = CHUNCK_SIZE*nchunck; > char* buffer = (char*) malloc(buffer_size); > size_t h5data_size = 0; > while (1) { > // Make sure buffer is big enough to receive READ_SIZE new bytes. > if (h5data_size+READ_SIZE > buffer_size) { > nchunck += 1; > buffer_size = nchunck*CHUNCK_SIZE; > buffer = (char*) realloc(buffer,buffer_size); > } > // copy from stdin to buffer. > n = fread(buffer+h5data_size,1,READ_SIZE,stdin); > h5data_size += n; > > if (n <= 0) break; > } > > // Create HDF5 image file from buffer. > unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | > H5LT_FILE_IMAGE_DONT_RELEASE; > hid_t fileid = H5LTopen_file_image(buffer, h5data_size, flags); > H5::H5File *h5file; > h5file = new H5::H5File; > h5file->setId(fileid); > > // Read C array in HDF5 file image. > int data[NDATA]; > H5::DataSet dset = h5file->openDataSet("pow2"); > dset.read(data,dset.getDataType()); > > for (int i=0 ; i<NDATA ; i++) { > printf("%d\n",data[i]); > } > > free(buffer); > > return 0; > } > > Compile and run: > > g++ -o read_h5_stdin read_h5_stdin.cxx -lhdf5_hl -lhdf5 -lhdf5_cpp > cat f.hdf5 | ./read_h5_stdin > ./write_h5_stdout | ./read_h5_stdin > > Building ITK with recent HDF5 version > ------------------------------------- > > ITK must be compiled and and linked with a HDF5 > 1.8.9 (the current ITK 4.4.0 > embeds HDF5==1.8.7). > > itkHDF5unixpipeImageIO module > ----------------------------- > > In order to read/write HDF5 in standard input/output, we have created a > itkHDF5unixpipeImageIO module. To this purpose, we have copied the directory > Modules/IO/HDF5 into Modules/External/HDF5unixpipe (a bad practice that should > be fixed as it duplicates a lot of code). > > src/itkHDF5unixpipeImageIO.cxx is the same as src/itkHDF5ImageIO.cxx, except > that: > > - CanWriteFile and CanReadFile test on file extensions .h5stdout and .h5stdin, > CanReadFile no more test on file existance. > > - An helper function ReadStandardInput is added: > > void > HDF5unixpipeImageIO > ::ReadStandardInput() > { > // Do not call the function twice. > if (this->m_StandardInputReaden) return; > > /* Size of m_H5FileImageBuffer is H5FileImageBufferSize = > chunckSize*nChunck. > * Size of actual data is m_H5FileImageDataSize <= > H5FileImageBufferSize. > */ > > size_t readSize = 512; > size_t chunckSize = 8192; > size_t nChunck = 1, nBytesReaden; > size_t H5FileImageBufferSize = chunckSize*nChunck; > > this->m_H5FileImageBuffer = (char*) malloc(H5FileImageBufferSize); > this->m_H5FileImageDataSize = 0.; > > while (1) { > /* make sure m_H5FileImageBuffer is big enough to receive > readSize new bytes */ > if (this->m_H5FileImageDataSize+readSize > H5FileImageBufferSize) > { > nChunck += 1; > H5FileImageBufferSize = nChunck*chunckSize; > this->m_H5FileImageBuffer = (char*) > realloc(this->m_H5FileImageBuffer,H5FileImageBufferSize); > } > > /* copy from stdin to m_H5FileImageBuffer */ > nBytesReaden = > fread(this->m_H5FileImageBuffer+this->m_H5FileImageDataSize,1,readSize,stdin); > this->m_H5FileImageDataSize += nBytesReaden; > > if (nBytesReaden <= 0) break; > } > > this->m_StandardInputReaden = true; > } > > In ReadImageInformation(): > > this->m_H5File = new H5::H5File(this->GetFileName(), > H5F_ACC_RDONLY); > > is replaced with: > > // Read stdin into m_H5FileImageBuffer. > this->ReadStandardInput(); > > // Create HDF5 image file from buffer. > unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE; > hid_t fileid = H5LTopen_file_image(this->m_H5FileImageBuffer, > this->m_H5FileImageDataSize, flags); > this->m_H5File = new H5::H5File; > this->m_H5File->setId(fileid); > > in WriteImageInformation(void): > > this->m_H5File = new H5::H5File(this->GetFileName(), > H5F_ACC_TRUNC); > > is replaced with: > > hbool_t backing_store = 0; > H5::FileAccPropList fapl; > fapl.setCore((size_t)1,backing_store); > this->m_H5File = new H5::H5File("nosuch.h5",H5F_ACC_TRUNC, > H5::FileCreatPropList::DEFAULT,fapl); > > In Write(const void *buffer), at this end of the try block, we add: > > // Get HDF5 image file buffer. > this->m_H5File->flush(H5F_SCOPE_GLOBAL); > hid_t fileid = this->m_H5File->getId(); > size_t imageFileBufferSize = H5Fget_file_image(fileid, NULL, 0); > char* imageFileBuffer = (char*) malloc(imageFileBufferSize); > H5Fget_file_image(fileid, imageFileBuffer, imageFileBufferSize); > > // Write imageFileBuffer to standard output. > fwrite(imageFileBuffer,1,imageFileBufferSize,stdout); > > > ImageUnixPipeReader module > -------------------------- > > In the directory ./IO/ImageBase/include, we copied itkImageFileReader.h > in itkImageUnixPipeReader.h and itkImageFileReader.hxx in > itkImageUnixPipeReader.hxx (again, a bad practice that should > be fixed as it duplicates a lot of code). > > The only change is to remove the use of the TestFileExistanceAndReadability > function. > > Note that itkHDF5unixpipe can be used directly with ImageFileWritter. > > Using the new modules > --------------------- > > The new modules to read/write in standard input/output can now be used as the > ones that read in files, but by specifying "virtual" file names with extension > ".h5stdin" and ".h5stdout", so that ImageUnixPipeReader and ImageFileWritter > know they can make use of itkHDF5unixpipe, if we want to add support of other > image formats. > > Missing features > ---------------- > > A module ImageUnixPipeWriter should be added. ImageUnixPipeWriter and > ImageUnixPipeReader, should not test on file extension (there is no file > name), > but require an ImageIO to be explicity passed. > > Questions > --------- > > 1. Is there any chance that this functionnalty could be integrated in ITK? > (if not, I suppose we should put everything in Modules/external). > > 2. In either case, how to avoid code duplication? Can we use class > inheritance? > > References > ---------- > > [1] (in french) http://inrimage.gforge.inria.fr/WWW/index.html > [2] http://www.itk.org/Wiki/ITK/Git/Develop > [3] http://www.itk.org/pipermail/insight-users/2004-September/010274.html > [4] > http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf > _______________________________________________ > Powered by www.kitware.com > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Kitware offers ITK Training Courses, for more information visit: > http://kitware.com/products/protraining.php > > Please keep messages on-topic and check the ITK FAQ at: > http://www.itk.org/Wiki/ITK_FAQ > > Follow this link to subscribe/unsubscribe: > http://www.itk.org/mailman/listinfo/insight-developers _______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Kitware offers ITK Training Courses, for more information visit: http://kitware.com/products/protraining.php Please keep messages on-topic and check the ITK FAQ at: http://www.itk.org/Wiki/ITK_FAQ Follow this link to subscribe/unsubscribe: http://www.itk.org/mailman/listinfo/insight-developers
