Thanks Bill! I'll use it as a model. ----- Mail original ----- > De: "Bill Lorensen" <[email protected]> > À: "David Froger" <[email protected]> > Cc: "Insight Developers" <[email protected]> > Envoyé: Mardi 1 Octobre 2013 13:44:41 > Objet: Re: [Insight-developers] read stdin, write stdout > > We did a memory read/write imageio for Slicer4. The code is here: > http://viewvc.slicer.org/viewvc.cgi/Slicer4/trunk/Libs/MRML/IDImageIO/ > A similar approach might be used to read/write pipes. > > Bill > > > On Tue, Oct 1, 2013 at 1:43 AM, David Froger <[email protected]> > wrote: > > Dear all, > > > > We are looking for a proper way to integrate in ITK reading/writing > > in standard input/output HDF5 images. > > > > We write a "proove concept" that works, but involves a lot of code > > duplication that should be avoided. The rest of this long mail is > > divided > > into these sections: > > - Motivations > > - HDF5 and unix pipe > > - Building ITK with recent HDF5 version > > - itkHDF5unixpipeImageIO module > > - ImageUnixPipeReader module > > - Using the new modules > > - Questions > > - References > > > > We can create topic branch as described in the Git workflow [2] if > > need. > > > > Thanks for reading! > > > > Best regards, > > David Froger > > > > > > Motivations > > ----------- > > > > We are rewriting command line tools based on the INRimage [1] image > > format/library so that they are now based ITK. > > > > A particularity of these command line tools is that they can be > > piped with unix > > pipe. For example, the 'so' command subtracts the pixel values of > > two images, > > while the 'ical' command computes min/max/average pixel values of a > > image. To > > test if two images are equals, we pipe the commands as follow: > > so image1.inr image2.inr | ical > > > > While we are aware of the ITK pipeline and that we can use it to > > pipe ITK > > sources/mappers/filters into one processus (as suggested in [3] for > > example), > > we want to keep the possibility of using unix pipe in order to keep > > a backward > > compatibilty because we use numerous bash and perl scripts. > > > > HDF5 and unix pipe > > ------------------ > > > > The new file image operations library [4], introduced in HDF5 at > > the 1.8.9 version > > on May 2012 makes easy to read/write HDF5 images on standard > > input/output. > > > > An example of writting an HDF5 image to standard output is: > > > > #include <stdio.h> > > #include "hdf5.h" > > #include "H5Cpp.h" > > #include "H5LTpublic.h" > > > > // C array -> HDF5 image file -> buffer -> stdout > > > > #define NDATA 5 > > > > int main() > > { > > // Create C array. > > int data[NDATA] = {2, 4, 8, 16, 32}; > > > > // Open HDF5 image file. > > hbool_t backing_store = 0; > > H5::FileAccPropList fapl; > > fapl.setCore((size_t)1,backing_store); > > H5::H5File > > file("nosuch.h5",H5F_ACC_TRUNC,H5::FileCreatPropList::DEFAULT,fapl); > > > > // Write C array in HDF5 image file. > > hsize_t dims[1]= {NDATA}; > > H5::DataSpace space(1,dims); > > H5::DataSet dset = > > file.createDataSet("pow2",H5::PredType::NATIVE_INT,space); > > dset.write(data, H5::PredType::NATIVE_INT); > > > > // Get HDF5 image file buffer. > > file.flush(H5F_SCOPE_GLOBAL); > > hid_t fileid = file.getId(); > > size_t size = H5Fget_file_image(fileid, NULL, 0); > > char* buffer = new char[size]; > > H5Fget_file_image(fileid, buffer, size); > > > > // Write buffer to standard output. > > fwrite(buffer,1,size,stdout); > > } > > > > Compile and run: > > > > g++ -o write_h5_stdout write_h5_stdout.cxx -lhdf5_hl -lhdf5 > > -lhdf5_cpp > > ./write_h5_stdout > f.hdf5 > > > > A example of reading this f.hdf5 file from the standard input is: > > > > #include <stdio.h> > > #include <stdlib.h> > > #include "hdf5.h" > > #include "H5Cpp.h" > > #include "H5LTpublic.h" > > > > #define NDATA 5 > > #define READ_SIZE 512 > > #define CHUNCK_SIZE 8192 > > > > // stdin -> buffer -> HDF5 image file -> C array > > > > main() > > { > > // size of buffer is buffer_size=CHUNCK_SIZE*nchunck, size > > of h5data is > > // h5data_size <= buffer_size > > if (CHUNCK_SIZE < READ_SIZE) { > > fprintf(stderr, "ERROR: CHUNCK_SIZE < READ_SIZE\n"); > > exit(EXIT_FAILURE); > > } > > > > // Read stdin into buffer. > > size_t nchunck = 1, n; > > size_t buffer_size = CHUNCK_SIZE*nchunck; > > char* buffer = (char*) malloc(buffer_size); > > size_t h5data_size = 0; > > while (1) { > > // Make sure buffer is big enough to receive READ_SIZE > > new bytes. > > if (h5data_size+READ_SIZE > buffer_size) { > > nchunck += 1; > > buffer_size = nchunck*CHUNCK_SIZE; > > buffer = (char*) realloc(buffer,buffer_size); > > } > > // copy from stdin to buffer. > > n = fread(buffer+h5data_size,1,READ_SIZE,stdin); > > h5data_size += n; > > > > if (n <= 0) break; > > } > > > > // Create HDF5 image file from buffer. > > unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | > > H5LT_FILE_IMAGE_DONT_RELEASE; > > hid_t fileid = H5LTopen_file_image(buffer, h5data_size, > > flags); > > H5::H5File *h5file; > > h5file = new H5::H5File; > > h5file->setId(fileid); > > > > // Read C array in HDF5 file image. > > int data[NDATA]; > > H5::DataSet dset = h5file->openDataSet("pow2"); > > dset.read(data,dset.getDataType()); > > > > for (int i=0 ; i<NDATA ; i++) { > > printf("%d\n",data[i]); > > } > > > > free(buffer); > > > > return 0; > > } > > > > Compile and run: > > > > g++ -o read_h5_stdin read_h5_stdin.cxx -lhdf5_hl -lhdf5 > > -lhdf5_cpp > > cat f.hdf5 | ./read_h5_stdin > > ./write_h5_stdout | ./read_h5_stdin > > > > Building ITK with recent HDF5 version > > ------------------------------------- > > > > ITK must be compiled and and linked with a HDF5 > 1.8.9 (the > > current ITK 4.4.0 > > embeds HDF5==1.8.7). > > > > itkHDF5unixpipeImageIO module > > ----------------------------- > > > > In order to read/write HDF5 in standard input/output, we have > > created a > > itkHDF5unixpipeImageIO module. To this purpose, we have copied the > > directory > > Modules/IO/HDF5 into Modules/External/HDF5unixpipe (a bad practice > > that should > > be fixed as it duplicates a lot of code). > > > > src/itkHDF5unixpipeImageIO.cxx is the same as > > src/itkHDF5ImageIO.cxx, except > > that: > > > > - CanWriteFile and CanReadFile test on file extensions .h5stdout > > and .h5stdin, > > CanReadFile no more test on file existance. > > > > - An helper function ReadStandardInput is added: > > > > void > > HDF5unixpipeImageIO > > ::ReadStandardInput() > > { > > // Do not call the function twice. > > if (this->m_StandardInputReaden) return; > > > > /* Size of m_H5FileImageBuffer is H5FileImageBufferSize = > > chunckSize*nChunck. > > * Size of actual data is m_H5FileImageDataSize <= > > H5FileImageBufferSize. > > */ > > > > size_t readSize = 512; > > size_t chunckSize = 8192; > > size_t nChunck = 1, nBytesReaden; > > size_t H5FileImageBufferSize = chunckSize*nChunck; > > > > this->m_H5FileImageBuffer = (char*) > > malloc(H5FileImageBufferSize); > > this->m_H5FileImageDataSize = 0.; > > > > while (1) { > > /* make sure m_H5FileImageBuffer is big enough to > > receive readSize new bytes */ > > if (this->m_H5FileImageDataSize+readSize > > > H5FileImageBufferSize) { > > nChunck += 1; > > H5FileImageBufferSize = nChunck*chunckSize; > > this->m_H5FileImageBuffer = (char*) > > realloc(this->m_H5FileImageBuffer,H5FileImageBufferSize); > > } > > > > /* copy from stdin to m_H5FileImageBuffer */ > > nBytesReaden = > > > > fread(this->m_H5FileImageBuffer+this->m_H5FileImageDataSize,1,readSize,stdin); > > this->m_H5FileImageDataSize += nBytesReaden; > > > > if (nBytesReaden <= 0) break; > > } > > > > this->m_StandardInputReaden = true; > > } > > > > In ReadImageInformation(): > > > > this->m_H5File = new H5::H5File(this->GetFileName(), > > H5F_ACC_RDONLY); > > > > is replaced with: > > > > // Read stdin into m_H5FileImageBuffer. > > this->ReadStandardInput(); > > > > // Create HDF5 image file from buffer. > > unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | > > H5LT_FILE_IMAGE_DONT_RELEASE; > > hid_t fileid = H5LTopen_file_image(this->m_H5FileImageBuffer, > > this->m_H5FileImageDataSize, flags); > > this->m_H5File = new H5::H5File; > > this->m_H5File->setId(fileid); > > > > in WriteImageInformation(void): > > > > this->m_H5File = new H5::H5File(this->GetFileName(), > > H5F_ACC_TRUNC); > > > > is replaced with: > > > > hbool_t backing_store = 0; > > H5::FileAccPropList fapl; > > fapl.setCore((size_t)1,backing_store); > > this->m_H5File = new H5::H5File("nosuch.h5",H5F_ACC_TRUNC, > > H5::FileCreatPropList::DEFAULT,fapl); > > > > In Write(const void *buffer), at this end of the try block, we add: > > > > // Get HDF5 image file buffer. > > this->m_H5File->flush(H5F_SCOPE_GLOBAL); > > hid_t fileid = this->m_H5File->getId(); > > size_t imageFileBufferSize = H5Fget_file_image(fileid, NULL, > > 0); > > char* imageFileBuffer = (char*) malloc(imageFileBufferSize); > > H5Fget_file_image(fileid, imageFileBuffer, > > imageFileBufferSize); > > > > // Write imageFileBuffer to standard output. > > fwrite(imageFileBuffer,1,imageFileBufferSize,stdout); > > > > > > ImageUnixPipeReader module > > -------------------------- > > > > In the directory ./IO/ImageBase/include, we copied > > itkImageFileReader.h > > in itkImageUnixPipeReader.h and itkImageFileReader.hxx in > > itkImageUnixPipeReader.hxx (again, a bad practice that should > > be fixed as it duplicates a lot of code). > > > > The only change is to remove the use of the > > TestFileExistanceAndReadability > > function. > > > > Note that itkHDF5unixpipe can be used directly with > > ImageFileWritter. > > > > Using the new modules > > --------------------- > > > > The new modules to read/write in standard input/output can now be > > used as the > > ones that read in files, but by specifying "virtual" file names > > with extension > > ".h5stdin" and ".h5stdout", so that ImageUnixPipeReader and > > ImageFileWritter > > know they can make use of itkHDF5unixpipe, if we want to add > > support of other > > image formats. > > > > Missing features > > ---------------- > > > > A module ImageUnixPipeWriter should be added. ImageUnixPipeWriter > > and > > ImageUnixPipeReader, should not test on file extension (there is no > > file name), > > but require an ImageIO to be explicity passed. > > > > Questions > > --------- > > > > 1. Is there any chance that this functionnalty could be integrated > > in ITK? > > (if not, I suppose we should put everything in Modules/external). > > > > 2. In either case, how to avoid code duplication? Can we use class > > inheritance? > > > > References > > ---------- > > > > [1] (in french) http://inrimage.gforge.inria.fr/WWW/index.html > > [2] http://www.itk.org/Wiki/ITK/Git/Develop > > [3] > > http://www.itk.org/pipermail/insight-users/2004-September/010274.html > > [4] > > http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf > > _______________________________________________ > > Powered by www.kitware.com > > > > Visit other Kitware open-source projects at > > http://www.kitware.com/opensource/opensource.html > > > > Kitware offers ITK Training Courses, for more information visit: > > http://kitware.com/products/protraining.php > > > > Please keep messages on-topic and check the ITK FAQ at: > > http://www.itk.org/Wiki/ITK_FAQ > > > > Follow this link to subscribe/unsubscribe: > > http://www.itk.org/mailman/listinfo/insight-developers > > > > -- > Unpaid intern in BillsBasement at noware dot com > _______________________________________________ Powered by www.kitware.com
Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Kitware offers ITK Training Courses, for more information visit: http://kitware.com/products/protraining.php Please keep messages on-topic and check the ITK FAQ at: http://www.itk.org/Wiki/ITK_FAQ Follow this link to subscribe/unsubscribe: http://www.itk.org/mailman/listinfo/insight-developers
