Hi all,
I'm working on a project that deals with HDF4 files provided by NASA.
This is part of a big project in University of Minnesota (
http://spatialhadoop.cs.umn.edu/) that uses Hadoop as a distributed
environment to process large data including satellite data. I used to use
HDFJava wrappers to deal with HDF files and it worked OK. However, since it
depends on native libraries, I frequently have troubles distributing the
code over a heterogeneous cluster of different architectures and systems.
So, I decided to write my own Java-native code to open HDF files.
Currently, I'm just writing the minimal code to support the files I'm
working on.
I followed the specifications provided on the website and I was
successfully able to get the structure of the HDF file and reach the data
blocks that contain the main data (temperature in my case). These blocks
are compressed using deflate with level 1 as indicated by the file.
However, when I try to decompress these blocks, they do not give me the
expected size. I feel that there's some kind of incompatible decompression
parameters that I need to fix. Here's the code I wrote to decompress the
file
byte[] compressedData = readRawData(in); // Reads the raw data from the
block
DeflaterInputStream dis = new DeflaterInputStream(new
ByteArrayInputStream(compressedData), new Deflater(1, true)); // Creates a
decompressor
ByteArrayOutputStream baos = new ByteArrayOutputStream(); // A
temporary place to store decompressed data
// This loop reads the uncompressed data and caches it in memory
byte[] buffer = new byte[4096];
int bufferLength;
while ((bufferLength = dis.read(buffer)) > 0) {
baos.write(buffer, 0, bufferLength);
}
dis.close();
baos.close();
// Retrieve the decompressed data
uncompressedData = baos.toByteArray();
The header of the compressed data indicate that the size after
decompression should be 2880000 bytes, but my code finds that it is
1613097, which is totally different. When creating the deflater, I tried to
set the 'nowrap' parameter to true or false but both do not work. The same
file works with HDFJava libraries so it should be a bug in my code.
Thanks in advance for your help. Please let me know if you need more
details about this issue.
Best regards,
Ahmed Eldawy
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
