Hi all,
We are conducting a project involving replacing (Linux) system's
libz.so with our own hardware based implementation, but this requires us to
replace libzip.so with our own so that small zip processing doesn't go through
hardware, as hardware actually cannot process these requests correctly due to
structural differences between hardware and software implementations of the
deflate algorithm. Anyhow these changes work for other format files when MapRed
is used where compression operations of user data do go through hardware
library, but not ORC files. We found out that ORC files actually go through
libzip.so instead of libz.so. So my questions are, in order to make ORC
compression/decompression processing goes to libz.so ( hardware ) instead of
libzip.so ( software ):
1. are there any places in Hadoop/Hive configuration files we can change to
make this happen?
2. if not, what should be changed in libzip.so so that requests for ORC format
can be forwarded to libz.so instead? An equivalent question is: in linzip.so,
is there a way to detect incoming data format? If so, which sources in jdk
should be looked at?
Many thanks in advance for any help.
--
======Not=Sent=From=Any=Phone=or=Pad=But=Stored=On=Hydra=======
Lihao Xu email: [email protected]
