Hi all,
We are conducting a project involving replacing (Linux) system's
libz.so with our own hardware based implementation, but this requires us
to replace libzip.so with our own so that small zip processing doesn't go
through hardware, as hardware actually cannot process these requests
correctly due to structural differences between hardware and software
implementations of the deflate algorithm. Anyhow these changes work for
other format files when MapRed is used where compression operations of
user data do go through hardware library, but not ORC files. We found out
that ORC files actually go through libzip.so instead of libz.so. So my
questions are, in order to make ORC compression/decompression processing
goes to libz.so ( hardware ) instead of libzip.so ( software ):
1. are there any places in Hadoop/Hive configuration files we can change
to make this happen?
2. if not, what should be changed in libzip.so so that requests for ORC
format can be forwarded to libz.so instead? An equivalent question is: in
linzip.so, is there a way to detect incoming data format? If so, which
sources in jdk should be looked at?
Many thanks in advance for any help.
--
======Not=Sent=From=Any=Phone=or=Pad=But=Stored=On=Hydra=======
Lihao Xu email: li...@ieee.org
