[
https://issues.apache.org/jira/browse/HADOOP-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564755#action_12564755
]
Ankur commented on HADOOP-1824:
-------------------------------
Here is what I did
1. Implemented JNI callbacks in C that callback Java for open, read, close,
seek and tell on a FSDataInputStream.
2. Implemented some JAVA test code to verify that callbacks work correctly.
3. Made changes to existing Makefile of minizip to compile and build my C code
as shared object.
4. Placed the ".so" file in $LD_LIBRARY_PATH directory.
The integration was successful and worked beautifully. The callbacks worked
perfectly to ensure that zip file opened
as FSDataInputStream was opened ad read correclty :-)
However, Sigh. I found that the minizip parsing code did'nt work correctly for
Zip file > 2 GB. :-(
The code uses uLong (unsigned long 4 bytes) instead of jlong (signed long long
8 bytes).
Replacing uLong with jlong would'nt work as code performs a lot of bit shifting
operations. (I tried this.)
Also the parsing code relies on directory structure entries being in 32 - bit
format and will require RE-WORK
based upon knowledge of 64-bit entries keeping in mind backward compatibility
with 32 bit entries.
Note:- The IO callback APIs implemented by me make use of jlong in read(),
seek() and tell().
QUESTION: Is the RE-WORK really required or is there a workaround that I am
missing ???
> want InputFormat for zip files
> ------------------------------
>
> Key: HADOOP-1824
> URL: https://issues.apache.org/jira/browse/HADOOP-1824
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Affects Versions: 0.15.2
> Reporter: Doug Cutting
> Attachments: ZipInputFormat_fixed.patch
>
>
> HDFS is inefficient with large numbers of small files. Thus one might pack
> many small files into large, compressed, archives. But, for efficient
> map-reduce operation, it is desireable to be able to split inputs into
> smaller chunks, with one or more small original file per split. The zip
> format, unlike tar, permits enumeration of files in the archive without
> scanning the entire archive. Thus a zip InputFormat could efficiently permit
> splitting large archives into splits that contain one or more archived files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
