[
https://issues.apache.org/jira/browse/HADOOP-10409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13939637#comment-13939637
]
Chris Nauroth commented on HADOOP-10409:
----------------------------------------
Hi, [~tthompso]. I've been bitten by this too. BTW, you can force your builds
to fail if bzip2-devel is missing with a command-line switch. This is in
hadoop-common-project/hadoop-common/pom.xml:
{code}
<profiles>
<profile>
<id>native</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<require.bzip2>false</require.bzip2>
<snappy.prefix></snappy.prefix>
<snappy.lib></snappy.lib>
<snappy.include></snappy.include>
<require.snappy>false</require.snappy>
</properties>
{code}
However, note that it's false by default, so the default behavior is just to
let the build proceed silently without producing the bzip2-related symbols in
the so. You have to pass -Drequire.bzip2 on the command line.
I checked BUILDING.txt, and this switch is not documented there. This is
something we could add to make it more obvious.
> Bzip2 error message isn't clear
> -------------------------------
>
> Key: HADOOP-10409
> URL: https://issues.apache.org/jira/browse/HADOOP-10409
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io
> Affects Versions: 2.3.0
> Reporter: Travis Thompson
>
> If you compile hadoop without {{bzip2-devel}} installed (on RHEL), bzip2
> doesn't get compiled into libhadoop, as is expected. This is not documented
> however and the error message thrown from {{hadoop checknative -a}} is not
> helpful.
> {noformat}
> [tthompso@eat1-hcl4060 bin]$ hadoop checknative -a
> 14/03/13 00:51:02 WARN bzip2.Bzip2Factory: Failed to load/initialize
> native-bzip2 library system-native, will use pure-Java version
> 14/03/13 00:51:02 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
> Native library checking:
> hadoop: true
> /export/apps/hadoop/hadoop-2.3.0.li7-1-bin/lib/native/libhadoop.so.1.0.0
> zlib: true /lib64/libz.so.1
> snappy: true /usr/lib64/libsnappy.so.1
> lz4: true revision:99
> bzip2: false
> 14/03/13 00:51:02 INFO util.ExitUtil: Exiting with status 1
> {noformat}
> You can see that it wasn't compiled in here:
> {noformat}
> [mislam@eat1-hcl4060 ~]$ strings
> /export/apps/hadoop/latest/lib/native/libhadoop.so | grep initIDs
> Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_initIDs
> Java_org_apache_hadoop_io_compress_lz4_Lz4Decompressor_initIDs
> Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs
> Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs
> Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs
> Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_initIDs
> {noformat}
> After installing bzip2-devel and recompiling:
> {noformat}
> [tthompso@eat1-hcl4060 ~]$ hadoop checknative -a
> 14/03/14 23:00:08 INFO bzip2.Bzip2Factory: Successfully loaded & initialized
> native-bzip2 library system-native
> 14/03/14 23:00:08 INFO zlib.ZlibFactory: Successfully loaded & initialized
> native-zlib library
> Native library checking:
> hadoop: true
> /export/apps/hadoop/hadoop-2.3.0.11-2-bin/lib/native/libhadoop.so.1.0.0
> zlib: true /lib64/libz.so.1
> snappy: true /usr/lib64/libsnappy.so.1
> lz4: true revision:99
> bzip2: true /lib64/libbz2.so.1
> {noformat}
> {noformat}
> tthompso@esv4-hcl261:~/hadoop-common(li-2.3.0⚡) » strings
> ./hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so
> |grep initIDs
> Java_org_apache_hadoop_io_compress_lz4_Lz4Compressor_initIDs
> Java_org_apache_hadoop_io_compress_lz4_Lz4Decompressor_initIDs
> Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs
> Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs
> Java_org_apache_hadoop_io_compress_zlib_ZlibCompressor_initIDs
> Java_org_apache_hadoop_io_compress_zlib_ZlibDecompressor_initIDs
> Java_org_apache_hadoop_io_compress_bzip2_Bzip2Compressor_initIDs
> Java_org_apache_hadoop_io_compress_bzip2_Bzip2Decompressor_initIDs
> {noformat}
> The error message thrown should hint that perhaps libhadoop wasn't compiled
> with the bzip2 headers installed. It would also be nice if compile time
> dependencies were documented somewhere... :)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
