[
https://issues.apache.org/jira/browse/HADOOP-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041968#comment-14041968
]
Ayappan commented on HADOOP-10744:
----------------------------------
The problem is due to the below lines of code in the lz4.c file
(src/main/native/src/org/apache/hadoop/io/compress/lz4/lz4.c)
// Little Endian or Big Endian ?
// Overwrite the #define below if you know your architecture endianess
#if defined (__GLIBC__)
# include <endian.h>
# if (__BYTE_ORDER == __BIG_ENDIAN)
# define LZ4_BIG_ENDIAN 1
# endif
#elif (defined(__BIG_ENDIAN__) || defined(__BIG_ENDIAN) ||
defined(_BIG_ENDIAN)) && !(defined(__LITTLE_ENDIAN__) ||
defined(__LITTLE_ENDIAN) || defined(_LITTLE_ENDIAN))
# define LZ4_BIG_ENDIAN 1
#elif defined(__sparc) || defined(__sparc__) \
|| defined(__powerpc__) || defined(__ppc__) || defined(__PPC__) \
|| defined(__hpux) || defined(__hppa) \
|| defined(_MIPSEB) || defined(__s390__)
# define LZ4_BIG_ENDIAN 1
#else
// Little Endian assumed. PDP Endian and other very rare endian format are
unsupported.
#endif
The flow goes like this in Power little Endian system. __GLIB__ is not defined
, it falls on next elif. The condition fails due to __LITTLE_ENDIAN__... flags,
falls on next elif and here it get satisfied! and sets LZ4_BIG_ENDIAN. The
thing is these flags __powerpc__, __ppc__, __PPC__ are also defined in PowerPC
Little Endian Architecture.
There is a similar file called lz4hc.c which also has the same lines code and
still it correctly recognizes Little Endian Architecture. The reason behind
that is..there is a include <stdlib.h> which actually turns on the __GLIBC__
flag.
And since now the decision is based on BYTE_ORDER, it identifies whether it is
Big Endian or Little Endian.
So lz4.c file should also have the "include <stdlib.h>" to make it work
correctly in Power Little Endian Architecture
> LZ4 Compression fails to recognize PowerPC Little Endian Architecture
> ---------------------------------------------------------------------
>
> Key: HADOOP-10744
> URL: https://issues.apache.org/jira/browse/HADOOP-10744
> Project: Hadoop Common
> Issue Type: Test
> Components: io, native
> Affects Versions: 2.2.0, 2.3.0, 2.4.0
> Environment: PowerPC Little Endian (ppc64le)
> Reporter: Ayappan
> Attachments: HADOOP-10744.patch
>
>
> Lz4 Compression fails to identify the PowerPC Little Endian Architecture. It
> recognizes it as Big Endian and several testcases(
> TestCompressorDecompressor, TestCodec, TestLz4CompressorDecompressor) fails
> due to this.
> Running org.apache.hadoop.io.compress.TestCompressorDecompressor
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.435 sec <<<
> FAILURE! - in org.apache.hadoop.io.compress.TestCompressorDecompressor
> testCompressorDecompressor(org.apache.hadoop.io.compress.TestCompressorDecompressor)
> Time elapsed: 0.308 sec <<< FAILURE!
> org.junit.internal.ArrayComparisonFailure:
> org.apache.hadoop.io.compress.lz4.Lz4Compressor_org.apache.hadoop.io.compress.lz4.Lz4Decompressor-
> byte arrays not equals error !!!: arrays first differed at element [1428];
> expected:<4> but was:<10>
> at
> org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
> at org.junit.Assert.internalArrayEquals(Assert.java:473)
> at org.junit.Assert.assertArrayEquals(Assert.java:294)
> at
> org.apache.hadoop.io.compress.CompressDecompressTester$CompressionTestStrategy$2.assertCompression(CompressDecompressTester.java:325)
> at
> org.apache.hadoop.io.compress.CompressDecompressTester.test(CompressDecompressTester.java:135)
> at
> org.apache.hadoop.io.compress.TestCompressorDecompressor.testCompressorDecompressor(TestCompressorDecompressor.java:58)
> .......
> .......
> .....
--
This message was sent by Atlassian JIRA
(v6.2#6252)
