[
https://issues.apache.org/jira/browse/HDFS-15570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
isurika resolved HDFS-15570.
----------------------------
Resolution: Fixed
> Hadoop Erasure Coding ISA-L Check Request..
> -------------------------------------------
>
> Key: HDFS-15570
> URL: https://issues.apache.org/jira/browse/HDFS-15570
> Project: Hadoop HDFS
> Issue Type: Test
> Components: erasure-coding, hadoop-client
> Affects Versions: 3.2.1
> Environment: *-OS Version* : CentOS Linux release 8.0.1905 (Core)
> *-Hadoop Version* : hadoop-3.2.1
>
> Reporter: isurika
> Priority: Trivial
> Fix For: 3.2.1
>
>
> I am testing the performance of erasure coding in Hadoop 3.2.1 version
> environment.
> Apply ISA-L by referring to the manual
> ([https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html])
> *▶ JOB LIST (Proceed in manual order)*
> # ISA-L library installed on all servers
> ld ISA-L library : [https://github.com/01org/isa-l]
> # hadoop build source (hadoop-3.2.1-src.tar.gz / mvn package -Pdist,native
> -Dtar -Drequire.isal -Drequire.snappy)
> # Deploy the built hadoop-3.2.1 folder to all servers
> But In Test (File upload Time/Select Orc table in Hive)
> There is no improvement in performance when tested.
> Ask for advice on work
> *[Question 1]*
> Is there anything wrong with the installation? Are there any missing tasks?
> *[Question 2]*
> Why doesn't it speed up ? (File upload Time/Select Orc table in Hive)
> *[Question 3]*
> Whether to use ISA-L in Hadoop, How to check?
> ※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L.
> But no
> ※ The warn log that occurs before applying ISA-L is It no longer occurs.
> ----------------------------------------------
> WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is
> not available in your platform... using builtin-java codec where applicable
> ----------------------------------------------
>
> *▶Reference information*
> =======================================
> *-CPU INFO (namenode :2 / datanode : 5)*
> ---------------------------------------------
> namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz
> datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
> ---------------------------------------------
> *-hadoop checknative (After source build)*
> ---------------------------------------------
> Native library checking:
> hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0
> zlib: true /lib64/libz.so.1
> zstd : false
> snappy: true /lib64/libsnappy.so.1
> lz4: true revision:10301
> bzip2: false
> openssl: true /lib64/libcrypto.so
> *ISA-L: true /lib64/libisal.so.2*
> ---------------------------------------------
> *-hdfs ec -listCodecs*
> Erasure Coding Codecs: Codec [Coder List]
> RS [RS_NATIVE, RS_JAVA]
> RS-LEGACY [RS-LEGACY_JAVA]
> XOR [XOR_NATIVE, XOR_JAVA]
> =======================================
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
