[jira] [Commented] (HADOOP-16601) Add support for hardware crc32 of nativetask checksums on aarch64 arch
[
https://issues.apache.org/jira/browse/HADOOP-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955103#comment-16955103
]
Hadoop QA commented on HADOOP-16601:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-16601 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16601 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16606/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add support for hardware crc32 of nativetask checksums on aarch64 arch
> --
>
> Key: HADOOP-16601
> URL: https://issues.apache.org/jira/browse/HADOOP-16601
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2.1
>Reporter: MacChen01
>Priority: Major
> Labels: performance
> Fix For: site
>
> Attachments: HADOOP-16601.patch, aarch64_nativetask_test.log,
> aarch64_nativetask_test.log
>
>
> Add support for aarch64 CRC instructions in nativetask module, optimize the
> CRC32 and CRC32C.
> Use the benchmark tools : nttest , the improvement is quite substantial:
> *CRC32 Zlib polynomial 0x04C11DB7*
> |KeyValueType-IO|Before(MB/s)|After(MB/s)|Improvement|
> |TextType-Write|425.98|602.92|+42%|
> |TextType-Read|796.06|1716.59|+116%|
> |BytesType-Write|474.25|686.84|+45%|
> |BytesType-Read|844.96|1955.03|+131%|
> |UnknownType-Write|434.84|608.81|+40%|
> |UnknownType-Read|805.76|1733.82|+115%|
>
>
> *CRC32C Castagnoli polynomial 0x1EDC6F41*
>
> |KeyValueType-IO|Before(MB/s)|After(MB/s)|Improvement|
> |TextType-Write|423.39|606.55|+43%|
> |TextType-Read|799.20|1783.28|+123%|
> |BytesType-Write|473.95|696.47|+47%|
> |BytesType-Read|846.30|2018.06|+138%|
> |UnknownType-Write|434.07|612.31|+41%|
> |UnknownType-Read|807.16|1783.95|+121%|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16601) Add support for hardware crc32 of nativetask checksums on aarch64 arch
[
https://issues.apache.org/jira/browse/HADOOP-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955100#comment-16955100
]
Hadoop QA commented on HADOOP-16601:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HADOOP-16601 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-16601 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16605/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add support for hardware crc32 of nativetask checksums on aarch64 arch
> --
>
> Key: HADOOP-16601
> URL: https://issues.apache.org/jira/browse/HADOOP-16601
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 3.2.1
>Reporter: MacChen01
>Priority: Major
> Labels: performance
> Fix For: site
>
> Attachments: HADOOP-16601.patch, aarch64_nativetask_test.log
>
>
> Add support for aarch64 CRC instructions in nativetask module, optimize the
> CRC32 and CRC32C.
> Use the benchmark tools : nttest , the improvement is quite substantial:
> *CRC32 Zlib polynomial 0x04C11DB7*
> |KeyValueType-IO|Before(MB/s)|After(MB/s)|Improvement|
> |TextType-Write|425.98|602.92|+42%|
> |TextType-Read|796.06|1716.59|+116%|
> |BytesType-Write|474.25|686.84|+45%|
> |BytesType-Read|844.96|1955.03|+131%|
> |UnknownType-Write|434.84|608.81|+40%|
> |UnknownType-Read|805.76|1733.82|+115%|
>
>
> *CRC32C Castagnoli polynomial 0x1EDC6F41*
>
> |KeyValueType-IO|Before(MB/s)|After(MB/s)|Improvement|
> |TextType-Write|423.39|606.55|+43%|
> |TextType-Read|799.20|1783.28|+123%|
> |BytesType-Write|473.95|696.47|+47%|
> |BytesType-Read|846.30|2018.06|+138%|
> |UnknownType-Write|434.07|612.31|+41%|
> |UnknownType-Read|807.16|1783.95|+121%|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16601) Add support for hardware crc32 of nativetask checksums on aarch64 arch
[ https://issues.apache.org/jira/browse/HADOOP-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955101#comment-16955101 ] MacChen01 commented on HADOOP-16601: [~ste...@apache.org] Thanks~ * Modify the patch according th Yetus's suggestion. * Run the native tests on arm64 , the execute file is :hadoop-rel-release-3.1.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/test/nttest, attaching the results file:[^aarch64_nativetask_test.log] * The issue is optimized for performance, adding new tests in test_bulk_crc32.c if need it. * Testing the hadoop test case (include: terasort/wordcount/kmeans..) with the test tool HiBench-HiBench-7.0 , all testcases had passed. > Add support for hardware crc32 of nativetask checksums on aarch64 arch > -- > > Key: HADOOP-16601 > URL: https://issues.apache.org/jira/browse/HADOOP-16601 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.2.1 >Reporter: MacChen01 >Priority: Major > Labels: performance > Fix For: site > > Attachments: HADOOP-16601.patch, aarch64_nativetask_test.log > > > Add support for aarch64 CRC instructions in nativetask module, optimize the > CRC32 and CRC32C. > Use the benchmark tools : nttest , the improvement is quite substantial: > *CRC32 Zlib polynomial 0x04C11DB7* > |KeyValueType-IO|Before(MB/s)|After(MB/s)|Improvement| > |TextType-Write|425.98|602.92|+42%| > |TextType-Read|796.06|1716.59|+116%| > |BytesType-Write|474.25|686.84|+45%| > |BytesType-Read|844.96|1955.03|+131%| > |UnknownType-Write|434.84|608.81|+40%| > |UnknownType-Read|805.76|1733.82|+115%| > > > *CRC32C Castagnoli polynomial 0x1EDC6F41* > > |KeyValueType-IO|Before(MB/s)|After(MB/s)|Improvement| > |TextType-Write|423.39|606.55|+43%| > |TextType-Read|799.20|1783.28|+123%| > |BytesType-Write|473.95|696.47|+47%| > |BytesType-Read|846.30|2018.06|+138%| > |UnknownType-Write|434.07|612.31|+41%| > |UnknownType-Read|807.16|1783.95|+121%| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16601) Add support for hardware crc32 of nativetask checksums on aarch64 arch
[ https://issues.apache.org/jira/browse/HADOOP-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16953630#comment-16953630 ] Steve Loughran commented on HADOOP-16601: - * it's best if you file a github PR with this JIRA at the start of the title * Look at the Yetus complaints about style, swap tabs for spaces we really love unit tests, and new patches to improve that coverage * you MUST run the native tests on arm64 and give us the results; yetus won't be testing your code * and if you can think of new tests to put into test_bulk_crc32.c, that'd be good too. Are there ways to break things? > Add support for hardware crc32 of nativetask checksums on aarch64 arch > -- > > Key: HADOOP-16601 > URL: https://issues.apache.org/jira/browse/HADOOP-16601 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.2.1 >Reporter: MacChen01 >Priority: Major > Labels: performance > Fix For: site > > Attachments: HADOOP-16601.patch > > > Add support for aarch64 CRC instructions in nativetask module, optimize the > CRC32 and CRC32C. > Use the benchmark tools : nttest , the improvement is quite substantial: > *CRC32 Zlib polynomial 0x04C11DB7* > |KeyValueType-IO|Before(MB/s)|After(MB/s)|Improvement| > |TextType-Write|425.98|602.92|+42%| > |TextType-Read|796.06|1716.59|+116%| > |BytesType-Write|474.25|686.84|+45%| > |BytesType-Read|844.96|1955.03|+131%| > |UnknownType-Write|434.84|608.81|+40%| > |UnknownType-Read|805.76|1733.82|+115%| > > > *CRC32C Castagnoli polynomial 0x1EDC6F41* > > |KeyValueType-IO|Before(MB/s)|After(MB/s)|Improvement| > |TextType-Write|423.39|606.55|+43%| > |TextType-Read|799.20|1783.28|+123%| > |BytesType-Write|473.95|696.47|+47%| > |BytesType-Read|846.30|2018.06|+138%| > |UnknownType-Write|434.07|612.31|+41%| > |UnknownType-Read|807.16|1783.95|+121%| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16601) Add support for hardware crc32 of nativetask checksums on aarch64 arch
[
https://issues.apache.org/jira/browse/HADOOP-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16937544#comment-16937544
]
Hadoop QA commented on HADOOP-16601:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
36m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 21 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 50 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
12s{color} | {color:green} hadoop-mapreduce-client-nativetask in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | HADOOP-16601 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12981172/HADOOP-16601.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 065d168b62bb 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3f89084 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16547/artifact/out/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16547/artifact/out/whitespace-tabs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16547/testReport/ |
| Max. process+thread count | 308 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16547/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add support for hardware crc32 of nativetask checksums on aarch64 arch
> --
>
> Key: HADOOP-16601
>
