[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[ https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444894#comment-17444894 ] Chengwei Wang commented on HADOOP-17998: Hi [~sodonnell] , [~hexiaoqiao] , [~weichiu], can you help to reiview the PR? > Enable get command run with multi-thread > > > Key: HADOOP-17998 > URL: https://issues.apache.org/jira/browse/HADOOP-17998 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.1 >Reporter: Chengwei Wang >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17998.001.patch, HADOOP-17998.002.patch > > Time Spent: 20m > Remaining Estimate: 0h > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[ https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443584#comment-17443584 ] Chengwei Wang commented on HADOOP-17998: Hi [~ste...@apache.org], can you help to review the PR? > Enable get command run with multi-thread > > > Key: HADOOP-17998 > URL: https://issues.apache.org/jira/browse/HADOOP-17998 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.1 >Reporter: Chengwei Wang >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17998.001.patch, HADOOP-17998.002.patch > > Time Spent: 20m > Remaining Estimate: 0h > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[ https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442076#comment-17442076 ] Chengwei Wang commented on HADOOP-17998: Hi [~ste...@apache.org] , PR [https://github.com/apache/hadoop/pull/3645] has be created. > Enable get command run with multi-thread > > > Key: HADOOP-17998 > URL: https://issues.apache.org/jira/browse/HADOOP-17998 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.1 >Reporter: Chengwei Wang >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17998.001.patch, HADOOP-17998.002.patch > > Time Spent: 10m > Remaining Estimate: 0h > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[ https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441773#comment-17441773 ] Steve Loughran commented on HADOOP-17998: - can you submit this as a github PR? its how we review things now. thanks > Enable get command run with multi-thread > > > Key: HADOOP-17998 > URL: https://issues.apache.org/jira/browse/HADOOP-17998 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chengwei Wang >Priority: Major > Attachments: HADOOP-17998.001.patch, HADOOP-17998.002.patch > > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[
https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441639#comment-17441639
]
Chengwei Wang commented on HADOOP-17998:
Test case: 1 dir 240 files 13G
*1. Get with single thread.*
{code:java}
time hadoop fs -get /tmp/data/20211101 .
real 6m28.785s
user 0m18.844s
sys 0m44.953s
{code}
*2. Get with 10 threads.*
{code:java}
time hadoop fs -get -t 10 /tmp/data/20211101 .
real 0m58.721s
user 0m21.386s
sys 0m54.066s
{code}
> Enable get command run with multi-thread
>
>
> Key: HADOOP-17998
> URL: https://issues.apache.org/jira/browse/HADOOP-17998
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chengwei Wang
>Priority: Major
> Attachments: HADOOP-17998.001.patch, HADOOP-17998.002.patch
>
>
> CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and
> HADOOP-14698， and make put dirs or multiple files faster.
> So, It's necessary to enable get and copyToLocal command run with
> multi-thread.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[
https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441618#comment-17441618
]
Hadoop QA commented on HADOOP-17998:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
1s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 10m
11s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/249/artifact/out/branch-mvninstall-root.txt{color}
| {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
17s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 30s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 22m
52s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
30s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
2s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
2s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 6s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/249/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt{color}
| {color:orange} hadoop-common-project/hadoop-common: The patch generated 1
new + 32 unchanged - 9 fixed = 33 total (was 41) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 16s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[ https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441537#comment-17441537 ] Chengwei Wang commented on HADOOP-17998: Submit [^HADOOP-17998.002.patch] to fix checkstyle and license, the failures about javadoc and unit test seems not related to this patch. > Enable get command run with multi-thread > > > Key: HADOOP-17998 > URL: https://issues.apache.org/jira/browse/HADOOP-17998 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chengwei Wang >Priority: Major > Attachments: HADOOP-17998.001.patch, HADOOP-17998.002.patch > > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[
https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441510#comment-17441510
]
Hadoop QA commented on HADOOP-17998:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
11s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
8s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 13s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 22m
27s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 5s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HADOOP-Build/248/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt{color}
| {color:orange} hadoop-common-project/hadoop-common: The patch generated 8
new + 34 unchanged - 7 fixed = 42 total (was 41) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 33s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
8s{color} |
[jira] [Commented] (HADOOP-17998) Enable get command run with multi-thread
[ https://issues.apache.org/jira/browse/HADOOP-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440891#comment-17440891 ] Chengwei Wang commented on HADOOP-17998: Submit path v001 [^HADOOP-17998.001.patch] > Enable get command run with multi-thread > > > Key: HADOOP-17998 > URL: https://issues.apache.org/jira/browse/HADOOP-17998 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Chengwei Wang >Priority: Major > Attachments: HADOOP-17998.001.patch > > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
